return to OCLUG Web Site
A Django site.
October 15, 2012

Pythian
pythian
» Adding Networks to Exadata: Fun with policy routing

I’ve noticed that Exadata servers are now configured to use Linux policy routing. Peeking at My Oracle Support, note 1306154.1 goes in a bit more detail about this configuration. It’s apparently delivered by default with factory build images 11.2.2.3.0 and later. The note goes on to explain that this configuration was implemented because of asymetric routing problems associated with the management network:

Database servers are deployed with 3 logical network interfaces configured: management network (typically eth0), client access network (typically bond1 or bondeth0), and private network (typically bond0 or bondib0). The default route for the system uses the client access network and the gateway for that network. All outbound traffic that is not destined for an IP address on the management or private networks is sent out via the client access network. This poses a problem for some connections to the management network in some customer environments.


It goes onto mention a bug where this was reported:

@ BUG:11725389 – TRACK112230: MARTIAN SOURCE REPORTED ON DB NODES BONDETH0 INTERFACE

The bug is not public, but the title does show the type of error messages that would appear if a packet with a non-local source address comes out.

This configuration is implemented using RedHat Oracle Linux-style /etc/sysconfig/network-scripts files, with matched rule- and route- files for each interface.

A sample configuration, where the management network is in the 10.10.10/24 subnet, is:

[root@exa1db01 network-scripts]# cat rule-eth0
from 10.10.10.93 table 220
to 10.10.10.93 table 220
[root@exa1db01 network-scripts]# cat route-eth0
10.10.10.0/24 dev eth0 table 220
default via 10.10.10.1 dev eth0 table 220

This configuration tells traffic originating from the 10.10.10.93 IP (which is the management interface IP on this particular machine), and also traffic destined to this address, to be directed away from the regular system routing table, to a special routing table 220. route-eth0 configures table 220 with two router: one for the local network, and a default route through a router on the 10.10.10.1 network.

This contrasts with default gateway of the machine itself:

[root@exa1db01 network-scripts]# grep GATEWAY /etc/sysconfig/network
GATEWAYDEV=bondeth0
GATEWAY=10.50.50.1

The difference between this type of policy routing and regular routing is that traffic with the _source_ address of 10.10.10.93 will automatically go through default gateway 10.10.10.1, regardless of the destination. (The bible for Linux routing configuration is the Linux Advanced Routing and Traffic Control HOWTO, for those looking for more details)

I ran into an issue with this configuration when adding a second external network on the bondeth1 interface. I set up the additional interface configuration for a network, 10.50.52.0/24:

[root@exa1db01 network-scripts]# cat ifcfg-bondeth1
DEVICE=bondeth1
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.50.52.104
NETMASK=255.255.255.0
NETWORK=10.50.52.0
BROADCAST=10.50.52.255
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100"
IPV6INIT=no
GATEWAY=10.50.52.1

I also added rule and route entries:

[root@exa1db01 network-scripts]# cat rule-bondeth1
from 10.50.52.104 table 211
to 10.50.52.104 table 211
[root@exa1db01 network-scripts]# cat route-bondeth1
10.50.52.0/24 dev bondeth1 table 211
10.100.52.0/24 via 10.50.52.1 dev bondeth1 table 211
default via 10.50.52.1 dev bondeth1 table 211

This was a dedicated data guard network to a remote server, IP 10.100.52.10.

The problem with this configuration was: it didn’t work. Using tcpdump, I could see incoming requests come in on the bondeth1 interface, but the replies come out the system default route on bondeth0, and not reaching their destination. After some digging, I did find the cause of the problem: in order to determine the packet source IP, the kernel was looking up the destination in the default routing table (table 255). And the route for the 10.100.52.0 network was in non-default table 211. So the packet followed the default route instead, got a source address in the client-access network, and never matched any of the routing rules for the data guard network.

The solution ended up being rather simple: taking out the “table 211″ for the data guard network route, effectively putting it in the default routing table:

[root@exa1db01 network-scripts]# cat route-bondeth1
10.50.52.0/24 dev bondeth1 table 211
default via 10.50.52.1 dev bondeth1 table 211
10.100.52.0/24 via 10.50.52.1 dev bondeth1

And then we ran into a second issue: the main interface IP could now be reached, but not the virtual IP (VIP). This is because the rule configuration, taken from the samples, doesn’t list the VIP address at all. To avoid this issue, and in case of VIP addresses migrating from other cluster nodes, we set up a netmask in the rule file, making all addresses in the data guard network use this particular routing rule:

[root@exa1db01 network-scripts]# cat rule-bondeth1
from 10.50.52.0/24 table 211
to 10.50.52.0/24 table 211

So to sum up, when setting up interfaces in a policy-routed Exadata system remember to:

  • Set up the interface itself and any bonds using ifcfg- files
  • Create a rule- file for the interface, encompassing every possible address the interface could have. I added the entire IP subnet. Add “from” and “to” lines with a unique routing table number
  • Create a route- file for the interface, listing a local network route and a default route with the default router of the subnet, all using the table number defined on the previous step
  • Add to the route- file any static routes required on this interface, but don’t add a table qualifier

The final configuration:

[root@exa1db01 network-scripts]# cat ifcfg-eth8
DEVICE=eth8
HOTPLUG=no
IPV6INIT=no
HWADDR=00:1b:21:xx:xx:xx
ONBOOT=yes
MASTER=bondeth1
SLAVE=yes
BOOTPROTO=none
[root@exa1db01 network-scripts]# cat ifcfg-eth12
DEVICE=eth12
HOTPLUG=no
IPV6INIT=no
HWADDR=00:1b:21:xx:xx:xx
ONBOOT=yes
MASTER=bondeth1
SLAVE=yes
BOOTPROTO=none
[root@exa1db01 network-scripts]# cat ifcfg-bondeth1
DEVICE=bondeth1
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.50.52.104
NETMASK=255.255.255.0
NETWORK=10.50.52.0
BROADCAST=10.50.52.255
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100"
IPV6INIT=no
GATEWAY=10.50.52.1
[root@exa1db01 network-scripts]# cat rule-bondeth1
from 10.50.52.0/24 table 211
to 10.50.52.0/24 table 211
[root@exa1db01 network-scripts]# cat route-bondeth1
10.50.52.0/24 dev bondeth1 table 211
default via 10.50.52.1 dev bondeth1 table 211
10.100.52.0/24 via 10.50.52.1 dev bondeth1

February 14, 2011

Pythian
pythian
» Installing TOra with Oracle Support on Ubuntu 10.10

Good afternoon sports fans. I’ve had a couple of requests to update my world famous blog on installing TOra. Frankly I have been wanting to get this out for a while but duties other than blogging have taken precedence. That and I think my blogging ran out of entropy and needed some other IO to get going again. Well it’s now time for you all to let out that breath you have been holding since the ‘perfect 10′ was released (on 10.10.10 no less).

Once again the instructions are much the same, but there was a couple of things I had to change which were mostly around the order of things. I installed this on Kubuntu 10.10 x86_64 systems with the newest Oracle 11.2 client. The version of TOra this time around is 2.1.2 which works fine except for one really really frustrating thing, that I will mention at the end to keep you in suspense. I ran through this 3 times on 3 separate systems (VM, work and home) and in all cases I got through it mostly unharmed with only a few humps. So here we go, it’s time to…

Make some place to build the package

Normally I will create a safe haven for this type of work, away from the other clutter on my drive. When doing this I did it all under ~/Documents/blogs/tora-meercat with subdirectories for oracle and building tora separate. It’s all up to you, but I keep it sorted because they probably will not get deleted for my normal retentions of ~30 years (and counting).

Hereby referred to as /path/to/tora.

Get the packages

First thing you’ll want is the newest oracle client packages. You can get them from the Oracle instant client download page. You need the following:

  • oracle-instantclient11.2-basiclite-11.2.0.2.0.x86_64.rpm
  • oracle-instantclient11.2-devel-11.2.0.2.0.x86_64.rpm
  • oracle-instantclient11.2-sqlplus-11.2.0.2.0.x86_64.rpm

Now we need the utilities and dependencies to build the package. I had to swap the order from last time because the TOra build deps now need packages that we used to install later. Say yes to any suggestions it has, some of them may already be installed.

sudo apt-get install libqt3-mt-dev libqt3-compat-headers libqscintilla-dev build-essential g++ gcc autoconf automake flex zlib1g-dev docbook-xsl debhelper alien libaio1 dpatch fakeroot xsltproc texi2html texinfo libqt3-mt-psql libqt3-mt-odbc config-package-dev cmake qt4-dev-tools

And now the TOra build deps.

sudo apt-get build-dep tora

Now go to your build directory and get the source.

cd /path/to/tora
apt-get source tora

Next we will install our Oracle packages. Just like last time, we do this:

sudo alien -i *.rpm

Environment Variables

Now that we laid the groundwork, it’s time to set up the build environment.

export ORACLE_HOME="/usr/lib/oracle/11.2/client64"
export LD_LIBRARY_PATH="${ORACLE_HOME}/lib"
export TNS_ADMIN="${ORACLE_HOME}"

This time around we did not set the CMAKE environment. The reason is simple. It did not work, and was apparently not needed. I am not sure why, but here we are. Instead we’re back to making a symbolic link to the oracle includes.

sudo ln -s /usr/include/oracle/11.2/client64 $ORACLE_HOME/include

Once again I want to ensure that I the above environment variables are available when we run this in our X desktop. We’ve done this different ways in past blogs, today we’re going to add them to our .xsessionrc.


$ cat ~/.xsessionrc
export ORACLE_HOME=/usr/lib/oracle/11.2/client64
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export TNS_ADMIN=$ORACLE_HOME

You’ll need to log out and in again for it to take effect, so run it from the command line until you get a chance to do that.

Increment the version (not required)

Feel free to skip to the next section.

This time I wanted to choose the “increase my package version to avoid update issues” method we’ve discussed in previous editions of this blog. Check back to the older blogs to see the other methods of avoiding updates overwriting your package, but I like to mix things up a bit. If you want to do this, change the bottom to whatever you like but a note that the system is VERY picky about the formatting. Watch all the spacing and line breaks, they all need to be there. There is also a double space between the email address and date, so watch for it.


cat << EOF >> /path/to/tora/tora-2.1.2/debian/changelog
> tora (2.2.2-1) unstable; urgency=low
>
> * Package renumbered to prevent updates from clobbering it
> -- Brad Hudson Mon, 10 Jan 2011 12:24:00 -0500
> EOF

Interlude. Do you recall a few weeks back when I posted a comment that the blog was about done? This is where I was. Then 2+ weeks of back to back (to back) raging infernos. Then a bit of time to decompress. Now, where were we … :)

Compile and create the .deb file

Easy as pie. Here’s a twist, run this using the time util and see how long it takes to compile. On my work PC it took 6m57.792s, give or take a ms.


time fakeroot debian/rules binary

Install it

If all goes to plan you should have a .deb package file, install it as below. Your package name may differ if you chose not to increment the version as I had above.


sudo dpkg -i tora_2.2.2-1ubuntu1_amd64.deb

To test it, simply run tora in the same shell in which you compiled it. Your menu item will not see Oracle until you log out and in again.

One really frustrating thing

I have run through this process on at least 3 systems. It works on all 3 and functions correctly while I am using it, I can run basic sql and browse the schema (which is about all I really ever use it for). It also has a problem on all 3. No matter where I launch the program from it will hang when I try to close it. I have been unable to locate the source of grief, but I have ruled out connectivity for sure because I have done this from both local and remote network segments. It does not have this issue when Oracle support is not available. I never had this issue in previous versions, it may be a bug. I ran an strace on the program and once it hangs it does not seem to be doing anything, but I can still ctrl-c (or xkill) the program. I have not been able to find any open bug reports on it but my time has been very limited. If you find something out about this or find a cure please post a comment. I’ll do same if I get to solving it.

That’s it for this edition. I guess I am only two months away from having another one of these due out. Thanks for reading!

September 30, 2010

Pythian
pythian
» Linux, Windows, virtual machines and you — DIY VPN jump box

Being in the remote administration business is a strange beast and offers lots of challenges, but when you are working for multiple clients sometimes connecting to the servers can be challenging enough. Here’s a little idea that I had this morning that may save someone some grief, so I thought I would jot it down for all to see.

One of the issues I have connecting to some clients revolves around my linux desktop. Sure I can connect to many VPN devices using vpnc or other tools, but in some cases client policy prohibits such reasonable behaviour due to a) single vendor plugins; b) bad java or plugin issues; c) host checking software or; d) Antivirus requirements that do not recognize linux agents. My problem is that I do not want to administer Unix servers from a Windows system, it’s just … wrong. Like, fundamentally wrong. (on cue, someone I know is calling me a zealot)

Conventional wisdom would dictate that I need to run a Windows VM, open the VPN client, and then connect using Putty or similar. Sure this works. It works fine. Except that some VMs do not interact with the clipboard properly, and I prefer to connect from my linux console because that’s where all my tools are. When I am documenting tickets I rely heavily on being able to seamlessly get things from my console to my documents without having the third party involved. I want to be able to work on all systems the same way, because it improves efficiency. That and the fact that I would rather use less Windows than more.

The solution I came up with is simplistic, but allows me to use the Windows VPN client, and bypass Windows for everything else. Basically a DIY VPN jump box. All you need is a Windows VM, and Cygwin. It’s just crazy enough to work, and does!

Here’s what you do.

Pre-flight

  • Start your Windows VM. (Don’t have one already? Check out VirtualBox. Installing it is out of scope.)
  • Log in with an administrator account if you are running Windows XP, Win7 and Vista users can log in as a normal user providing you have rights to run things as administrator.
  • Verify the IP of your VM. This means you will need to use bridged networking, none of this will work with NAT type of networking so caveat emptor.
    ipconfig
    

Cygwin install

  1. Grab the cygwin installer from the Cygwin site. You should also check out the license if you are into that sort of thing.
  2. Run the installer, Win7 and Vista users should right click on the program and “Run as administrator”. Documentation on the installer can be found at the network setup help section. You should be able to take the defaults for most of it. It does an annoying thing where it asks you for a location to download the packages, and it defaults to c:\Program Files\Mozilla, I changed it to my Download folder.
  3. When you get to the “Choose a download site” (aka mirror) list choose, one that is close to you. If you have no preference then any of the mirrors should work, but speed may vary depending on geographic proximity or network link speed at the mirror.
  4. The next screen is the package selection screen. It looks daunting, but here what you need to get this working.
    • In the search box type openssh. This will Narrow down the package selection to one group, called Net.
    • Click the + next to the word Net to expand the group.
    • In the New column there should be a Skip with a little circley-arrowy icon next to it. Click on the circley-arrowy icon, which will replace one of the n/a columns with a selected box (in the bin column). Now sshd will be selected for install.
    • If you need any other special packages, like telnet, you can search for them here. Incidentally, to get telnet you need to search for inetutils.
  5. Now follow the defaults and wait for Cygwin to complete the install. This could take a while.

Post Cygwin install

There is a few steps you’ll need to do manually now to get the ssh daemon running.

  1. If you do not have a password for your Windows user then set one up now (and have someone swat you on the nose with a rolled up newspaper. Bad SA. Bad.). ssh needs a password to work.
  2. Start a cygwin bash session either by using the desktop icon if you chose to create one, or using the link under Start menu->Programs->Cygwin. Win7 and Vista users, right click on the console icon and select ‘Run As Administrator’ the first time.
  3. In the console, run the following command to set up the ssh host keys and whatnot. This could take a while as well, so don’t get discouraged and ^C in the middle of it like I did. This process also sets the service to start on boot, if you do not want it to start automatically you will need to disable it manually.
    ssh-host-config -y
    
  4. When you get back to a prompt, follow-it up with this which turns on the service immediately. Net savings 10+ clicks!
    cygrunsrv -S sshd
    
  5. If you are running Win7, Vista or any sort of firewall program you will need to allow port 22, or program C:\Cygwin\usr\sbin\sshd.exe.

Reap the benefits

  1. Connect to your Windows VM desktop.
  2. Start the VPN client and connect to the VPN.
  3. ssh into your Windows VM using the IP you found in the pre-flight check.
  4. Now ssh or telnet into the system on the client end of the VPN tunnel.
  5. You could even ssh tunnel through the VM for GUI jumps or web access.

Hopefully someone will find this useful other than me. It’s so simple I really don’t know why I did not think of it before, but I think it’s probably because I only have one or two VPNs with soft clients. Some people have many more. I plan to play with it a bit to see how low I can set the resource allocation to the VM. I think I can probably cut a Win7 VM down to 256MB with the right combination of settings and still have good results because in this case I only care about network. I’ll let you know how it turns out.