-->
  • Recent Articles

    IPMP Configuration On Solaris servers

    IPMP (Internet Protocol Multi Pathing) is a standard feature of Solaris. Internet Protocol Multi Pathing is used to provide failover and outbound load
    spreading for network interfaces. You can use the normal Solaris, ifconfig, command to set it up.

    IPMP has the following features:

    It eliminates a single network adapter as a single point of failure in these cases, ( a. Network adapter failure b. Network link failure)
    It enables interfaces to fail over within approximately 10 seconds when using the default configuration.
    It can be configured by adjusting the parameters in the ”/etc/default/mpathd” file.
    It can be configured for use with both IPv4 and IPv6.
    It enables interfaces to be configured as standby interfaces.
    It can failback automatically . In /etc/default/mpathd There is a parameter called FAILBACK=yes, this could be changed to FAILBACK=no if you DON'T want the interface to fail back automatically.

    IPMP failure detection:
    Probe­-based failure detection
    -detects network error by sending ICMP ECHO_REQUEST messages
    Link based failure detection
    -detects network error by checking the IFF_RUNING


    Before configuring IP multi pathing, you must set the interfaces to use their own unique MAC (Media Access Control) addresses. You can do this by setting
    the local-mac-address? OBP parameter to true. You can either do this with the eeprom command, or setting it on the OBP.Below is an example using the eeprom.
    This method works for both SPARC and x86 systems. You could also use this command while Solaris is running.
    bash-3.00# eeprom "local-mac-address?=true"
    bash-3.00# eeprom local-mac-address?
    local-mac-address?=true
    bash-3.00#


    LINK-BASED IPMP:


    Link­-Based IPMP Requirements:

    Solaris 9 12/02 OS, at a minimum, must be installed.
    Network interfaces must use any of the following drivers: hme, eri, ce, ge, bge, qfe, dmfe, e1000g, ixgb, nge, nxge, rge, xge
    Unique MAC addresses must be configured on each of the interfaces.
    An IPMP group name must be assigned to interfaces

    Configuring Link­-Based IPMP:

    To make persistent Changes:
    root@<Hostname> # cat /etc/hostname.ce0
    <Hostname> netmask + broadcast + group ipmp0 up
    root@<Hostname> # cat /etc/hostname.ce1
    group ipmp0 standby up
    root@<Hostname> #

    ifconfig ce0 unplumb
    ifconfig ce0 plumb
    ifconfig ce1 plumb
    ifconfig ce0 `cat /etc/hostname.ce0` up
    ifconfig ce1 `cat /etc/hostname.ce1` up

    Test Plan:
    1. Once plumbed and up, we'll failover the IP to STANDBY NIC and try pinging that IP outside of the server.
    ifconfig -a
    if_mpadm -d ce0
    itsev1# ping <IP of the Host>
    2. if the IP is pinging outside of the server, then implementation is successful. Check the messages file as well for failover and failback information.
    if_mpadm -r ce0
     No test address configured on interface ce1; disabling probe-based failure detection on it
     No test address configured on interface ce0; disabling probe-based failure detection on it
     Successfully failed over from NIC ce0 to NIC ce1
     Successfully failed back to NIC ce0



    PROBE-BASED IPMP

    Probe-­Based IPMP Requirements:

    The Solaris 8 10/00 OS, as a minimum, must be installed.
    Unique MAC addresses must be configured on each network interface.
    Multiple network adapter interfaces must be connected on each subnet.
    An IPMP group name must be assigned to the group of interfaces.
    A test address is assigned to an interface.
    Additional hosts or devices must exist on the same subnet.

    Configuring Probe­-Based IPMP:


    For probe based multipathing, we need at least 3 IP addresses. 2 for test addresses and 1 for the active address.
    We will use the following IP addresses.
    10.194.36.23 on ce0 and this will be the active IP.
    10.194.36.24 on ce0:1 will be the test IP for ce0
    10.194.36.25 on ce1 will be the test IP for the ce1 interface.
    Cool. Let's configure probe based IP multi pathing. I will do this step by step cause it's very easy to get this wrong and then you wonder why it did not work.

    Again, I'll start from scratch plumbing both interfaces. I will first do ce0 and then I will move on to ce1.
    bash-3.00# ifconfig ce0 plumb
    bash-3.00# ifconfig ce1 plumb

    bash-3.00# ifconfig ce0 10.194.36.23 netmask + broadcast + up
    Setting netmask of ce0 to 255.255.255.128
    bash-3.00# ifconfig ce0 group ipmp0
    Now ce0 is setup with an IP address and put in the ipmp0 group.
    Next, we will configure the test IP address on ce0.
    bash-3.00# ifconfig ce0 addif 10.194.36.24 netmask + broadcast + -failover deprecated up
    Created new logical interface ce0:1
    Setting netmask of ce0:1 to 255.255.255.128

    bash-3.00# ifconfig -a
    lo0: flags=2001000849 mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    ce0: flags=1000843 mtu 1500 index 5
            inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
            groupname ipmp0
            ether 8:0:27:6f:c5:6e
    ce0:1: flags=9040843DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5
            inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
    ce1: flags=1000842 mtu 1500 index 4
            inet 0.0.0.0 netmask 0
            ether 8:0:27:1e:1d:e6
    bash-3.00#

    Notice that I did not use the word test anywhere in the ifconfig command. So, how did I configure it then? There are two things I want to point out here.
    1) I used the addif command to create a logical interface. This command means, add interface. This will check if there are any logical interfaces created
    and then just add the next available number to the logical interface. Some people use the logical interface as the failover IP. It doesn't really matter.
    You can make the logical or the physical the test. As long as there is a test interface.
    2) To create a test interface you specify two flags, -failover and deprecated. These two flags tell IPMP that this interface is a test interface.
    -failover means don't failover this IP address when the interface fails. This makes sense cause you don't want the test IP to failover in the event the physical
     interface fails. You want it to keep on pinging to check when the interface is fixed. deprecated means do not use this interface when initiating packets from
     this system. This is useful and needed when you use client software that initiated traffic from the host, or client system to the server where the server
     expects packets from a known interface.

    Now for the ce1 interface. It's already plumbed so I just need to configure an IP address, put it in the same group as ce0 and specify it as a test IP.
    bash-3.00# ifconfig ce1 10.194.36.25 netmask + broadcast + up
    Setting netmask of ce1 to 255.255.255.128
    bash-3.00# ifconfig ce1 group ipmp0 -failover deprecated
    bash-3.00# ifconfig -a
    lo0: flags=2001000849 mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    ce0: flags=1000843 mtu 1500 index 5
            inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
            groupname ipmp0
            ether 8:0:27:6f:c5:6e
    ce0:1: flags=9040843 mtu 1500 index 5
            inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
    ce1: flags=9040843 mtu 1500 index 4
            inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
            groupname ipmp0
            ether 8:0:27:1e:1d:e6

    I now it looks complicated but it really isn't. All we did was give ce1 an IP address, placed it in the ipmp0 group and made its a test interface.
    Go and test it.
    Again, I will disable the ce0 physical interface. Then I will ping with Ubuntu and I will check the ifconfig command if the IP did fail over.
    bash-3.00# if_mpadm -d ce0

    bash-3.00# ifconfig -a
    lo0: flags=2001000849 mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    ce0: flags=89000842
    mtu 0 index 2
            inet 0.0.0.0 netmask 0
            groupname ipmp0
            ether 8:0:27:6f:c5:6e
    ce0:1: flags=89040842
    mtu 1500 index 2
            inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
    ce1: flags=9040843
    mtu 1500 index 3
            inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
            groupname ipmp0
            ether 8:0:27:1e:1d:e6
    ce1:1: flags=1000843 mtu 1500 index 3
            inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
    bash-3.00#

    Look at the output of ifconfig -a. You will notice that ce1 now has a logical interface called ce1:1 and that the IP address 10.194.36.23, the active IP is
    plumbed and working. The test IP on ce0 is still pinging in the background to see if the interface has been fixed.
    Let me ping from Ubuntu to see if it's still working and let's have a look at the messages file.
    user@hostname:~$ ping 10.194.36.23
    PING 10.194.36.23 (10.194.36.23) 56(84) bytes of data.
    64 bytes from 10.194.36.23: icmp_req=1 ttl=255 time=0.537 ms
    64 bytes from 10.194.36.23: icmp_req=2 ttl=255 time=0.482 ms

    Jan 12 09:22:09 qserver in.mpathd[1248]: [ID 832587 daemon.error]
    Successfully failed over from NIC ce0 to NIC ce1

    Cool, it's working as planned. Now, let's fix the ce0 and see if the IP fails back.
    bash-3.00# if_mpadm -r ce0
    bash-3.00# ifconfig -a
    lo0: flags=2001000849 mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000
    ce0: flags=1000843 mtu 1500 index 2
            inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
            groupname ipmp0
            ether 8:0:27:6f:c5:6e
    ce0:1: flags=9040843 mtu 1500 index 2
            inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
    ce1: flags=9040843 mtu 1500 index 3
            inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
            groupname ipmp0
            ether 8:0:27:1e:1d:e6

    bash-3.00# tail -f /var/adm/messages
    Jan 12 09:29:49 qserver in.mpathd[1248]: [ID 620804 daemon.error]
    Successfully failed back to NIC ce0

    user@hostname:~$ ping 10.194.36.23
    PING 10.194.36.23 (10.194.36.23) 56(84) bytes of data.
    64 bytes from 10.194.36.23: icmp_req=1 ttl=255 time=0.433 ms
    64 bytes from 10.194.36.23: icmp_req=2 ttl=255 time=0.482 ms
    64 bytes from 10.194.36.23: icmp_req=3 ttl=255 time=0.490 ms

    Yep, still works. The interface has failed over and back and everything is still working.
    Again we need to place this setup in the hostname files, otherwise the config is lost across reboots. Let's set that up.
    For /etc/hostname.ce0
    10.194.36.23 netmask + broadcast + group ipmp0 up
    addif 10.194.36.24 netmask + broadcast + deprecated -failover up

    For /etc/hostname.ce1
    10.194.36.25 netmask + broadcast + group ipmp0 deprecated -failover up

    No comments