Applies To:: Next Steps Summary of Typical Solaris IP Multipathing (IPMP) Configurations (Doc ID 1010640.1)
Applies To:: Next Steps Summary of Typical Solaris IP Multipathing (IPMP) Configurations (Doc ID 1010640.1)
Applies To:: Next Steps Summary of Typical Solaris IP Multipathing (IPMP) Configurations (Doc ID 1010640.1)
1 of 7
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...
Welcome, Usman
Home
Knowledge
Knowledge
Certifications
Browse
Service Request
Favorites
Create SR
Keyword
Communities
Favorites
Profile
Contact Us
Help
Site Map
ID Filter
Sign Out
Advanced
Document Display
Type HOWTO
Migrated ID 214668
Status PUBLISHED(EXTERNAL)
Next Steps
Priority 3
To Bottom
Applies to:
Solaris SPARC Operating System - Version: 8 10/00 U2 to 10 9/10 - Release: 8.0 to 10.0
Solaris x64/x86 Operating System - Version: 8 10/00 U2 to 10 9/10 [Release: 8.0 to 10.0]
All Platforms
Goal
Excellent
Solution
Good
Poor
Did this resolve the problem?
Yes
Steps to Follow
Contents:
1. Production and test interfaces in the same IP subnet
1.1 With default router
1.2 Without default router
1.3 With dedicated hosts acting as test targets with "host-routes"
1.4 Configuration example for 1.1, 1.2 and 1.3
No
Just Browsing
Was this easily located?
Very Easy
Somewhat Easy
Not Easy
Feedback
Submit
Legend:
IPMP
T
p
t
--===
Good to know:
The operation of IP Multipathing (in.mpathd) depends on the routing configuration. Therefore in.mpathd always refers to the routing-table (IRE-cache)
to distinguish which test partner(s) are going to be used. Test partners are required for deciding if the interface is working properly.
in.mpathd by default chooses the default router as single test-target for probing. If no default router exists for the test-interface ip address, arbitrary
hosts on the link are detected by sending out "all hosts" multicast packets (224.0.0.1) on the wire to detect its test-partners. An "all routers"
multicasts (224.0.0.2) will never be sent! The first five hosts that are responding to the echo packets are chosen as targets for probing. In such a
6/21/2011 3:21 PM
Document 1010640.1
2 of 7
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...
non-default router environment, the in.mpathd always tries to find five probe targets via an "all hosts" multicast.
The in.mpathd detects failures and repairs by sending out 'icmp-echo' probes (like pinging) from all interfaces that are part of the IPMP group. If there
are no replies to five consecutive probes, the interface is considered to have failed and performs a failover of the network access to another interface
in the IPMP group. The probing rate depends on the failure detection time which is defined in /etc/default/mpathd. By default, failure detection time is
10 seconds. Thus the five probes will be sent within the failure detection time.
+----------------+
| default router |
+-------o--------+
| p=t:172.20.20.1
|
|
----------+-+------------------------+------------------------| |
p=t:172.20.20/24
| |
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
|
IPMP |
+---------+
IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group will send ICMP
requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of the /etc/defaultrouter file. There will
be NO "all hosts" multicast (224.0.0.1) sent out.
Advantages:
Easiest configuration for IPMP.
Disadvantages:
If the default router is down then IPMP failover does not work anymore. The in.mpathd does NOT send out multicasts to get other probe
targets, therefore all interfaces in the IPMP group get the state "failed". You can ignore this bug/feature when you have a default router which
is 100% online! Please look to RFE 4431511 and 4489960 for further information.
If you have a lot of IPMP groups, the default router has to reply to many ICMP requests. Take care about the default router. Do not overload
the default router.
The default router has to reliably answer ICMP echo requests. (e.g. firewalls sometimes do not)
+-----+
+-----+
| T 1 |
...
| T 5 |
+--o--+
+--o--+
| p=t:172.20.20.110
| p=t:172.20.20.150
|
|
|
|
-----------+-+--------------+---------------------+------| |
p=t:172.20.20.0/24
| |
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
|
IPMP |
+---------+
IPMP dynamically determines five arbitrary hosts on the link via an "all hosts" multicast (224.0.0.1). At least, you will need one probe target to get
IPMP working but be aware that one probe target is not reliable enough. As long as there are less than five targets available, the in.mpathd daemon
will continuously sent out "all hosts" multicasts to get a complete list of five probe targets.
Advantages:
easiest configuration for IPMP in a subnet without a default router.
very reliable due to the five targets.
Disadvantages:
a subnet without an default router is very rare.
+----------------+
+-----+
+-----+
| default router |
| T 1 |
...
| T 5 |
+-------o--------=
+--o--+
+--o--+
|
| p=t:172.20.20.110
| p=t:172.20.20.150
|
|
|
|
|
|
-----------+-+--------------+---------------------+------| |
p=t:172.20.20.0/24
| |
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
|
IPMP |
+---------+
6/21/2011 3:21 PM
Document 1010640.1
3 of 7
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...
Solaris 8 and 9 - Some "host routes" will be defined with a start script in /etc/rc2.d/S70staticroutes (refer to appendix A). When IPMP refers to the
routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the "host routes" are before the
default router in the routing table. If you have less than five "host routes", the default router (when available) will be used as a probe target too.
Examples:
1. Configuration with host1, host2 ... hostN (with N=5 or N>5), defaultrouter :
==> The first five hosts (host1 ... host5) will be defined as target, not the default router
2. Configuration with less than 5 hosts : for instance, host1, host2, defaultrouter :
==> The three systems (host1, host2, defaultrouter) will be defined as target.
The in.mpathd continuously tries to get five probe targets from the routing table but it will NOT send "all hosts" multicasts!
Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script will be necessary. If there is a
default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional probe targets defined via "host routes".
Advantages:
The default router is not a single point of failure anymore because you still have some other "host routes" as probe targets.
Higher availability.
Disadvantages:
More administrative work to do.
Due to static configuration you should regularly check that the probe targets are available.
localhost
host10 loghost
host10-test-qfe0
host10-test-qfe4
/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up
/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up
ifconfig output:
ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 172.20.20.210 netmask ffffff00 broadcast 172.20.20.255
qfe4: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 172.20.20.220 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:89:34
Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an active-standby
configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More Details available in the following Technical
Instruction Document 1012450.1 "Solaris 8 and up: How to Disable Outbound Load Balancing in an IPMP group".
+----------------+
| default router |
+-------o--------+
| p:172.20.20.1
| t:192.168.1.1
|
----------+-+------------------------+------------------------| |
p:172.20.20.0/24
| |
t:192.168.1.0/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
6/21/2011 3:21 PM
Document 1010640.1
4 of 7
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...
|
IPMP |
+---------+
IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group will send ICMP
requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of the /etc/defaultrouter file. There will
be NO "all hosts" multicast (224.0.0.1) sent out.
Advantages:
No additional production IP addresses required for the test interfaces
Disadvantages:
The interface of the default router has to reside in both the production AND test subnet.
Exceptional configuration of default router.
All other disadvantages already mentioned in section 1.1
2.2 With default router in production subnet net but without default router in test subnet
+----------------+
+-----+
+-----+
| default router |
| T 1 |
...
| T 5 |
+-------o--------=
+--o--+
+--o--+
p:172.20.20.1 |
| p:172.20.20.110
| p:172.20.20.150
|
| t:192.168.1.110
| t:192.168.1.150
|
|
|
-----------+-+--------------+---------------------+------| |
p:172.20.20.0/24
| |
t:192.168.1.0/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
|
IPMP |
+---------+
IPMP dynamically determines five arbitrary hosts on the link via an "all hosts" multicast (224.0.0.1). At least, you will need one probe target to get
IPMP working but be aware that one probe target is not reliable enough. As long as there are less than five targets available, the in.mpathd daemon
will continuously sent out "all hosts" multicasts to get a complete list of five probe targets.
Advantages:
easiest IPMP configuration if you don't have enough additional IP addresses available in the production subnet.
very reliable due to the five targets.
Disadvantages:
the probe targets must be available before you can setup the IPMP host.
more administrative work because you have to setup the probe targets as an additional interface in the test subnet.
Recommendation: Setup an additional logical network interface on target host.
(e.g. add a new interface to /etc/hostname.qfe0 with the 'addif' option)
If your test partners are on systems which run also IPMP you must add an logical network interface NOT flagged deprecated and NOT flagged
nofailover. An address associated with this interface will then be used as source address by responding properly to "all hosts" multicast
packets which are used for the automatic IPMP test partner detection. Be aware that Solaris 8 does NOT require the additional logical network
interface in the test subnet on the target host. So, in case of an upgrade from Solaris 8 to Solaris[TM] 9 or higher you have to change your
configuration.
Please refer to 2.5 for a detailed example.
+----------------+
+-----+
+-----+
| default router |
| T 1 |
...
| T 5 |
+-------o--------=
+--o--+
+--o--+
p:172.20.20.1 |
| p:172.20.20.110
| p:172.20.20.150
|
| t:192.168.1.110
| t:192.168.1.150
|
|
|
-----------+-+--------------+---------------------+------| |
p:172.20.20.0/24
| |
t:192.168.1.0/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
|
IPMP |
+---------+
Solaris 8 and 9 - Some "host routes" will be defined with a start script in /etc/rc2.d/S70staticroutes (refer to appendix A). When IPMP refers to the
routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the "host routes" are before the
default router in the routing table. If you have less than five "host routes", the default router (when available in the test subnet) will be used as a probe
target too.
Examples:
Please refer to the examples of section 1.3.
Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script will be necessary. If there is a
default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional probe targets defined via "host routes".
Advantages:
6/21/2011 3:21 PM
Document 1010640.1
5 of 7
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...
localhost
host10 loghost
host10-test-qfe0
host10-test-qfe4
/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up
/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up
ifconfig output:
ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
qfe4: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e8:89:34
Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an active-standby
configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More details available in the following Technical Instruction
Document 1012450.1 "Solaris 8 and up: How to Disable Outbound Load Balancing in an IPMP group".
2.5 Configuration example for 2.2 if you use IPMP on the test subnet
/etc/hosts
127.0.0.1
localhost
#
# IPMP active IP address:
#
172.20.20.10
host10 loghost
#
# IPMP test interfaces
#
192.168.1.210
host10-test-ce0
192.168.1.220
host10-test-ce1
#
# additional active interface in the 'test subnet'
# to be able to respond to probe packets from other
# IPMP setups (See 2.2). Will also failover.
#
192.168.1.211
host10-prod-ce0
/etc/hostname.ce0
host10 netmask + broadcast + group ipmp0 up \
addif host10-prod-ce0 netmask + broadcast + up \
addif host10-test-ce0 netmask + broadcast + deprecated -failover up
/etc/hostname.ce1
host10-test-ce1netmask + broadcast + deprecated -failover group ipmp0 up
ifconfig output:
6/21/2011 3:21 PM
Document 1010640.1
6 of 7
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...
See also:
System Administration Guide: IP Services > Configuring IPMP Groups > Configuring Target Systems
route
route
route
route
route
-p
-p
-p
-p
-p
add
add
add
add
add
-host
-host
-host
-host
-host
Note: If there is a default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional probe targets defined
via "host routes" (the first four).
If you don't like the default router to be used as a probe target at all, you'll have to delete the /etc/defaultrouter file and to define the default router as
the last static route, e.g.
# rm /etc/defaultrouter
#
#
#
#
#
#
route
route
route
route
route
route
-p
-p
-p
-p
-p
-p
add
add
add
add
add
add
Note: Due to the missing /etc/defaultrouter file, Solaris will automatically start the routing daemon 'in.routed' at boot time, so you should configure the
"quiet mode" for RIP to not supply any routing information to other systems:
# routeadm -m route:default quiet_mode=true
C. Summary
It's not easy to give a general recommendation because it depends on the network infrastructure which the customers have. Therefore you have to
discuss the various possibilities with your customer. Maybe the most used setups are 1.1 and 2.2.
To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle
Support Community, Oracle Solaris Kernel Community.
References
NOTE:1001790.1 - The differences between Network Adapter Failover in Sun[TM] Cluster 3.0 and IP Multipathing in Sun[TM] Cluster 3.1 or
Solaris[TM] Cluster 3.2
NOTE:1008064.1 - Solaris 10 IP Multipathing (IPMP) Link-based Only Failure Detection
NOTE:1012450.1 - Solaris[TM] 8 and up: How to disable outbound load balancing in an IPMP group
6/21/2011 3:21 PM
Document 1010640.1
7 of 7
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...
NOTE:1017689.1 - How to Use Solaris IP Multipathing (IPMP) for Network Path Failover
Related
Products
Sun Microsystems > Operating Systems > Solaris Operating System > Solaris SPARC Operating System > Network Interface > ipmp, ipmpstat,
mpathd, IP Multipathing
Sun Microsystems > Operating Systems > Solaris Operating System > Solaris x64/x86 Operating System > Network Interface > ipmp, ipmpstat,
mpathd, IP Multipathing
Knowledge Categories
ACTIVE STANDBY
CONFIGURATION
DEPRECATED
FAILOVER
IPMP
Return To Top
Favorites Profile Contact Us Help Site Map Sign Out
Copyright 2007, 2011, Oracle and/or its affiliates. All rights reserved
6/21/2011 3:21 PM