Nokia Training

Enterprise Solutions, Nokia

1 © NOKIA 2004
 IP Clustering

2 © NOKIA 2004
 IP Clusters in IPSO : Agenda

• Key concept for clustering

• Feature Summary
• IPSO Clustering Concepts
• Configuration
• Configuring IPSO Clusters for Firewall-1
• Cluster Management
• Single-point management of IPSO features

3 © NOKIA 2004
 Key Concept for Nokia Clustering

 Multicast concept
 Clustering algorithms process
 Configuration
• IPSO
• Checkpoint

4 © NOKIA 2004
 Feature Summary

• Provides Load Balancing and High Availability of Check Point NG FP2 later
• Scales throughput and connection per second performance
• Connections and VPNs maintained if a member fails
• Nodes can be added and removed from a running cluster
• Single-system configuration and monitoring via Voyager and CLI
• Monitoring and event notification via SNMP

5 © NOKIA 2004
 Feature Summary: IPSO 3.7 Enhancements

• Cluster Management
• Single point management of IPSO features across cluster members
• Automatic synchronization of shared configuration
• Easier to add systems to the cluster
• Forwarding Mode
• Used in topologies where multicast MAC address can’t be used
• Secondary cluster protocol network
• Eliminates cluster protocol network as single point of failure
• Minor fixes and enhancements
• Better NAT support
• Support for SecuRemote with IP Pools

6 © NOKIA 2004
 Cluster enhancements in IPSO 3.8

• Switch Failure Recovery

• IGMP snooping support
• Secure cluster protocol

7 © NOKIA 2004
 Switch Failure Recovery

• Problem: If all cluster members are connected to a single switch, and that switch
fails, all connectivity is lost.

Internet

Cluster

8 © NOKIA 2004
 Switch Failure Recovery: Example

Internet

Cluster Maste
r

9 © NOKIA 2004
 Switch Failure Recovery: Limitations

• Only connections between cluster members and the switch are monitored. Other
connectivity failures will not be detected.

Internet

Cluster

10 © NOKIA 2004
 IGMP snooping support

• Problem: Cluster traffic is broadcast when in multicast mode.

Cluster

Switch

Servers

11 © NOKIA 2004
 IGMP snooping support (cont.)
Cluster IP: 10.5.1.93
IP Multicast Group: 239.144.1.93
Cluster MAC: 01-00-5e-10-01-5d

Cluster

Switch

Servers

13 © NOKIA 2004
 IGMP snooping support: Configuration

14 © NOKIA 2004
 IGMP snooping support: Limitations

• Requires switches with IGMP support

• Requires regular IGMP queries on the data networks
• Some switches initiate IGMP queries as part of IGMP snooping
• May also use external multicast router

15 © NOKIA 2004
 Secure Cluster Protocol

• Problem: Clusters which share protocol networks can interfere with each other if
misconfigured.

• In IPSO 3.8, all cluster protocol messages are digitally signed, based on a shared
secret.
• Any cluster protocol message with invalid signature is ignored
• Digital signature is also a function of cluster message contents, as a further
protection against tampering

• No additional configuration required.

• “cadmin” password is used as the shared secret.

16 © NOKIA 2004
 Feature Summary: Limitations

• No dynamic routing protocols supported  IPSO3.8 support it.

• 10/100 and Gig Ethernet interfaces only
• Only one IP address/network per clustered interface
• Crossover cable cannot be used as cluster protocol network
• All cluster members must run the same version of IPSO
• All cluster members must run the same set of Check Point products
• IP1xx and all non-IPSO Nokia platforms are not supported
• No VSX support
• No IPv6 support
• No TP mode support

17 © NOKIA 2004
 IPSO Clustering Concepts: Multicast Mode
Multicast
packet

Each member Hash • Default : SRC & DST

based on C_algorithm • NAT_EXT: SRC
• NAT_INT: DST

Internet
1024 • output of1024 Modulo

Who owns this bucket?

• Based on Performance factor

Accept or Drop it

18 © NOKIA 2004
 IPSO Clustering Concepts: Forwarding Mode
Unicast
packet

Master receives packet

Master hash • Default : SRC & DST

Internet based on C_algorithm • NAT_EXT: SRC
• NAT_INT: DST

• 1024 Modulo
1024

Master
Who owns this packet ?

• Based on Performance factor

Accept or Drop

19 © NOKIA 2004
 Multicast MAC Address Formation
• IPSO 3.6 uses the following multicast MAC address for the heartbeat interface:
01:00:5e:00:01:90. The master sends keep-alives to this MAC address.
• The cluster Mac uses the format of:
01:50:5a:X.Y.Z
• Where X,Y, and Z are the last 3 octets of the Cluster IP address in hex
• For example:
Cluster IP: 10.10.11.3
Cluster MAC: 01:50:5a:0a:0b:03

20 © NOKIA 2004
 Cluster MAC Address
• Binding to Cluster IP address is either…
• MAC address of the Master (Forwarding mode) LEAST EFFICIENT
• Master receives all packets
• Master must forward packets to Members

• Multicast MAC address (Multicast MAC mode) MOST FLEXIBLE

• Master and all Members see all packets, apply filter
• Works with switches that can handle multicast Ethernet frames

• Cluster uses the above MAC address for ARP messages

21 © NOKIA 2004
 Cluster Member Performance Rating
• By default, nodes with higher throughput and performance capabilities have higher
performance
• If all nodes have the same performance rating, the master is the first node that
joins the cluster or the node with the smallest IP address.
• These performance factors do not take into account memory installed.
Problem: IP740 w/ 512MB RAM <> IP740 w/ 1GB

* Each performance factor / total

22 © NOKIA 2004
 How to determine who is the Cluster Master
• IPSO uses performance factors to determine Master status.
• When Nokia appliances with equivalent performance factor ratings
SIMULTANEOUSLY join the cluster, the Nokia appliance with the lowest IP
address asserts Master status.

23 © NOKIA 2004
 What is a Bucket?
• A bucket is a conceptual container assigned by the Cluster Master to a Cluster
Member.
• The Cluster Master has 1024 buckets to assign. The buckets are numbered 0 –
1023
• A bucket can hold an unlimited amount of connections.

24 © NOKIA 2004
 Work Assignment – Filter and Buckets
• All (or Master-only in Forwarding mode) see all packets
• Apply highly efficient proprietary hash function to packet…
• IP addresses  SRC or DST or Both
• Protocol
• TCP/UDP port numbers (coming)
• IPSec security association number (coming, CP release)

• Hash function outputs a number 1-1024 “work bucket”

• If Member is assigned the packet’s work bucket, it processes it, otherwise drops it

1 2 3 4 5 6 7 8 … 1024
• Forwarding mode – processes or forwards based on work assignments

25 © NOKIA 2004
 Dynamic Load Balancing
A

1 2 3 4 5 6 7 8 … 1024
B

1 2 3 4 5 6 7 8 … 1024
C

1 2 3 4 5 6 7 8 … 1024
D

1 2 3 4 5 6 7 8 … 1024

100%
% L o ad in g

50%

0%
26 © NOKIA 2004
A B C D
 Hash Selection
Hash Selection: Select the fields on which hashing should be done for any packet:
• None: None of the fields have been selected for hashing.
• Default: Hash on source and destination IP addresses.
• NAT Internal: Hash on destination IP address. Used on internal interface when
NAT is configured on the node.
• NAT External: Hash on source IP address. Used on external interface when NAT
is configured on the node.

27 © NOKIA 2004
 Tcpdump outputs on heartbeat interface for
Cluster ID “25”
• MAC Addresses used
00:a0:8e:40:92:7b : Master
00:a0:8e:30:05:6d : Member
01:00:5e:00:01:90 : Multicast MAC
• A successful join
22:12:45.019391 I 0:a0:8e:30:5:6d 1:0:5e:0:1:90 0800
106: "25" JOIN (ip cluster)
22:12:45.019420 O 0:a0:8e:40:92:7b 0:a0:8e:30:5:6d
0800 66: "25" JOIN_NAK (Operation now in
progress)(ip cluster)
• A unsuccessful join request
22:12:36.648517 I 0:a0:8e:30:5:6d 1:0:5e:0:1:90 0800
94: "25" JOIN (ip cluster)
22:12:36.648540 O 0:a0:8e:40:92:7b 0:a0:8e:30:5:6d
0800 66: "25" JOIN_NAK (Device not configured)
(ip cluster)

28 © NOKIA 2004
 Tcpdump outputs continued
• A bucket de assignment request
23:13:25.201883 I 0:a0:8e:30:5:6d 0:a0:8e:40:92:7b 0800 62: "25"
BUCKET_DEASSIGN Bucket 59 -> Node 65535(ip cluster)
23:13:25.201910 O 0:a0:8e:40:92:7b 1:0:5e:0:1:90 0800 66: "25"
BUCKET_DEASSIGN_ACK (Node 2) Bucket 59 -> Node 65535(ip cluster)

• A bucket assignment request

23:55:57.030392 I 0:a0:8e:30:5:6d 0:a0:8e:40:92:7b 0800 62: "25"
BUCKET_ASSIGN Bucket 155 -> Node 2(ip cluster)
23:55:57.030437 O 0:a0:8e:40:92:7b 1:0:5e:0:1:90 0800 66: "25"
BUCKET_ASSIGN_ACK (Node 2) Bucket 155(ip cluster)

29 © NOKIA 2004
 A Cluster heartbeat interface on a stable cluster:
 
17:04:12.961992 O 0:a0:8e:40:4f:4c 1:0:5e:0:1:90 0800 1146: "101" KEEPALIVE 210mSec, Members=2, Seq=165(ip cluster)
17:04:13.000832 I 0:a0:8e:30:16:dd 0:a0:8e:40:4f:4c 0800 202: "101" CLIENT_KEEPALIVE Member=2, Seq=1(ip cluster)

 
Sequence of events:
1)      The Cluster Master sends keepalives to the cluster MAC address.
2)      The Cluster member sends keepalives to the Physical interface MAC address
of the Cluster Master

30 © NOKIA 2004
 When a member joins a cluster

 
16:08:52.120201 I 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 118: "101" JOIN Perf=490(ip cluster)
16:08:52.120234 O 0:a0:8e:40:4f:4c 0:a0:8e:30:16:dd 0800 66: "101" JOIN_NAK (Operation now in progress)(ip cluster)
16:08:52.120280 O 0:a0:8e:40:4f:4c 1:0:5e:0:1:90 0800 1146: "101" KEEPALIVE 200mSec, Members=2, Seq=3951(ip
cluster)
16:08:52.120323 O 0:a0:8e:40:4f:4c 0:a0:8e:30:16:dd 0800 154: "101" JOIN_ACK Node 2, M=10.10.11.1,
10.10.11.2(1:50:5a:a:b:2) M=10.10.11.1 205.226.27.148(1:50:5a:e2:1b:94) M=205.226.27.147 10.10.13.2(1:50:5a:a:d:2)
M=10.10.13.1 10.10.12.2(1:50:5a:a:c:2) M=10.10.12.1(ip cluster)

 
Sequence of events for a join request:
1)      Joining member announces the Cluster MAC it’s Performance
rating and Cluster ID it wants to join.
2)      Cluster Master begins processing the Join request
3)      Cluster Master finishes the Join request and tells the new Cluster
member what the Cluster master’s Physical and Cluster IP’s are.
* Mac address of Nokia Cluster IP(224.0.1.144) is 01:00:5e:00:01:90

31 © NOKIA 2004
 When a master leaves the cluster
 
16:24:15.543025 O 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 118: "101" JOIN Perf=490(ip cluster)
16:24:15.642854 O 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 118: "101" JOIN Perf=490(ip cluster)
16:24:15.742850 O 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 118: "101" JOIN Perf=490(ip cluster)
16:24:15.842856 O 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 70: "101" OFFER_MASTER 0 Members Perf 490(ip cluster)
16:24:15.942845 O 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 70: "101" OFFER_MASTER 0 Members Perf 490(ip cluster)
16:24:16.042858 O 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 70: "101" OFFER_MASTER 0 Members Perf 490(ip cluster)
16:24:16.142957 O 0:a0:8e:30:16:dd ff:ff:ff:ff:ff:ff 0806 42: arp who-has 10.10.11.2 tell 10.10.11.2 (1:50:5a:a:b:2)
 

Sequence of events when a Cluster Master Fails:

1)      The entire cluster reinitializes
2)      The cluster member sends out 3 JOIN messages which go unacknowledged.
3)      The cluster member sends out 3 OFFER_MASTER messages.
4)      Cluster member assumes MASTER status.
5)      A gratuitous arp for the cluster IP’s are sent out on all interfaces.
6)      Switches update the proper information.

32 © NOKIA 2004
 IGMP for Heartbeat communication

 
When a member joins a cluster, an IGMPv2 Leave message is sent out on each interface to 224.0.0.2.
 
 
15:43:44.993556 O 0:a0:8e:30:16:dd 1:0:5e:0:0:2 0800 46: 10.10.11.3 > 224.0.0.2: igmp leave 224.0.1.144 [tos 0xc0] [ttl
1]
 
Following an IGMP Leave, an IGMP Report is sent out each interface to 224.0.1.144
 
15:43:44.994131 O 0:a0:8e:30:16:dd 1:0:5e:0:1:90 0800 46: 10.10.11.3 > 224.0.1.144: igmpV2 report 224.0.1.144 [tos
0xc0] [ttl 1]

33 © NOKIA 2004
 Member joins

IGMP leave messages(224.0.0.2) for Nokia Cluster(224.0.1.144)

IGMP report for Nokia Cluster(224.0.1.144)

Cluster join request for 224.0.1.144 with CID & PF

Master reply to member saying “waiting”

In the process of joining

Master approved saying “Master IP information”

34 © NOKIA 2004
 ipsctl variables
• The view all cluster related variables use:
ipsctl -a net:ip:cluster

• To view bucket assignments use:

ipsctl -a net:ip:cluster | grep bucket

• The Cluster Master holds a list of all bucket assignments

• A Cluster Member holds a list of it’s own buckets.

35 © NOKIA 2004
 Configuration

-IPSO

37 © NOKIA 2004
 Configuration Overview

• Create the cluster on one node, specifying “cadmin” password

• Enter the cluster configuration settings
• Select the IPSO features which will have shared configuration
• Enable the cluster on that node
• On other nodes, join the cluster using the cluster ID , the “cadmin” password and
the IP address of any existing member
• As members join, they will receive configuration settings for the shared IPSO
features from the cluster Master

38 © NOKIA 2004
 Configuration: Creating an IPSO Cluster

39 © NOKIA 2004
 Configuration: Creating an IPSO cluster (cont.)

40 © NOKIA 2004
 Configuration: Cluster settings

41 © NOKIA 2004
 Configuration: Cluster settings (cont.)

42 © NOKIA 2004
 Configuration: Shared IPSO Features

43 © NOKIA 2004
 Configuration: Enabling the Cluster

44 © NOKIA 2004
 Configuration: Joining a Cluster

45 © NOKIA 2004
 Configuration: Joining a Cluster (cont.)

46 © NOKIA 2004
 Configuration: Joining a Cluster (cont.)

47 © NOKIA 2004
 Switch configuration for Clustering
Internet (In case of catalyst 2900)

Need ARP mapping Inbound port

P3 Mac-address-table static 0150.5aXX.xxxx fa 0/3 fa 0/1 fa 0/2

Output port
P1 P2

P1 P2 Mac-address-table static 0150.5aXX.xxxx fa 0/10 fa 0/1 fa 0/2

Mac-address-table static 0150.5aXX.xxxx fa 0/11 fa 0/1 fa 0/2
P10 P11

*note: Clustering itself works properly regardless of adding CAM Tables that prevent multicast traffic from being flooded
on all ports of switch.
48 © NOKIA 2004
 Need ARP mapping

Mac-address-table static 0150.5aXX.xxxx fa 0/3 fa 0/1 fa 0/20

Mac-address-table static 0150.5aXX.xxxx fa 0/3 fa 0/2 fa 0/20
Mac-address-table static 0150.5aXX.xxxx fa 0/20 fa 0/1 P3
P3 Mac-address-table static 0150.5aXX.xxxx fa 0/20 fa 0/2
P20

P1 P2

P1 P2
P20

P10 P11 P12

Mac-address-table static 0150.5aXX.xxxx fa 0/12 fa 0/2 fa 0/20
Mac-address-table static 0150.5aXX.xxxx fa 0/20 fa 0/2

Mac-address-table static 0150.5aXX.xxxx fa 0/10 fa 0/1 fa 0/20

Mac-address-table static 0150.5aXX.xxxx fa 0/11 fa 0/1 fa 0/20
Mac-address-table static 0150.5aXX.xxxx fa 0/20 fa 0/1

*note: Clustering itself works properly regardless of adding CAM Tables that prevent multicast traffic from being flooded
on all ports of switch.
49 © NOKIA 2004