Here is the setup I used:
4 VMs, 1 client, 1 director and 2 Realservers
client has one NIC in NAT mode
director has 1 NICs in NAT mode
Realservers have one NIC each in NAT mode
client has one NIC in NAT mode
director has 1 NICs in NAT mode
Realservers have one NIC each in NAT mode
The Director and Realservers need not be in the same Layer 2 domain. Install arptable using apt-get on realservers.
Director
eth0 Link encap:Ethernet HWaddr 00:0c:29:07:96:cf
inet addr:192.168.25.135 Bcast:192.168.25.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe07:96cf/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3379 errors:0 dropped:0 overruns:0 frame:0
TX packets:2201 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1295133 (1.2 MB) TX bytes:233436 (233.4 KB)
Interrupt:19 Base address:0x2000
eth0:110 Link encap:Ethernet HWaddr 00:0c:29:07:96:cf
inet addr:192.168.25.110 Bcast:192.168.25.110 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:19 Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:168 (168.0 B) TX bytes:168 (168.0 B)
Install ipvsadm using apt-get. Restart director.
sudo bash -c 'echo 0 > /proc/sys/net/ipv4/ip_forward'
sudo bash -c 'echo 1 > /proc/sys/net/ipv4/conf/all/send_redirects'
sudo bash -c 'echo 1 > /proc/sys/net/ipv4/conf/default/send_redirects'
sudo bash -c 'echo 1 > /proc/sys/net/ipv4/conf/eth0/send_redirects'
sudo /sbin/ifconfig eth0:110 192.168.25.110 broadcast 192.168.25.110 netmask 255.255.255.255
sudo /sbin/route add -host 192.168.25.110 dev eth0:110
Then set up the load balancer
sudo /sbin/ipvsadm -C
sudo /sbin/ipvsadm -A -t 192.168.25.110:8080 -s rr
sudo /sbin/ipvsadm -a -t 192.168.25.110:8080 -r 192.168.25.131:8080 -i -w 1
sudo /sbin/ipvsadm -a -t 192.168.25.110:8080 -r 192.168.25.140:8080 -i -w 1
$ sudo /sbin/ipvsadm -l -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.25.110:8080 rr
-> 192.168.25.131:8080 Tunnel 1 0 0
-> 192.168.25.140:8080 Tunnel 1 0 0
Client
eth0 Link encap:Ethernet HWaddr 00:0c:29:d0:bc:7f
inet addr:192.168.25.128 Bcast:192.168.25.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fed0:bc7f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2245 errors:0 dropped:0 overruns:0 frame:0
TX packets:1104 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1327858 (1.3 MB) TX bytes:100896 (100.8 KB)
Interrupt:19 Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
RealServer 1
eth0 Link encap:Ethernet HWaddr 00:0c:29:ae:54:3c
inet addr:192.168.25.140 Bcast:192.168.25.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feae:543c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2064 errors:0 dropped:0 overruns:0 frame:0
TX packets:2620 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:259049 (259.0 KB) TX bytes:238162 (238.1 KB)
Interrupt:19 Base address:0x2024
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
tun1 Link encap:IPIP Tunnel HWaddr
inet addr:192.168.25.110 P-t-P:192.168.25.110 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MTU:1480 Metric:1
RX packets:48 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3352 (3.3 KB) TX bytes:0 (0.0 B)
RealServer 2
eth0 Link encap:Ethernet HWaddr 00:0c:29:b9:69:38
inet addr:192.168.25.131 Bcast:192.168.25.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb9:6938/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:299 errors:0 dropped:0 overruns:0 frame:0
TX packets:226 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:27935 (27.9 KB) TX bytes:31852 (31.8 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
tun1 Link encap:IPIP Tunnel HWaddr
inet addr:192.168.25.110 P-t-P:192.168.25.110 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MTU:1480 Metric:1
RX packets:48 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3352 (3.3 KB) TX bytes:0 (0.0 B)
Then configure Realservers for Tunneling to work properly
RealServer 1
$ sudo modprobe ipip
$ modprobe tun
$ sudo ip tunnel add tun1 mode ipip local 192.168.25.140
$ sudo ifconfig tun1 192.168.25.110 broadcast 192.168.25.110 netmask 255.255.255.255
$ sudo ifconfig tun1 up
$ sudo route add -host 192.168.25.110/32 dev tun1
Reverse path filter was introduced to support Strong Send and Receive. Which now most of the Operating Systems have as default setting. In Strong Send and Receive OS transmits outgoing packet only from the same interface from which it has received it. In tunneling packets are received on tunnel interface but they go out from different interface. Hence, if RPF is enabled on back-end servers response will not be delivered to clients. Hence, Reverse Path Filtering should be disabled on Realservers.
This is very important. Things will not work and you will pull your hair out for two days unless you do the following
$ sudo bash -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
$ sudo bash -c 'echo 0 > /proc/sys/net/ipv4/conf/tun1/rp_filter'
$ sudo bash -c 'echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter'
$ sudo bash -c 'echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter'
Handle ARP problem
$ sudo arptables -A INPUT -d 192.168.25.110 -j DROP
$ sudo arptables -L -n
Chain INPUT (policy ACCEPT)
-j DROP -d 192.168.25.110
Chain OUTPUT (policy ACCEPT)
Chain FORWARD (policy ACCEPT)
Do likewise for RealServer 2
Then start the web server on RealServer 1 and 2
bruce@ubuntu:~/webserver$ cat index.html
<html>
<head>
<meta http-equiv="Pragma" content="no-cache">
<!-- Pragma content set to no-cache tells the browser not to cache the page
This may or may not work in IE -->
<meta http-equiv="expires" content="0">
<!-- Setting the page to expire at 0 means the page is immediately expired
Any vales less then one will set the page to expire some time in past and
not be cached. This may not work with Navigator -->
</head>
<title>Fake WWW server 1</title>
<body>
This is fake WWW server 1
</body>
</html>
bruce@ubuntu:~/webserver$ python -m SimpleHTTPServer 8080
or install telnet and use that instead.
$ sudo apt-get install telnetd
$ sudo /etc/init.d/openbsd-inetd restart
$ sudo apt-get install telnetd
$ sudo /etc/init.d/openbsd-inetd restart
Now connect to director from client. I use lynx
openflow@mininet-vm:~$ sudo ip neigh flush all
openflow@mininet-vm:~$ lynx -dump http://192.168.25.110:8080/
This is fake WWW server 1
openflow@mininet-vm:~$ lynx -dump http://192.168.25.110:8080/
This is fake WWW server 2
Wireshark capture on RealServer 1 shows IP Tunneling in progress
Capture on tun1
Capture on tun1
Capture on eth0
