Sto cercando di implementare un bilanciamento del carico keepalived basato su NAT per i nostri proxy di calamari in uscita. Ho impostato 2 VM keepalived che condividono un VIP:
[root@keepalived-01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 52:54:b3:65:10:17 brd ff:ff:ff:ff:ff:ff
inet 10.65.10.17/24 brd 10.65.10.255 scope global eth0
inet 10.65.10.101/32 scope global eth0
[root@keepalived-01 ~]#
e quindi dovrebbe inoltrare il traffico ai proxy calamari:
[root@keepalived-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.65.10.101:3128 rr
-> 10.65.10.12:3128 Masq 1 0 0
-> 10.65.10.13:3128 Masq 1 0 0
[root@keepalived-01 ~]#
Le mie regole di iptables sono:
[root@keepalived-01 ~]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5666 -j ACCEPT
-A INPUT -s 10.2.100.5/32 -j ACCEPT
-A INPUT -d 224.0.0.0/8 -i eth0 -p vrrp -j ACCEPT
-A INPUT -p tcp -m tcp --dport 9100 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3128 -j ACCEPT
-A OUTPUT -d 224.0.0.0/8 -o eth0 -p vrrp -j ACCEPT
[root@keepalived-01 ~]#
ip_forwarding è stato abilitato sulla macchina keepalived.
Il failover VIP funziona bene e il controllo dello stato per i proxy di calamari funziona bene. La configurazione keepalived è:
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< Global definitions >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Router ID = keepalived-01.tekno.veritrans.co.id
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Smtp server connection timeout = 30
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Email notification from = root@keepalived-01.tekno.veritrans.co.id
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: VRRP IPv4 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: VRRP IPv6 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: SNMP Trap disabled
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< SSL definitions >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Using autogen SSL context
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: ------< Global definitions >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< LVS Topology >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: System is compiled with LVS v1.2.1
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: VIP = 10.65.10.101, VPORT = 3128
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Router ID = keepalived-01.tekno.veritrans.co.id
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: delay_loop = 6, lb_algo = rr
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: protocol = TCP
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: alpha is OFF, omega is OFF
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: quorum = 1, hysteresis = 0
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: lb_kind = NAT
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: RIP = 10.65.10.13, RPORT = 3128, WEIGHT = 1
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: RIP = 10.65.10.12, RPORT = 3128, WEIGHT = 1
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< Health checkers >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: [10.65.10.13]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Keepalive method = TCP_CHECK
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Connection dest = [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Connection timeout = 5
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Keepalive method = TCP_CHECK
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Connection dest = [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Connection timeout = 5
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Using LinkWatch kernel netlink reflector...
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Activating healthchecker for service [10.65.10.13]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Smtp server connection timeout = 30
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Activating healthchecker for service [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Email notification from = root@keepalived-01.tekno.veritrans.co.id
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: VRRP IPv4 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: VRRP IPv6 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: SNMP Trap disabled
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: ------< VRRP Topology >------
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: VRRP Instance = squid
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Want State = MASTER
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Runing on device = eth0
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Virtual Router ID = 1
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Priority = 100
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Advert interval = 1sec
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Authentication type = SIMPLE_PASSWORD
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Password = MIdk33pA
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Virtual IP = 1
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: 10.65.10.101/32 dev eth0 scope global
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Using LinkWatch kernel netlink reflector...
Jan 15 13:15:55 keepalived-01 bash: root(ishan): /usr/sbin/keepalived -d
Jan 15 13:15:56 keepalived-01 Keepalived_vrrp[27563]: VRRP_Instance(squid) Transition to MASTER STATE
Il problema che sto affrontando è quando colpisco il VIP, il SYNpacchetto raggiunge il proxy squid e ricevo il SYN.pacchetto. Ma quando invio ., il VIP non lo inoltra ai proxy calamari. Anche tutti i P.pacchetti successivi non vengono trasmessi. Ma quando interrompo la connessione, il Fpacchetto raggiunge il proxy. Sono bloccato da molto tempo ormai. Qualsiasi aiuto sarebbe molto apprezzato.
Prova di connettersi:
[root@ns1 ~]# curl -x http://10.65.10.101:3128 google.com -vvv
* About to connect() to proxy 10.65.10.101 port 3128 (#0)
* Trying 10.65.10.101... connected
* Connected to 10.65.10.101 (10.65.10.101) port 3128 (#0)
> GET http://google.com HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: google.com
> Accept: */*
> Proxy-Connection: Keep-Alive
>
^C
[root@ns1 ~]#
Avvio connessione da: 10.65.10.14
VIP: 10.65.10.101
Proxy: 10.65.10.12
tcpdump su VIP:
13:21:15.718171 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [S], seq 2999891684, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:15.718200 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [S], seq 2999891684, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:15.718451 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:15.718467 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:15.718610 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [.], ack 1, win 229, length 0
13:21:15.718682 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:15.919022 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:16.321066 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:16.918216 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:16.918239 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:16.918488 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [.], ack 1, win 229, length 0
13:21:17.124999 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:18.733058 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:18.918241 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:18.918263 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:18.918491 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [.], ack 1, win 229, length 0
13:21:20.950227 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [F.], seq 212, ack 1, win 229, length 0
13:21:20.950256 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [F.], seq 212, ack 1, win 229, length 0
13:21:20.950541 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [.], ack 1, win 115, options [nop,nop,sack 1 {212:213}], length 0
13:21:20.950562 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [.], ack 1, win 115, options [nop,nop,sack 1 {212:213}], length 0
13:21:21.949007 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:27.416158 IP 10.65.10.14.33300 > 10.65.10.101.squid: Flags [S], seq 3034291567, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:27.416186 IP 10.65.10.14.33300 > 10.65.10.13.squid: Flags [S], seq 3034291567, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
tcpdump su proxy calamari:
13:21:15.718678 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [S], seq 2999891684, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:15.718727 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:16.918423 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:18.918448 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:20.950749 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [F.], seq 212, ack 1, win 229, length 0
13:21:20.950784 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [.], ack 1, win 115, options [nop,nop,sack 1 {212:213}], length 0