problem s CARPem a PF

[Marek Soudny]

Ahoj,

resim uz par tydnu neprijemnou vec ohledne carp+pf. Mam dvojici serveru, 
na kazdem bezi haproxy (L7) a relayd (L4) loadbalancery. IPcka si 
predavaji pres carp, routovani a syncovani stavu FW resi pf.

Zacalo se dit, a nevim uz presne kdy, tedy ani proc, ze oba dva nody 
byly carp MASTER pro dany vhid. Coz samozrejme prinasi problemy.

Na backup nodu ted ale pozoruju jeste dalsi neprijemnost. Ma mnohem vic 
pf stavu, nez master. A to me uz dost zarazi, protoze "nad-stavy" jsou 
prave na jednom interfacu, ktery jsem a) zakomentoval v pf a za b) 
stopnu jsem relayd (ktery balancing na dane IP resi) - na backupu.

Uz netusim, kam se podivat, protoze configy by "mely" byt spravne. Jedna 
se o VMware virtualy (kdyz nepojede vmware, stejne nebude co 
balancovat). Nevite prosim vas nekdo, kam se mam podivat, co jsem 
prehlidnul? Ze zksuenosti vim, ze kdyz probelm resite dostatecne dlouho, 
tak uz prehlidnete zakladni drobnosti, kde vetsinou problem je? Na 
backup nodu probiha prepinani MASTER/BACKUP jako na bezicim pasu, 
zatimco "master" node o nicem nevi.

sys-lb-p01 je MASTER, sys-lb-p02 je FAILOVER/BACKUP node:

[root at sys-lb-p01 ~]# freebsd-version -kru
12.1-RELEASE-p3
12.1-RELEASE-p3
12.1-RELEASE-p4

[root at sys-lb-p02 ~]# freebsd-version -kru
12.1-RELEASE-p3
12.1-RELEASE-p3
12.1-RELEASE-p4

[root at sys-lb-p01 ~]# cat /etc/sysctl.conf | grep -v \# | grep .
net.link.ether.inet.log_arp_movements=0
net.inet.carp.preempt=1
net.inet.tcp.tso=0
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

[root at sys-lb-p02 ~]# cat /etc/sysctl.conf | grep -v \# | grep .
net.link.ether.inet.log_arp_movements=0
net.inet.carp.preempt=1
net.inet.tcp.tso=0
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

[root at sys-lb-p01 ~]# ifconfig -a | grep carp
	carp: MASTER vhid 100 advbase 1 advskew 0
	carp: MASTER vhid 101 advbase 1 advskew 0
	carp: MASTER vhid 101 advbase 1 advskew 0
	carp: MASTER vhid 101 advbase 1 advskew 0
	carp: MASTER vhid 101 advbase 1 advskew 0
	carp: MASTER vhid 102 advbase 1 advskew 0
	carp: MASTER vhid 102 advbase 1 advskew 0
	carp: MASTER vhid 102 advbase 1 advskew 0

[root at sys-lb-p02 ~]# ifconfig -a | grep carp
	carp: BACKUP vhid 100 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 102 advbase 1 advskew 200
	carp: BACKUP vhid 102 advbase 1 advskew 200
	carp: BACKUP vhid 102 advbase 1 advskew 200

[root at sys-lb-p01 ~]# pfctl -ss | wc -l
     6735

[root at sys-lb-p02 ~]# ifconfig -a | grep carp
	carp: BACKUP vhid 100 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 101 advbase 1 advskew 200
	carp: BACKUP vhid 102 advbase 1 advskew 200
	carp: BACKUP vhid 102 advbase 1 advskew 200
	carp: BACKUP vhid 102 advbase 1 advskew 200
[root at sys-lb-p02 ~]# pfctl -ss | wc -l
    28947

[root at sys-lb-p01 ~]# grep carp /var/log/messages | tail
Apr 27 09:15:38 sys-lb-p01 kernel: carp: 102 at vmx2.701: MASTER -> BACKUP 
(more frequent advertisement received)
Apr 27 09:15:40 sys-lb-p01 kernel: carp: demoted by -240 to 0 (pfsync 
bulk done)
Apr 27 09:15:40 sys-lb-p01 kernel: carp: 102 at vmx2.701: BACKUP -> MASTER 
(preempting a slower master)
Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.251: BACKUP -> MASTER 
(preempting a slower master)
Apr 27 09:15:41 sys-lb-p01 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
(preempting a slower master)
Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.146: BACKUP -> MASTER 
(preempting a slower master)
Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.162: BACKUP -> MASTER 
(preempting a slower master)
Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.65: BACKUP -> MASTER 
(preempting a slower master)
Apr 27 09:15:41 sys-lb-p01 kernel: carp: 102 at vmx2.190: BACKUP -> MASTER 
(preempting a slower master)
Apr 27 09:15:41 sys-lb-p01 kernel: carp: 102 at vmx2.233: BACKUP -> MASTER 
(preempting a slower master)

[root at sys-lb-p02 ~]# grep carp /var/log/messages | tail
Apr 28 11:21:27 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
(master timed out)
Apr 28 11:21:27 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
(more frequent advertisement received)
Apr 28 11:38:16 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
(master timed out)
Apr 28 11:38:16 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
(more frequent advertisement received)
Apr 28 11:43:18 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
(master timed out)
Apr 28 11:43:18 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
(more frequent advertisement received)
Apr 28 11:53:18 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
(master timed out)
Apr 28 11:53:18 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
(more frequent advertisement received)
Apr 28 12:06:47 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
(master timed out)
Apr 28 12:06:47 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
(more frequent advertisement received)


Diky za jakekoliv nakopnuti,
Marek