problem s CARPem a PF

[Marek Soudny]

Prave jsem zjistil, ze ackoliv bych mel mit TSO vypnuty 
(/etc/sysctl.conf: net.inet.tcp.tso=0), tak se v optios u vsech network 
interfacu ukazuje jako povoleny. Tak to jsem z toho teda blazen...

pridam "-tso" do /etc/rc.conf k network interfacum, a uvidim, co to udela.

M.

On 28. 04. 20 12:14, Marek Soudny wrote:
> Ahoj,
> 
> resim uz par tydnu neprijemnou vec ohledne carp+pf. Mam dvojici serveru, 
> na kazdem bezi haproxy (L7) a relayd (L4) loadbalancery. IPcka si 
> predavaji pres carp, routovani a syncovani stavu FW resi pf.
> 
> Zacalo se dit, a nevim uz presne kdy, tedy ani proc, ze oba dva nody 
> byly carp MASTER pro dany vhid. Coz samozrejme prinasi problemy.
> 
> Na backup nodu ted ale pozoruju jeste dalsi neprijemnost. Ma mnohem vic 
> pf stavu, nez master. A to me uz dost zarazi, protoze "nad-stavy" jsou 
> prave na jednom interfacu, ktery jsem a) zakomentoval v pf a za b) 
> stopnu jsem relayd (ktery balancing na dane IP resi) - na backupu.
> 
> Uz netusim, kam se podivat, protoze configy by "mely" byt spravne. Jedna 
> se o VMware virtualy (kdyz nepojede vmware, stejne nebude co 
> balancovat). Nevite prosim vas nekdo, kam se mam podivat, co jsem 
> prehlidnul? Ze zksuenosti vim, ze kdyz probelm resite dostatecne dlouho, 
> tak uz prehlidnete zakladni drobnosti, kde vetsinou problem je? Na 
> backup nodu probiha prepinani MASTER/BACKUP jako na bezicim pasu, 
> zatimco "master" node o nicem nevi.
> 
> sys-lb-p01 je MASTER, sys-lb-p02 je FAILOVER/BACKUP node:
> 
> [root at sys-lb-p01 ~]# freebsd-version -kru
> 12.1-RELEASE-p3
> 12.1-RELEASE-p3
> 12.1-RELEASE-p4
> 
> [root at sys-lb-p02 ~]# freebsd-version -kru
> 12.1-RELEASE-p3
> 12.1-RELEASE-p3
> 12.1-RELEASE-p4
> 
> [root at sys-lb-p01 ~]# cat /etc/sysctl.conf | grep -v \# | grep .
> net.link.ether.inet.log_arp_movements=0
> net.inet.carp.preempt=1
> net.inet.tcp.tso=0
> net.inet.ip.forwarding=1
> net.inet6.ip6.forwarding=1
> 
> [root at sys-lb-p02 ~]# cat /etc/sysctl.conf | grep -v \# | grep .
> net.link.ether.inet.log_arp_movements=0
> net.inet.carp.preempt=1
> net.inet.tcp.tso=0
> net.inet.ip.forwarding=1
> net.inet6.ip6.forwarding=1
> 
> [root at sys-lb-p01 ~]# ifconfig -a | grep carp
>      carp: MASTER vhid 100 advbase 1 advskew 0
>      carp: MASTER vhid 101 advbase 1 advskew 0
>      carp: MASTER vhid 101 advbase 1 advskew 0
>      carp: MASTER vhid 101 advbase 1 advskew 0
>      carp: MASTER vhid 101 advbase 1 advskew 0
>      carp: MASTER vhid 102 advbase 1 advskew 0
>      carp: MASTER vhid 102 advbase 1 advskew 0
>      carp: MASTER vhid 102 advbase 1 advskew 0
> 
> [root at sys-lb-p02 ~]# ifconfig -a | grep carp
>      carp: BACKUP vhid 100 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 102 advbase 1 advskew 200
>      carp: BACKUP vhid 102 advbase 1 advskew 200
>      carp: BACKUP vhid 102 advbase 1 advskew 200
> 
> [root at sys-lb-p01 ~]# pfctl -ss | wc -l
>      6735
> 
> [root at sys-lb-p02 ~]# ifconfig -a | grep carp
>      carp: BACKUP vhid 100 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 101 advbase 1 advskew 200
>      carp: BACKUP vhid 102 advbase 1 advskew 200
>      carp: BACKUP vhid 102 advbase 1 advskew 200
>      carp: BACKUP vhid 102 advbase 1 advskew 200
> [root at sys-lb-p02 ~]# pfctl -ss | wc -l
>     28947
> 
> [root at sys-lb-p01 ~]# grep carp /var/log/messages | tail
> Apr 27 09:15:38 sys-lb-p01 kernel: carp: 102 at vmx2.701: MASTER -> BACKUP 
> (more frequent advertisement received)
> Apr 27 09:15:40 sys-lb-p01 kernel: carp: demoted by -240 to 0 (pfsync 
> bulk done)
> Apr 27 09:15:40 sys-lb-p01 kernel: carp: 102 at vmx2.701: BACKUP -> MASTER 
> (preempting a slower master)
> Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.251: BACKUP -> MASTER 
> (preempting a slower master)
> Apr 27 09:15:41 sys-lb-p01 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
> (preempting a slower master)
> Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.146: BACKUP -> MASTER 
> (preempting a slower master)
> Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.162: BACKUP -> MASTER 
> (preempting a slower master)
> Apr 27 09:15:41 sys-lb-p01 kernel: carp: 101 at vmx1.65: BACKUP -> MASTER 
> (preempting a slower master)
> Apr 27 09:15:41 sys-lb-p01 kernel: carp: 102 at vmx2.190: BACKUP -> MASTER 
> (preempting a slower master)
> Apr 27 09:15:41 sys-lb-p01 kernel: carp: 102 at vmx2.233: BACKUP -> MASTER 
> (preempting a slower master)
> 
> [root at sys-lb-p02 ~]# grep carp /var/log/messages | tail
> Apr 28 11:21:27 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
> (master timed out)
> Apr 28 11:21:27 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
> (more frequent advertisement received)
> Apr 28 11:38:16 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
> (master timed out)
> Apr 28 11:38:16 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
> (more frequent advertisement received)
> Apr 28 11:43:18 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
> (master timed out)
> Apr 28 11:43:18 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
> (more frequent advertisement received)
> Apr 28 11:53:18 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
> (master timed out)
> Apr 28 11:53:18 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
> (more frequent advertisement received)
> Apr 28 12:06:47 sys-lb-p02 kernel: carp: 100 at vmx0: BACKUP -> MASTER 
> (master timed out)
> Apr 28 12:06:47 sys-lb-p02 kernel: carp: 100 at vmx0: MASTER -> BACKUP 
> (more frequent advertisement received)
> 
> 
> Diky za jakekoliv nakopnuti,
> Marek