keepalived 服务配置

Posted on Edited on In ,

keepalived 服务配置

keepalived 起初是专为LVS设计的，专门用来监控LVS集群系统中各个服务节点的状态，后来又加入VRRP的功能，因此除了配合LVS服务外，也可以作为其他服务（nginx，haproxy）的高可用软件。
VRRP（virtual router redundancy protocol）虚拟路由器冗余协议：处理的目的就是为了解决静态路由传销的单点故障问题，
keepalived两大功能：healthcheck & failover 。

主机规划

主机名 IP VIP 功能
node8 192.168.66.18 192.168.66.250(DR) lvs+keepalived(主节点)
node9 192.168.66.19 192.168.66.250(DR) lvs+keepalived(备节点)
node1 192.168.66.11 nginx
node2 192.168.66.12 nginx

安装 lvs 软件

两台机器上都要操作

1
yum install ipvsadm -y

开启内核转发

1
2
3
cat /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/ip_forward

keepalived 安装

两台机器上都要操作

1
yum install curl gcc openssl-devel libnl3-devel net-snmp-devel -y

二进制安装

1
2
3
4
5
6
7
8
9
cd /usr/local/src
wget https://www.keepalived.org/software/keepalived-2.1.5.tar.gz
tar -zxf keepalived-2.1.5.tar.gz
cd keepalived-2.1.5.tar.gz
./configure --prefix=/usr/local/src/keepalived-2.1.5_bin
make && make install
cd /usr/local/src/keepalived-2.1.5_bin
cp sbin/keepalived /usr/sbin/
cp bin/genhash /usr/bin/

实战

添加 keepalived 启动文件

两台机器上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vim /etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D"

添加 keepalived 主配置文件

node8上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
mkdir -p /etc/keepalived/
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     biglittleant@admin.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id node8 #修改为本机名
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER # 主节点
    interface eth1
    virtual_router_id 51
    priority 100 # 要大于备节点的值
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.66.250
    }
}

include keepalived.d/*.conf

备节点配置文件

node9上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
mkdir -p /etc/keepalived/
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     biglittleant@admin.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id node9 #修改为本机名
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP # 备节点
    interface eth1
    virtual_router_id 51
    priority 99 # 要小于主节点的值
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.66.250
    }
}

include keepalived.d/*.conf

添加 virtual_server 配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
mkdir -p /etc/keepalived/keepalived.d
vim /etc/keepalived/keepalived.d/vs-192.168.66.250_80.conf
virtual_server 192.168.66.250 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 192.168.66.11 80 {
        weight 1
    }

    real_server 192.168.66.12 80 {
        weight 2
    }
}

添加 keepalived 启动文件

两台机器上都要操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 添加keepalived 启动文件
vim /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target

[Service]
Type=forking
PIDFile=/var/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

# 启动keepalvied & 加入开机自启动
systemctl daemon-reload
systemctl enable keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
systemctl start keepalived

验证服务启动

node8上操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Sep 15 17:10:34 node8 Keepalived_vrrp[30389]: Using LinkWatch kernel netlink reflector...
Sep 15 17:10:34 node8 Keepalived_vrrp[30389]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Got SIGHUP, reloading checker configuration
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Initializing ipvs
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Opening file 'keepalived.d/vs-192.168.66.250_80.conf'.
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: service [192.168.66.13]:80 no longer exist
Sep 15 17:10:34 node8 Keepalived_healthcheckers[30388]: Gained quorum 1+0=1 <= 2 for VS [192.168.66.250]:80
Sep 15 17:10:34 node8 systemd[1]: Reloaded LVS and VRRP High Availability Monitor.
Sep 15 17:10:35 node8 Keepalived_vrrp[30389]: VRRP_Instance(VI_1) Transition to MASTER STATE

[root@node8 ~]# ip addr list eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:89:7e:4b brd ff:ff:ff:ff:ff:ff
    inet 192.168.66.18/24 brd 192.168.66.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet 192.168.66.250/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe89:7e4b/64 scope link
       valid_lft forever preferred_lft forever

[root@node8 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.66.250:80 wrr persistent 50
  -> 192.168.66.11:80             Route   1      0          0
  -> 192.168.66.12:80             Route   2      0          0

手动关闭主节点，验证虚拟IP自动漂移

node8上操作

1
2
3
4
5
6
7
[root@node8 ~]# systemctl stop keepalived
# 发现主节点lvs配置没了
[root@node8 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@node8 ~]#

node9上操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 通过日志，查看Keepalived主备切换过程
# 期间备节点发送ARP广播，让所有客户端更新本地的ARP表，以便客户端访问新接管VIP服务的节点。

Sep 15 17:33:21 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 15 17:33:21 node9 Keepalived_vrrp[30272]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Sep 15 17:33:21 node9 systemd[1]: Reloaded LVS and VRRP High Availability Monitor.
Sep 15 17:34:48 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) setting protocol iptable drop rule
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:49 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:34:54 node9 Keepalived_vrrp[30272]: Sending gratuitous ARP on eth1 for 192.168.66.250

为什么连续发送5个arp，因为vrrp_garp_master_repeat默认为5；

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@node9 ~]# ip addr list eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:e5:bd:89 brd ff:ff:ff:ff:ff:ff
    inet 192.168.66.19/24 brd 192.168.66.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet 192.168.66.250/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fee5:bd89/64 scope link
       valid_lft forever preferred_lft forever

[root@node9 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.66.250:80 wrr persistent 50
  -> 192.168.66.11:80             Route   1      0          0
  -> 192.168.66.12:80             Route   2      0          0

重启启动主节点，验证是否切回

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 由于主节点优先级高于备节点，因此主节点抢占为MASTER，同时备节点成为BACKUP，并且移除VIP。

Sep 15 17:38:41 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 15 17:38:41 node8 Keepalived_vrrp[1214]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Sep 15 17:38:41 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) forcing a new MASTER election
Sep 15 17:38:42 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) setting protocol iptable drop rule
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:43 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250
Sep 15 17:38:48 node8 Keepalived_vrrp[1214]: Sending gratuitous ARP on eth1 for 192.168.66.250


[root@node8 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.66.250:80 wrr persistent 50
  -> 192.168.66.11:80             Route   1      0          0
  -> 192.168.66.12:80             Route   2      0          0
[root@node8 ~]# ip addr list eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:89:7e:4b brd ff:ff:ff:ff:ff:ff
    inet 192.168.66.18/24 brd 192.168.66.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet 192.168.66.250/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe89:7e4b/64 scope link
       valid_lft forever preferred_lft forever

报错汇总

Unable to load ipset library - libipset.so.11: cannot open shared object file: No such file or directory

yum 安装会报错，查看ipset的包，发现库文件是libipset.so.11, 怀疑是yum版本太旧导致。使用src安装最新版本，未发现此报错。

参考文档

git仓库
官方文档