lvs keepalived实现负载均衡及高可用
时间:2014-08-28 18:59 来源:linux.it.net.cn 作者:it
说明:
LVS全称Linux Virtual Server，即Linux虚拟服务器，是一个虚拟的服务器集群系统。该项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；十种调度算法（rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）
Keepalvied
Keepalived在这里主要用作web-RealServer健康状态检查及lvs-dr-master和lvs-dr-backup之间failover自动切换的实现
环境：
lvs-dr-master 61.164.122.6
lvs-dr-backup 61.164.122.7
LVS-DR-VIP 61.164.122.8
WEB1-Realserver 61.164.122.9
WEB2-Realserver 61.164.122.10
lvs-dr-master，lvs-dr-backup上安装lvs，keepalived软件。lvs-dr-master向局域网内广播61.164.122.8说这个ip是自己，从而实现负载调度，调配到web服务器的请求。(之所以会有2台，是为了当lvs-dr-master挂掉，lvs-dr-backup能够接替lvs-dr-maser继续实现负载调度。此时lvs-dr-slave会接替lvs-dr-master广播61.164.122.8)
WEB1-Realserver,WEB2-Realserver只需要运行realserver.sh，将61.164.122.8这个vip绑定到自己的lo:0上。不能理解为什么要绑定61.164.122.8到lo:0上。
我们向61.164.122.8发出web请求时，其实是lvs-dr-master接收到。他通过keepalived.conf里的配置，再决定将web请求转发给哪台web服务器。
实现：
一、lvs-dr-master，lvs-dr-backup安装LVS和Keepalvied软件包
# mkdir /usr/local/src/lvs
# cd /usr/local/src/lvs
# wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz
# wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz
# lsmod |grep ip_vs //查看是否开启lvs，安装完ipvsadm后可以再看看
# uname -r
2.6.18-53.el5PAE
# ln -s /usr/src/kernels/2.6.18-53.el5PAE-i686/ /usr/src/linux
# tar zxvf ipvsadm-1.24.tar.gz
# cd ipvsadm-1.24
# make && make install
# find / -name ipvsadm # 查看ipvsadm的位置
# tar zxvf keepalived-1.1.15.tar.gz
# cd keepalived-1.1.15
# ./configure && make && make install
# find / -name keepalived # 查看keepalived位置
# cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
# cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
# mkdir /etc/keepalived
# cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
# cp /usr/local/sbin/keepalived /usr/sbin/
# service keepalived start|stop #做成系统启动服务方便管理.
二、web服务器WEB1-Realserver，WEB2-Realserver配置Realserver脚本，将61.164.122.8这个vip绑定到自己的lo:0上。还是没办法理解为什么要绑定61.164.122.8到lo:0上。
# vi /usr/local/sbin/realserver.sh //内容如下
#!/bin/bash
# description: Config realserver lo and apply noarp
# Written by :NetSeek http://www.linuxtone.org
SNS_VIP=192.168.102.127
/etc/rc.d/init.d/functions
case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $SNS_VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
或者采用secondary ip address方式配置
# vi /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
# sysctl -p //写入修改
# ip addr add 61.164.122.8/32 dev lo
# ip add list //查看是否绑定
三、通过lvs实现负载均衡
1．lvs-dr-master，配置LVS脚本实现负载均衡。此脚本只是为了演示，方便大家理解lvs,在keepalived方案中不要启动此脚本。关于LVS的keepalvied的HA方案，完全由keepalived.conf一个文件搞定，不需要开启这个。
# vi /usr/local/sbin/lvs-dr.sh //内容如下
#!/bin/bash
# description: start LVS of DirectorServer
GW=192.168.102.2
# website director vip.
SNS_VIP=192.168.102.127
SNS_RIP1=192.168.102.128
SNS_RIP2=192.168.102.129
/etc/rc.d/init.d/functions
logger $0 called with $1
case "$1" in
start)
# set squid vip
/sbin/ipvsadm --set 30 5 60
/sbin/ifconfig eth0:0 $SNS_VIP broadcast $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP up
/sbin/route add -host $SNS_VIP dev eth0:0
/sbin/ipvsadm -A -t $SNS_VIP:80 -s nq -p 3
/sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP1:80 -g -w 1
/sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP2:80 -g -w 1
touch /var/lock/subsys/ipvsadm >/dev/null 2>&1
;;
stop)
/sbin/ipvsadm -C
/sbin/ipvsadm -Z
ifconfig eth0:0 down
route del $SNS_VIP
rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1
echo "ipvsadm stoped"
;;
status)
if [ ! -e /var/lock/subsys/ipvsadm ];then
echo "ipvsadm stoped"
exit 1
else
echo "ipvsadm OK"
fi
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
esac
exit 0
启动lvs-dr脚本和realserver启本后，在lvs-dr-master上可以查看LVS当前的状态了:
# watch ipvsadm -ln
这时候，如果你测试（web访问61.164.122.8），多访问几次，你会发现它会分别显示192.168.102.128，192.168.102.129的web内容。说明他已经有实现调度了。但如果关闭1台的web话，如这里的129，再多次web访问61.164.122.8就会现在，来自129的web是显示不出来的，而128的都正常。这里，我能理解是lvs只实现负载均衡，并没有办法自动剔除有问题的web服务器？
四、keepalvied实现负载均衡和和高可用性
1.lvs-dr-master主负载均衡服务器上配置keepalived.conf
# vi /etc/keepalived/keepalived.conf //内容如下
! Configuration File for keepalived
global_defs {
notification_email {
cnseek@gmail.com
}
notification_email_from sns-lvs@gmail.com
smtp_server 127.0.0.1
# smtp_connect_timeout 30
router_id LVS_DEVEL
}
# VIP1
vrrp_instance VI_1 {
state MASTER #备份服务器上将MASTER改为BACKUP
interface eth0
virtual_router_id 51
priority 100 # 备份服务上将100改为99。注意Master数值要要大。
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
61.164.122.8
#(如果有多个VIP，继续换行填写.)
}
}
virtual_server 61.164.122.8 80 {
delay_loop 6 #(每隔10秒查询realserver状态)
lb_algo wrr #(lvs 算法)
lb_kind DR #(Direct Route)
persistence_timeout 60 #(同一IP的连接60秒内被分配到同一台realserver)
protocol TCP #(用TCP协议检查realserver状态)
real_server 61.164.122.9 80 {
weight 3 #(权重)
TCP_CHECK {
connect_timeout 10 #(10秒无响应超时)
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 61.164.122.10 80 {
weight 3
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
2. lvs-dr-backup备服务器同上配置，需先安装lvs、keepalived然后配置/etc/keepalived/keepalived.conf，只需修改state BACKUP priority 99这2处即可。
3. 关闭lvs_dr.sh演示，开启keepalived实现负载均衡及高可用
# /usr/local/sbin/lvs-dr.sh stop //停止lvs-dr脚本
# /etc/init.d/keepalived start //启动keepalived 服务，keepalived利用keepalived.conf配置文件实现负载均衡和高可用。整个LVS负均衡HA方案，只需keepalived.conf一个文件即可搞定！
4. 查看lvs服务是否正常
# watch ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddressort Scheduler Flags
-> RemoteAddressort Forward Weight ActiveConn InActConn
TCP 61.164.122.8:80 wrr persistent 60
-> 61.164.122.10:80 Route 3 0 0
-> 61.164.122.9:80 Route 3 0 0
# tail -f /var/log/message 监听日志，查看状态，测试LVS负载均衡及高可用性是否有效。
5．停Master服务器的keepalived服务，查看BAKCUP服务器是否能正常接管服务。
附录：
暂无！
原文参考：
http://bbs.linuxtone.org/thread-1077-1-1.html
http://www.keepalived.org/pdf/sery-lvs-cluster.pdf
(责任编辑：IT)
说明:
Keepalvied 环境： lvs-dr-master 61.164.122.6 lvs-dr-backup 61.164.122.7 LVS-DR-VIP 61.164.122.8 WEB1-Realserver 61.164.122.9 WEB2-Realserver 61.164.122.10
lvs-dr-master，lvs-dr-backup上安装lvs，keepalived软件。lvs-dr-master向局域网内广播61.164.122.8说这个ip是自己，从而实现负载调度，调配到web服务器的请求。(之所以会有2台，是为了当lvs-dr-master挂掉，lvs-dr-backup能够接替lvs-dr-maser继续实现负载调度。此时lvs-dr-slave会接替lvs-dr-master广播61.164.122.8)
实现： # mkdir /usr/local/src/lvs # cd /usr/local/src/lvs # wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz # wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz # lsmod |grep ip_vs //查看是否开启lvs，安装完ipvsadm后可以再看看 # uname -r 2.6.18-53.el5PAE # ln -s /usr/src/kernels/2.6.18-53.el5PAE-i686/ /usr/src/linux # tar zxvf ipvsadm-1.24.tar.gz # cd ipvsadm-1.24 # make && make install # find / -name ipvsadm # 查看ipvsadm的位置 # tar zxvf keepalived-1.1.15.tar.gz # cd keepalived-1.1.15 # ./configure && make && make install # find / -name keepalived # 查看keepalived位置 # cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/ # cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/ # mkdir /etc/keepalived # cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/ # cp /usr/local/sbin/keepalived /usr/sbin/ # service keepalived start|stop #做成系统启动服务方便管理.
#!/bin/bash # description: Config realserver lo and apply noarp # Written by :NetSeek http://www.linuxtone.org SNS_VIP=192.168.102.127 /etc/rc.d/init.d/functions case "$1" in start) ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP /sbin/route add -host $SNS_VIP dev lo:0 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce sysctl -p >/dev/null 2>&1 echo "RealServer Start OK" ;; stop) ifconfig lo:0 down route del $SNS_VIP >/dev/null 2>&1 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce echo "RealServer Stoped" ;; *) echo "Usage: $0 {start|stop}" exit 1 esac exit 0
net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 # sysctl -p //写入修改 # ip addr add 61.164.122.8/32 dev lo # ip add list //查看是否绑定
#!/bin/bash # description: start LVS of DirectorServer GW=192.168.102.2 # website director vip. SNS_VIP=192.168.102.127 SNS_RIP1=192.168.102.128 SNS_RIP2=192.168.102.129 /etc/rc.d/init.d/functions logger $0 called with $1 case "$1" in start) # set squid vip /sbin/ipvsadm --set 30 5 60 /sbin/ifconfig eth0:0 $SNS_VIP broadcast $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP up /sbin/route add -host $SNS_VIP dev eth0:0 /sbin/ipvsadm -A -t $SNS_VIP:80 -s nq -p 3 /sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP1:80 -g -w 1 /sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP2:80 -g -w 1 touch /var/lock/subsys/ipvsadm >/dev/null 2>&1 ;; stop) /sbin/ipvsadm -C /sbin/ipvsadm -Z ifconfig eth0:0 down route del $SNS_VIP rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1 echo "ipvsadm stoped" ;; status) if [ ! -e /var/lock/subsys/ipvsadm ];then echo "ipvsadm stoped" exit 1 else echo "ipvsadm OK" fi ;; *) echo "Usage: $0 {start|stop|status}" exit 1 esac exit 0
启动lvs-dr脚本和realserver启本后，在lvs-dr-master上可以查看LVS当前的状态了:
四、keepalvied实现负载均衡和和高可用性 ! Configuration File for keepalived global_defs { notification_email { cnseek@gmail.com } notification_email_from sns-lvs@gmail.com smtp_server 127.0.0.1 # smtp_connect_timeout 30 router_id LVS_DEVEL } # VIP1 vrrp_instance VI_1 { state MASTER #备份服务器上将MASTER改为BACKUP interface eth0 virtual_router_id 51 priority 100 # 备份服务上将100改为99。注意Master数值要要大。 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 61.164.122.8 #(如果有多个VIP，继续换行填写.) } } virtual_server 61.164.122.8 80 { delay_loop 6 #(每隔10秒查询realserver状态) lb_algo wrr #(lvs 算法) lb_kind DR #(Direct Route) persistence_timeout 60 #(同一IP的连接60秒内被分配到同一台realserver) protocol TCP #(用TCP协议检查realserver状态) real_server 61.164.122.9 80 { weight 3 #(权重) TCP_CHECK { connect_timeout 10 #(10秒无响应超时) nb_get_retry 3 delay_before_retry 3 connect_port 80 } } real_server 61.164.122.10 80 { weight 3 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } }
3. 关闭lvs_dr.sh演示，开启keepalived实现负载均衡及高可用 # /usr/local/sbin/lvs-dr.sh stop //停止lvs-dr脚本 # /etc/init.d/keepalived start //启动keepalived 服务，keepalived利用keepalived.conf配置文件实现负载均衡和高可用。整个LVS负均衡HA方案，只需keepalived.conf一个文件即可搞定！
# watch ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddressort Scheduler Flags -> RemoteAddressort Forward Weight ActiveConn InActConn TCP 61.164.122.8:80 wrr persistent 60 -> 61.164.122.10:80 Route 3 0 0 -> 61.164.122.9:80 Route 3 0 0
# tail -f /var/log/message 监听日志，查看状态，测试LVS负载均衡及高可用性是否有效。
5．停Master服务器的keepalived服务，查看BAKCUP服务器是否能正常接管服务。
附录： 原文参考： http://bbs.linuxtone.org/thread-1077-1-1.html http://www.keepalived.org/pdf/sery-lvs-cluster.pdf (责任编辑：IT) |