Loading... <p>同学们都很活跃,自己好久没写了,也动动笔,下面文章写着玩的,如有错误请联系(NET7-粤-义薄云天)</p> <p></p> <p style="text-align:center"><span style="font-family:微软雅黑;font-weight:bold;font-size:19px">高可用集群部署文档</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">目录:</span></h2> <p><span style="font-family:微软雅黑;font-size:14px">目录:</span> 1</p> <p><span style="font-family:微软雅黑;font-size:14px">项目需求:</span> 2</p> <p><span style="font-family:微软雅黑;font-size:14px">实现方式:</span> 2</p> <p><span style="font-family:微软雅黑;font-size:14px">拓扑图:</span> 3</p> <p><span style="font-family:微软雅黑;font-size:14px">系统及软件版本:</span> 3</p> <p><span style="font-family:微软雅黑;font-size:14px">安装步骤:</span> 4</p> <p><span style="font-family:微软雅黑;font-size:14px">IP分配:</span> 4</p> <p><span style="font-family:微软雅黑;font-size:14px">LVS和keepalived的安装和配置:</span> 4</p> <p><span style="font-family:微软雅黑;font-size:14px">LVS主配置:</span> 4</p> <p><span style="font-family:微软雅黑;font-size:14px">LVS2备 配置:</span> 7</p> <p><span style="font-family:微软雅黑;font-size:14px">web服务器配置</span> 9</p> <p><span style="font-family:微软雅黑;font-size:14px">Mysql-MHA高可用:</span> 13</p> <p><span style="font-family:微软雅黑;font-size:14px">Mysql主配置:</span> 13</p> <p><span style="font-family:微软雅黑;font-size:14px">manager管理端操作:</span> 15</p> <p><span style="font-family:微软雅黑;font-size:14px">VIP切换:</span> 16</p> <p><span style="font-family:微软雅黑;font-size:14px">测试:</span> 26</p> <p><span style="font-family:微软雅黑;font-size:14px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">项目需求:</span></h2> <p><span style="font-family:微软雅黑;font-size:12px">1.</span><span style="font-family:微软雅黑;font-size:12px">两台WEB/J2EE应用服务器同时提供服务,且负载均衡,当一台服务器当机时,另一台继续提供服务。</span></p> <p><span style="font-family:微软雅黑;font-size:12px">2.</span><span style="font-family:微软雅黑;font-size:12px">.两台MYSQL数据库实时同步,保证数据一致性及完整性。</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">实现方式:</span></h2> <p><span style="font-family:微软雅黑;font-weight:bold;font-size:12px">lvs + keepalived + nginx + tomcat + redis + mysql</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">1.</span><span style="font-family:微软雅黑;font-size:12px">使用lvs+Keepalived集群</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">LVS的通过控制IP来实现负载均衡。IPVS是其具体的实现模块。IPVS的主要作用:安装在Director Server上面,在Director Server虚拟一个对外访问的IP(VIP)。用户访问VIP,到达Director Server,Director Server根据一定的规则选择一个Real Server,处理完成后然后返回给客户端数据。使用lvs+Keepalived集群技术( DR模式 )让两台WEB/J2EE服务器实现负载均衡和高可用。</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">2.</span><span style="font-family:微软雅黑;font-size:12px">redis一主一从,实现会话保持</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">redis主从复制,保证session不丢失,实现会话保持。当用户往Master端写入数据时,通过Redis Sync机制将数据文件发送至Slave,Slave也会执行相同的操作确保数据一致。</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">3.</span><span style="font-family:微软雅黑;font-size:12px">使用MysqlMHA高可用</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">MysqlMHA是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中,MHA能做到0~30秒之内自动完成数据库的故障切换操作,并且在进行故障切换的过程中,MHA能最大程度上保证数据库的一致性,以达到真正意义上的高可用。使用MysqlMHA高可用建立稳定的mysql数据库集群,可以随时切换mysql数据库</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">4.</span><span style="font-family:微软雅黑;font-size:12px">nginx + tomcat实现页面访问</span></p> <p><span style="font-family:微软雅黑;font-size:12px">现公司使用的即是前端nginx +后端 tomcat 访问网站</span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">拓扑图:</span></h2> <p><span style="font-family:微软雅黑;font-size:12px"><img src="//cto.wang/usr/uploads/2016/07/20160703180425-62.png" title="1439201761656328.png" alt="图片1.png" /> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">系统及软件版本:</span></h2> <table width="568" style="width: 522px"> <tbody> <tr class="firstRow"> <td width="142" valign="top" style="padding: 0px 7px;border-width: 1px;border-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">系统</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">CentOS 5.6 X86-64</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">kernel </span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">2.6.18-238.el5</span></p> </td> </tr> <tr> <td width="142" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">LVS</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">1.2.1</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">keepalived</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">1.2.13</span></p> </td> </tr> <tr> <td width="142" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">mysql</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">5.1.73</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">mysqlMHA</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">manager-0.55</span></p> </td> </tr> <tr> <td width="142" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">nginx</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">1.6.2</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">tomcat</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">6.0.44</span></p> </td> </tr> <tr> <td width="142" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">jdk</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">jdk-6u41-linux-amd64</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">redis</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">2.8.19</span></p> </td> </tr> </tbody> </table> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">安装步骤:</span></h2> <p><span style="font-family:微软雅黑;font-size:12px">系统分区及安装:</span></p> <p><span style="font-family:微软雅黑;font-size:12px">最小化安装,并修复bash漏洞,openssl心脏流血漏洞,glibc幽灵漏洞。</span></p> <p><span style="font-family:微软雅黑;font-size:12px">修复系统漏洞略,系统安装略。</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">IP分配:</span></h2> <table width="568" style="width: 522px"> <tbody> <tr class="firstRow"> <td width="142" valign="top" style="padding: 0px 7px;border-width: 1px;border-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">LVS主</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.1</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">LVS备</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.2</span></p> </td> </tr> <tr> <td width="142" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">LVS VIP</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.100</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">MHA VIP</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.80</span></p> </td> </tr> <tr> <td width="142" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">web1</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.3</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">web2</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.4</span></p> </td> </tr> <tr> <td width="142" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">Mysql主</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.5</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">Mysql备</span></p> </td> <td width="142" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.6</span></p> </td> </tr> </tbody> </table> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">LVS和keepalived的安装和配置:</span></h2> <h3><span style="font-family: 微软雅黑;font-size: 14px">LVS主配置:</span></h3> <p><span style="font-family:微软雅黑;font-size:12px">setenforce 0</span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim /etc/sysconfig/selinux</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install -y ipvsadm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">wget </span>http://mirrors.opencas.cn/epel/5/x86_64/epel-release-5-4.noarch.rpm</p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh epel-release-5-4.noarch.rpm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install gcc gcc-c++ vim wget lrzsz ntpdate sysstat dstat wget man tree popt popt-devel popt-static libnl libnl-devel openssl* -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install kernel-devel</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tar -xzf keepalived-1.2.13.tar.gz </span></p> <p><span style="font-family:微软雅黑;font-size:12px">cd keepalived-1.2.13</span></p> <p><span style="font-family:微软雅黑;font-size:12px">./configure –prefix=/usr/local/keepalived –disable-fwmark</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/</span></p> <p><span style="font-family:微软雅黑;font-size:12px">chmod +x /etc/rc.d/init.d/keepalived</span></p> <p><span style="font-family:微软雅黑;font-size:12px">mkdir /etc/keepalived</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp /etc/keepalived/keepalived.conf{,.bak}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sed -i 's/net\.ipv4\.ip\_forward\ \=\ 0/net\.ipv4\.ip\_forward\ \=\ 1/' /etc/sysctl.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">service keepalived restart</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ipvsadm -ln</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"># vim /etc/keepalived/keepalived.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">! Configuration File for keepalived</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">global_defs {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> notification_email {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> sysadmin@firewall.loc</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> notification_email_from Alexandre.Cassen@firewall.loc</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> smtp_server 192.168.200.1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> smtp_connect_timeout 30</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> router_id LVS_DEVEL</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vrrp_sync_group LVS_DEVEL {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> group {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> VI_1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vrrp_instance VI_1 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> state MASTER</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> interface eth0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> lvs_sync_daemon_interface eth0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> garp_master_delay 5</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> virtual_router_id 99</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> priority 151</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> advert_int 1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> authentication {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> auth_type PASS</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> auth_pass 1234567</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> virtual_ipaddress {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 192.168.3.100</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> track_interface {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> eth0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">virtual_server 192.168.3.100 80 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> delay_loop 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> lb_algo wrr</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> lb_kind DR</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> nat_mask 255.255.255.0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># persistence_timeout 3</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> protocol TCP</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> real_server 192.168.3.3 80 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> weight 1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> TCP_CHECK {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_timeout 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> nb_get_retry 3</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> delay_before_retry 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_port 80</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> real_server 192.168.3.4 80 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> weight 1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> TCP_CHECK {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_timeout 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> nb_get_retry 3</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> delay_before_retry 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_port 80</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">service keepalived start</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tail -f /var/log/messages</span></p> <p><span style="font-family:微软雅黑;font-size:12px">LVS1 日志:</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:03 dr1 Keepalived_vrrp[3645]: Opening file '/etc/keepalived/keepalived.conf'.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:03 dr1 Keepalived_vrrp[3645]: Configuration is using : 66961 Bytes</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:03 dr1 Keepalived_vrrp[3645]: Using LinkWatch kernel netlink reflector…</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:03 dr1 kernel: IPVS: sync thread started: state = MASTER, mcast_ifn = eth0, syncid = 99</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:03 dr1 Keepalived_vrrp[3645]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(11,12)]</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:04 dr1 Keepalived_vrrp[3645]: VRRP_Instance(VI_1) Transition to MASTER STATE</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_vrrp[3645]: VRRP_Instance(VI_1) Entering MASTER STATE</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_vrrp[3645]: VRRP_Instance(VI_1) setting protocol VIPs.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_vrrp[3645]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_vrrp[3645]: IPVS: No daemon is running</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_vrrp[3645]: IPVS: Daemon has already run</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_vrrp[3645]: VRRP_Group(LVS_DEVEL) Syncing instances to MASTER state</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_vrrp[3645]: Netlink reflector reports IP 192.168.3.100 added</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 Keepalived_healthcheckers[3644]: Netlink reflector reports IP 192.168.3.100 added</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:05 dr1 avahi-daemon[3312]: Registering new address record for 192.168.3.100 on eth0.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:10 dr1 Keepalived_vrrp[3645]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100</span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables设置</span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A INPUT -p udp -m udp –dport 3478 -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A OUTPUT -p udp -m udp –sport 3478 -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A INPUT -p tcp -m tcp –dport 3478 -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A OUTPUT -p tcp -m tcp –sport 3478 -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A INPUT -p tcp -m tcp –dport 80 -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A OUTPUT -p tcp -m tcp –sport 80 -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -I INPUT -i eth0 -d 224.0.0.0/8 -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A INPUT -i eth0 -p vrrp -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px">iptables -A OUTPUT -o eth0 -p vrrp -j ACCEPT </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h3><span style="font-family: 微软雅黑;font-size: 14px">LVS2备 配置:</span></h3> <p><span style="font-family:微软雅黑;font-size:12px">setenforce 0</span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim /etc/sysconfig/selinux</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install -y ipvsadm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">wget </span>http://mirrors.opencas.cn/epel/5/x86_64/epel-release-5-4.noarch.rpm</p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh epel-release-5-4.noarch.rpm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install gcc gcc-c++ vim wget lrzsz ntpdate sysstat dstat wget man tree popt popt-devel popt-static libnl libnl-devel openssl* -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tar -xzf keepalived-1.2.13.tar.gz </span></p> <p><span style="font-family:微软雅黑;font-size:12px">cd keepalived-1.2.13</span></p> <p><span style="font-family:微软雅黑;font-size:12px">./configure –prefix=/usr/local/keepalived –disable-fwmark</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/</span></p> <p><span style="font-family:微软雅黑;font-size:12px">chmod +x /etc/rc.d/init.d/keepalived</span></p> <p><span style="font-family:微软雅黑;font-size:12px">mkdir /etc/keepalived</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp /etc/keepalived/keepalived.conf{,.bak}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sed -i 's/net\.ipv4\.ip\_forward\ \=\ 0/net\.ipv4\.ip\_forward\ \=\ 1/' /etc/sysctl.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">service keepalived restart</span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim /etc/keepalived/keepalived.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">! Configuration File for keepalived</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">global_defs {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> notification_email {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> sysadmin@firewall.loc</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> notification_email_from Alexandre.Cassen@firewall.loc</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> smtp_server 192.168.200.1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> smtp_connect_timeout 30</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> router_id LVS_DEVEL</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vrrp_sync_group LVS_DEVEL {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> group {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> VI_1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vrrp_instance VI_1 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> state SLAVE</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> interface eth0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> lvs_sync_daemon_interface eth0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> garp_master_delay 5</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> virtual_router_id 99</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> priority 150</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> advert_int 1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> authentication {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> auth_type PASS</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> auth_pass 1234567</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> virtual_ipaddress {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 192.168.3.100</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> track_interface {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> eth0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">virtual_server 192.168.3.100 80 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> delay_loop 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> lb_algo wrr</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> lb_kind DR</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> nat_mask 255.255.255.0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># persistence_timeout 3</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> protocol TCP</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> real_server 192.168.3.3 80 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> weight 1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> TCP_CHECK {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_timeout 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> nb_get_retry 3</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> delay_before_retry 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_port 80</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> real_server 192.168.3.4 80 {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> weight 1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> TCP_CHECK {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_timeout 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> nb_get_retry 3</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> delay_before_retry 2</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> connect_port 80</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">service keepalived start</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tail -f /var/log/messages</span></p> <p><span style="font-family:微软雅黑;font-size:12px">LVS2 日志:</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_healthcheckers[3634]: Opening file '/etc/keepalived/keepalived.conf'.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_healthcheckers[3634]: Configuration is using : 14654 Bytes</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_healthcheckers[3634]: Using LinkWatch kernel netlink reflector…</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_healthcheckers[3634]: Activating healthchecker for service [192.168.3.3]:80</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_healthcheckers[3634]: Activating healthchecker for service [192.168.3.4]:80</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_vrrp[3635]: Opening file '/etc/keepalived/keepalived.conf'.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_vrrp[3635]: Configuration is using : 66959 Bytes</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_vrrp[3635]: Using LinkWatch kernel netlink reflector…</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 kernel: IPVS: sync thread started: state = BACKUP, mcast_ifn = eth0, syncid = 99</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_vrrp[3635]: VRRP_Instance(VI_1) Entering BACKUP STATE</span></p> <p><span style="font-family:微软雅黑;font-size:12px">Jul 7 10:58:23 dr2 Keepalived_vrrp[3635]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(11,12)]</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h3><span style="font-family: 微软雅黑;font-size: 14px">web服务器配置</span></h3> <p><span style="font-family:微软雅黑;font-size:12px">yum install gcc gcc-c++ vim wget lrzsz ntpdate sysstat dstat wget man tree -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">chmod 755 jdk-6u41-linux-x64-rpm.bin</span></p> <p><span style="font-family:微软雅黑;font-size:12px">./jdk-6u41-linux-x64-rpm.bin </span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install openssl openssl-devel</span></p> <p><span style="font-family:微软雅黑;font-size:12px">java -version</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tar -xzf apache-tomcat-6.0.44.tar.gz -C /usr/local/</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cd /usr/local/</span></p> <p><span style="font-family:微软雅黑;font-size:12px">mv apache-tomcat-6.0.44/ tomcat</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tar -zxf pcre-8.35.tar.gz</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cd pcre-8.35</span></p> <p><span style="font-family:微软雅黑;font-size:12px">./configure –prefix=/usr/local/pcre</span></p> <p><span style="font-family:微软雅黑;font-size:12px">make &&make install &&cd ..</span></p> <p><span style="font-family:微软雅黑;font-size:12px">groupadd www</span></p> <p><span style="font-family:微软雅黑;font-size:12px">useradd -g www www -s /sbin/nologin</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tar -xzf nginx-1.6.2.tar.gz</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cd nginx-1.6.2</span></p> <p><span style="font-family:微软雅黑;font-size:12px">./configure –user=www –group=www –prefix=/usr/local/nginx –with-http_stub_status_module –with-http_gzip_static_module –with-pcre=/root/pcre-8.35 –with-http_realip_module</span></p> <p><span style="font-family:微软雅黑;font-size:12px">make -j4</span></p> <p><span style="font-family:微软雅黑;font-size:12px">make install</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cd /usr/local/tomcat/conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp context.xml context.xml.def</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim /usr/local/tomcat/conf/context.xml</span></p> <p><span style="font-family:微软雅黑;font-size:12px"><?xml version='1.0' encoding='utf-8'?></span></p> <p><span style="font-family:微软雅黑;font-size:12px"><Context></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <!– Default set of monitored resources –></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <WatchedResource>WEB-INF/web.xml</WatchedResource></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <Valve className="com.radiadesign.catalina.session.RedisSessionHandlerValve" /></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <Manager className="com.radiadesign.catalina.session.RedisSessionManager"</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> host="192.168.3.3"</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> port="6379" </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> database="0" </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> maxInactiveInterval="60" /></span></p> <p><span style="font-family:微软雅黑;font-size:12px"></Context></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">tar -xzf redis-2.8.19.tar.gz </span></p> <p><span style="font-family:微软雅黑;font-size:12px">make -j4</span></p> <p><span style="font-family:微软雅黑;font-size:12px">make install</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp redis.conf /etc/</span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim /etc/redis.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">/usr/local/bin/redis-server /etc/redis.conf &</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim /usr/local/tomcat/webapps/ROOT/test.jsp</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"><%@ page language="java" %></span></p> <p><span style="font-family:微软雅黑;font-size:12px"><html></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <head><title>TomcatA</title></head></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <body></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <h1><font color="blue">192.168.3.3:8080 Tomcat1 </h1></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <table align="centre" border="1"></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <tr></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <td>Session ID</td></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <% session.setAttribute("abc","abc"); %></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <td><%= session.getId() %></td></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </tr></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <tr></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <td>Created on</td></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> <td><%= session.getCreationTime() %></td></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </tr></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </table></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </body></span></p> <p><span style="font-family:微软雅黑;font-size:12px"></html></span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">/usr/local/tomcat/bin/catalina.sh start</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">grep -v "^.*#" /usr/local/nginx/conf/nginx.conf|sed '/^$/d'</span></p> <p><span style="font-family:微软雅黑;font-size:12px">worker_processes 1;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">events {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> worker_connections 1024;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">http {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> include mime.types;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> default_type application/octet-stream;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> sendfile on;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> keepalive_timeout 65;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> upstream mycluser {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> server 192.168.3.3:8080;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> server {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> listen 80;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> server_name localhost;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> location / {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> root html;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> index index.html index.htm index.jsp;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> proxy_pass http://mycluser;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> proxy_set_header X-Real-IP $remote_addr;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> proxy_set_header Host $host;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> proxy_redirect off;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> error_page 500 502 503 504 /50x.html;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> location = /50x.html {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> root html;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">/usr/local/nginx/sbin/nginx</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">/usr/local/bin/redis-server /etc/redis.conf &</span></p> <p><span style="font-family:微软雅黑;font-size:12px">web2服务器需 vim /etc/redis.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">slaveof 192.168.3.3 6379</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">编辑脚本并启动:</span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim realserver.sh </span></p> <p><span style="font-family:微软雅黑;font-size:12px">#!/bin/bash</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># Script to start LVS DR real server.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># description: LVS DR real server</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#</span></p> <p><span style="font-family:微软雅黑;font-size:12px">. /etc/rc.d/init.d/functions</span></p> <p><span style="font-family:微软雅黑;font-size:12px">VIP=192.168.3.100</span></p> <p><span style="font-family:微软雅黑;font-size:12px">host=`/bin/hostname`</span></p> <p><span style="font-family:微软雅黑;font-size:12px">case "$1" in</span></p> <p><span style="font-family:微软雅黑;font-size:12px">start)</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # Start LVS-DR real server on this machine.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> /sbin/ifconfig lo down</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> /sbin/ifconfig lo up</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> /sbin/route add -host $VIP dev lo:0</span></p> <p><span style="font-family:微软雅黑;font-size:12px">;;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">stop)</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # Stop LVS-DR real server loopback device(s).</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> /sbin/ifconfig lo:0 down</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce</span></p> <p><span style="font-family:微软雅黑;font-size:12px">;;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">status)</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # Status of LVS-DR real server.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> islothere=`/sbin/ifconfig lo:0 | grep $VIP`</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> isrothere=`netstat -rn | grep "lo:0" | grep $VIP`</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if [ ! "$islothere" -o ! "isrothere" ];then</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # Either the route or the lo:0 device</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # not found.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo "LVS-DR real server Stopped."</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> else</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo "LVS-DR real server Running."</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> fi</span></p> <p><span style="font-family:微软雅黑;font-size:12px">;;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">*)</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # Invalid entry.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> echo "$0: Usage: $0 {start|status|stop}"</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> exit 1</span></p> <p><span style="font-family:微软雅黑;font-size:12px">;;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">esac</span></p> <p><span style="font-family:宋体;font-size:14px">iptables<span style="font-family:宋体">设置:</span></span></p> <p><span style="font-family:宋体;font-size:14px">iptables -A INPUT -p udp -m udp –dport 3478 -j ACCEPT </span></p> <p><span style="font-family:宋体;font-size:14px">iptables -A OUTPUT -p udp -m udp –sport 3478 -j ACCEPT </span></p> <p><span style="font-family:宋体;font-size:14px">iptables -A INPUT -p tcp -m tcp –dport 3478 -j ACCEPT </span></p> <p><span style="font-family:宋体;font-size:14px">iptables -A OUTPUT -p tcp -m tcp –sport 3478 -j ACCEPT </span></p> <p><span style="font-family:宋体;font-size:14px">iptables -A INPUT -p tcp -m tcp –dport 80 -j ACCEPT </span></p> <p><span style="font-family:宋体;font-size:14px">iptables -A OUTPUT -p tcp -m tcp –sport 80 -j ACCEPT </span></p> <p><span style="font-family:宋体;font-size:14px"> </span></p> <p><span style="font-family:宋体;font-size:14px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">Mysql-MHA高可用:</span></h2> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <table width="362"> <tbody> <tr class="firstRow"> <td width="114" valign="top" style="padding: 0px 7px;border-width: 1px;border-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">Hostname</span></p> </td> <td width="135" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">Ip</span></p> </td> <td width="113" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-width: 1px;border-top-color: windowtext;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">主机说明</span></p> </td> </tr> <tr> <td width="114" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">manager</span></p> </td> <td width="135" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.7</span></p> </td> <td width="113" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">管理端</span></p> </td> </tr> <tr> <td width="114" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">db1</span></p> </td> <td width="135" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.5</span></p> </td> <td width="113" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">现有主库</span></p> </td> </tr> <tr> <td width="114" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">db2</span></p> </td> <td width="135" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">192.168.3.6</span></p> </td> <td width="113" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">备用主库</span></p> </td> </tr> <tr> <td width="114" valign="top" style="padding: 0px 7px;border-left-width: 1px;border-left-color: windowtext;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"></td> <td width="135" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"> <p><span style="font-family:微软雅黑;font-size:12px">VIP 192.168.3.80</span></p> </td> <td width="113" valign="top" style="padding: 0px 7px;border-left-style: none;border-right-width: 1px;border-right-color: windowtext;border-top-style: none;border-bottom-width: 1px;border-bottom-color: windowtext"></td> </tr> </tbody> </table> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h3><span style="font-family: 微软雅黑;font-size: 14px">Mysql主配置:</span></h3> <p><span style="font-family:微软雅黑;font-size:12px">安装mysql:</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum -y install ncurses*</span></p> <p><span style="font-family:微软雅黑;font-size:12px">/usr/sbin/groupadd mysql</span></p> <p><span style="font-family:微软雅黑;font-size:12px">/usr/sbin/useradd -g mysql mysql</span></p> <p><span style="font-family:微软雅黑;font-size:12px">chown -R mysql.mysql /var/mysql/data</span></p> <p><span style="font-family:微软雅黑;font-size:12px">chown -R mysql.mysql /var/log/mysql</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cd mysql-5.1.73</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum -y install libtool</span></p> <p><span style="font-family:微软雅黑;font-size:12px">./configure –prefix=/usr/local/mysql/ –with-server-suffix=-junsansi-edition –enable-assembler –enable-local-infile –enable-thread-safe-client –with-big-tables –with-charset=utf8 –with-extra-charsets=gbk,gb2312,utf8,ascii –with-readline –with-ssl –with-embedded-server –with-pthread –with-mysqld-user=mysql –with-mysqld-ldflags=-all-static –with-client-ldflags=-all-static –with-plugins=partition,innobase,innodb_plugin</span></p> <p><span style="font-family:微软雅黑;font-size:12px">make</span></p> <p><span style="font-family:微软雅黑;font-size:12px">make install</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp support-files/mysql.server /etc/init.d/mysqld</span></p> <p><span style="font-family:微软雅黑;font-size:12px">chmod 755 /etc/init.d/mysqld</span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim /etc/my.cnf </span></p> <p><span style="font-family:微软雅黑;font-size:12px">[mysqld]</span></p> <p><span style="font-family:微软雅黑;font-size:12px">datadir=/var/mysql/data</span></p> <p><span style="font-family:微软雅黑;font-size:12px">socket=/var/log/mysql/mysql.sock</span></p> <p><span style="font-family:微软雅黑;font-size:12px">user=mysql</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># Default to using old password format for compatibility with mysql 3.x</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># clients (those using the mysqlclient10 compatibility package).</span></p> <p><span style="font-family:微软雅黑;font-size:12px">old_passwords=1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"># Disabling symbolic-links is recommended to prevent assorted security risks;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># to do so, uncomment this line:</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># symbolic-links=0</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">[mysqld_safe]</span></p> <p><span style="font-family:微软雅黑;font-size:12px">log-error=/var/log/mysql/mysqld.log</span></p> <p><span style="font-family:微软雅黑;font-size:12px">pid-file=/var/log/mysql/mysqld.pid</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">/usr/local/mysql/bin/mysql_install_db –user=mysql –datadir=/var/mysql/data/ –basedir=/usr/local/mysql –log-output=file</span></p> <p><span style="font-family:微软雅黑;font-size:12px">service mysqld start</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-weight:bold;font-size:12px">db1现有主库操作:</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-keygen -t rsa</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-copy-id -i ~/.ssh/id_rsa.pub </span><span style="font-family:微软雅黑;font-size:12px">root@</span><span style="font-family:微软雅黑;font-size:12px">192.168.23.7</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-copy-id -i ~/.ssh/id_rsa.pub </span><span style="font-family:微软雅黑;font-size:12px">root@</span><span style="font-family:微软雅黑;font-size:12px">192.168.23.6</span></p> <p><span style="font-family:微软雅黑;font-size:12px">mysql -uroot -p</span></p> <p><span style="font-family:微软雅黑;font-size:12px">grant replication slave on *.* to mharep@'192.168.3.%' identified by 'passwd';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">grant all privileges on *.* to 'mha_manager'@'192.168.3.%' identified by '123456';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">flush privileges;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">\q</span></p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh perl-DBD-MySQL-3.0007-2.el5.x86_64.rpm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh mha4mysql-node-0.54-1.el5.noarch.rpm</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-weight:bold;font-size:12px">db2备用主库操作:</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-keygen -t rsa</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-copy-id -i ~/.ssh/id_rsa.pub </span><span style="font-family:微软雅黑;font-size:12px">root@</span><span style="font-family:微软雅黑;font-size:12px">192.168.23.5</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-copy-id -i ~/.ssh/id_rsa.pub </span><span style="font-family:微软雅黑;font-size:12px">root@</span><span style="font-family:微软雅黑;font-size:12px">192.168.23.7</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#ntpdate 132.163.4.101</span></p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh perl-DBD-MySQL-3.0007-2.el5.x86_64.rpm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh mha4mysql-node-0.54-1.el5.noarch.rpm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">/etc/my.cnf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#character-set-server=utf8 </span></p> <p><span style="font-family:微软雅黑;font-size:12px">server-id=2</span><span style="font-family:微软雅黑;font-size:12px"><br /></span><span style="font-family:微软雅黑;font-size:12px">log-bin=mysql-bin</span><span style="font-family:微软雅黑;font-size:12px"><br /></span><span style="font-family:微软雅黑;font-size:12px">binlog_format=mixed</span></p> <p><span style="font-family:微软雅黑;font-size:12px">relay_log_purge=0 #最好添加此选项</span></p> <p><span style="font-family:微软雅黑;font-size:12px">mysql -uroot -p</span></p> <p><span style="font-family:微软雅黑;font-size:12px">reset master;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">reset slave;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">grant replication slave on *.* to mharep@'192.168.3.%' identified by 'passwd';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">grant all privileges on *.* to 'mha_manager'@'192.168.3.%' identified by '123456';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">flush privileges;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">slave stop;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">change master to MASTER_HOST='192.168.3.5', MASTER_PORT=3306,MASTER_USER='mharep', MASTER_PASSWORD='passwd',master_log_file='</span>mysql-bin.000001<span style="font-family:微软雅黑;font-size:12px">', master_log_pos=</span>107<span style="font-family:微软雅黑;font-size:12px">;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">slave start;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">show slave status\G;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">\q</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h3><span style="font-family: 微软雅黑;font-size: 14px">manager管理端操作:</span></h3> <p><span style="font-family:微软雅黑;font-size:12px">ssh-keygen -t rsa</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-copy-id -i ~/.ssh/id_rsa.pub </span><span style="font-family:微软雅黑;font-size:12px">root@</span><span style="font-family:微软雅黑;font-size:12px">192.168.23.5</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh-copy-id -i ~/.ssh/id_rsa.pub </span><span style="font-family:微软雅黑;font-size:12px">root@</span><span style="font-family:微软雅黑;font-size:12px">192.168.23.6</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ntpdate 132.163.4.101</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install perl cpan -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh perl-Mail-Sender-0.8.16-1.el6.rf.noarch.rpm </span></p> <p><span style="font-family:微软雅黑;font-size:12px">rpm -ivh perl-Mail-Sendmail-0.79-1.2.el6.rf.noarch.rpm</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum localinstall perl-Log-Dispatch-2.26-1.el6.rf.noarch.rpm -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum localinstall perl-Parallel-ForkManager-0.7.5-2.2.el6.rf.noarch.rpm -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum localinstall mha4mysql-node-0.54-1.el5.noarch.rpm -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum localinstall mha4mysql-manager-0.55-1.el5.noarch.rpm -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">yum localinstall perl-Net-Telnet-3.03-2.el6.rfx.noarch.rpm -y</span></p> <p><span style="font-family:微软雅黑;font-size:12px">tar -xzf mha4mysql-manager-0.55.tar.gz</span></p> <p><span style="font-family:微软雅黑;font-size:12px">mkdir -p /usr/local/mha/scripts</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp mha4mysql-manager-0.55/samples/scripts/* /usr/local/mha/scripts</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cat >> /usr/local/mha/mha.conf << EOF</span></p> <p><span style="font-family:微软雅黑;font-size:12px">[server default]</span></p> <p><span style="font-family:微软雅黑;font-size:12px">user=mha_manager </span></p> <p><span style="font-family:微软雅黑;font-size:12px">password=123456</span></p> <p><span style="font-family:微软雅黑;font-size:12px">manager_workdir=/usr/local/mha</span></p> <p><span style="font-family:微软雅黑;font-size:12px">manager_log=/usr/local/mha/manager.log</span></p> <p><span style="font-family:微软雅黑;font-size:12px">remote_workdir=/usr/local/mha</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh_user=root </span></p> <p><span style="font-family:微软雅黑;font-size:12px">repl_user=mharep </span></p> <p><span style="font-family:微软雅黑;font-size:12px">repl_password=passwd</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ping_interval=1</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">#下面是mysql检测设置</span></p> <p><span style="font-family:微软雅黑;font-size:12px">secondary_check_script= masterha_secondary_check -s 192.168.3.5 -s 192.168.3.6</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#master_ip_failover_script=/usr/local/mha/scripts/master_ip_failover</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#shutdown_script= /usr/local/mha/scripts/power_manager</span></p> <p><span style="font-family:微软雅黑;font-size:12px">report_script= /usr/local/mha/scripts/send_report</span></p> <p><span style="font-family:微软雅黑;font-size:12px">master_ip_online_change_script= /usr/local/mha/scripts/master_ip_online_change</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">[server1]</span></p> <p><span style="font-family:微软雅黑;font-size:12px">hostname=192.168.3.5</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh_port=22</span></p> <p><span style="font-family:微软雅黑;font-size:12px">master_binlog_dir=/var/mysql/log #binlog_dir目录必须正确</span></p> <p><span style="font-family:微软雅黑;font-size:12px">candidate_master=1</span></p> <p><span style="font-family:微软雅黑;font-size:12px">[server2]</span></p> <p><span style="font-family:微软雅黑;font-size:12px">hostname=192.168.3.6</span></p> <p><span style="font-family:微软雅黑;font-size:12px">ssh_port=22</span></p> <p><span style="font-family:微软雅黑;font-size:12px">master_binlog_dir=/var/mysql/log</span></p> <p><span style="font-family:微软雅黑;font-size:12px">candidate_master=1</span></p> <p><span style="font-family:微软雅黑;font-size:12px">EOF</span></p> <p><span style="font-family:微软雅黑;font-size:12px">masterha_check_ssh –conf=/usr/local/mha/mha.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">masterha_check_repl –conf=/usr/local/mha/mha.conf </span></p> <p><span style="font-family:微软雅黑;font-size:12px">mv /usr/local/mha/scripts/master_ip_failover /usr/local/mha/scripts/master_ip_failover.def</span></p> <p><span style="font-family:微软雅黑;font-size:12px">cp master_ip_failover /usr/local/mha/scripts/master_ip_failover</span></p> <p><span style="font-family:微软雅黑;font-size:12px">chmod 755 /usr/local/mha/scripts/master_ip_failover</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sed -i 's/#master_ip_failover_script/master_ip_failover_script/' /usr/local/mha/mha.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h3><span style="font-family: 微软雅黑;font-size: 14px">VIP切换:</span></h3> <p><span style="font-family:微软雅黑;font-size:12px">masterha_check_repl –conf=/usr/local/mha/mha.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">nohup masterha_manager –conf=/usr/local/mha/mha.conf > /tmp/mha_manager.log < /dev/null 2>&1 &</span></p> <p><span style="font-family:微软雅黑;font-size:12px">masterha_check_status –conf=/usr/local/mha/mha.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">masterha_stop –conf=/usr/local/mha/mha.conf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">masterha_master_switch –conf=/usr/local/mha/mha.conf –master_state=alive –new_master_host=192.168.23.6 –new_master_port=3306 –orig_master_is_new_slave –running_updates_limit=10000</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim master_ip_online_change</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#!/usr/bin/env perl</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"># Copyright (C) 2011 DeNA Co.,Ltd.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># This program is free software; you can redistribute it and/or modify</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># it under the terms of the GNU General Public License as published by</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># the Free Software Foundation; either version 2 of the License, or</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># (at your option) any later version.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># This program is distributed in the hope that it will be useful,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># but WITHOUT ANY WARRANTY; without even the implied warranty of</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># GNU General Public License for more details.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># You should have received a copy of the GNU General Public License</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># along with this program; if not, write to the Free Software</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># Foundation, Inc.,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">## Note: This is a sample script and is not complete. Modify the script based on your environment.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">use strict;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use warnings FATAL => 'all';</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">use Getopt::Long;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use MHA::DBHelper;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use MHA::NodeUtil;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use Time::HiRes qw( sleep gettimeofday tv_interval );</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use Data::Dumper;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $_tstart;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $_running_interval = 0.1;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my (</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $command, $orig_master_host, $orig_master_ip,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_port, $orig_master_user, </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_host, $new_master_ip, $new_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_user, </span></p> <p><span style="font-family:微软雅黑;font-size:12px">);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $vip = '192.168.3.80/24'; # Virtual IP </span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $key = "1"; </span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $ssh_start_vip = "/sbin/ifconfig eth1:$key $vip";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $ssh_stop_vip = "/sbin/ifconfig eth1:$key down";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $ssh_user = "root";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $new_master_password='123456';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $orig_master_password='123456';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">GetOptions(</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'command=s' => \$command,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #'ssh_user=s' => \$ssh_user, </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'orig_master_host=s' => \$orig_master_host,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'orig_master_ip=s' => \$orig_master_ip,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'orig_master_port=i' => \$orig_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'orig_master_user=s' => \$orig_master_user,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #'orig_master_password=s' => \$orig_master_password,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'new_master_host=s' => \$new_master_host,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'new_master_ip=s' => \$new_master_ip,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'new_master_port=i' => \$new_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> 'new_master_user=s' => \$new_master_user,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #'new_master_password=s' => \$new_master_password,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit &main();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub current_time_us {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my ( $sec, $microsec ) = gettimeofday();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $curdate = localtime($sec);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> return $curdate . " " . sprintf( "%06d", $microsec );</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub sleep_until {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $elapsed = tv_interval($_tstart);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $_running_interval > $elapsed ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> sleep( $_running_interval – $elapsed );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub get_threads_util {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $dbh = shift;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $my_connection_id = shift;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $running_time_threshold = shift;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $type = shift;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $running_time_threshold = 0 unless ($running_time_threshold);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $type = 0 unless ($type);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my @threads;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $sth = $dbh->prepare("SHOW PROCESSLIST");</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $sth->execute();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> while ( my $ref = $sth->fetchrow_hashref() ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $id = $ref->{Id};</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $user = $ref->{User};</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $host = $ref->{Host};</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $command = $ref->{Command};</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $state = $ref->{State};</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $query_time = $ref->{Time};</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $info = $ref->{Info};</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( $my_connection_id == $id );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( defined($query_time) && $query_time < $running_time_threshold );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( defined($command) && $command eq "Binlog Dump" );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( defined($user) && $user eq "system user" );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( defined($command)</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> && $command eq "Sleep"</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> && defined($query_time)</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> && $query_time >= 1 );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $type >= 1 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( defined($command) && $command eq "Sleep" );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( defined($command) && $command eq "Connect" );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $type >= 2 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( defined($info) && $info =~ m/^select/i );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> next if ( defined($info) && $info =~ m/^show/i );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> push @threads, $ref;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> return @threads;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub main {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $command eq "stop" ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Gracefully killing connections on the current master</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # 1. Set read_only= 1 on the new master</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # 2. DROP USER so that no app user can establish new connections</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # 3. Set read_only= 1 on the current master</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # 4. Kill current queries</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # * Any database access failure will result in script die.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $exit_code = 1;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> eval {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Setting read_only=1 on the new master (to avoid accident)</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $new_master_handler = new MHA::DBHelper();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # args: hostname, port, user, password, raise_error(die_on_error)_or_not</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_handler->connect( $new_master_ip, $new_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_user, $new_master_password, 1 );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print current_time_us() . " Set read_only on the new master.. ";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_handler->enable_read_only();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $new_master_handler->is_read_only() ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print "ok.\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> else {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> die "Failed!\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_handler->disconnect();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # Connecting to the orig master, die if any database error happens</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $orig_master_handler = new MHA::DBHelper();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->connect( $orig_master_ip, $orig_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_user, $orig_master_password, 1 );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Drop application user so that nobody can connect. Disabling per-session binlog beforehand</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #$orig_master_handler->disable_log_bin_local();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #print current_time_us() . " Drpping app user on the orig master..\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #FIXME_xxx_drop_app_user($orig_master_handler);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Waiting for N * 100 milliseconds so that current connections can exit</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $time_until_read_only = 15;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $_tstart = [gettimeofday];</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my @threads = get_threads_util( $orig_master_handler->{dbh},</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->{connection_id} );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> while ( $time_until_read_only > 0 && $#threads >= 0 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $time_until_read_only % 5 == 0 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> printf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> current_time_us(), $#threads + 1, $time_until_read_only * 100;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $#threads < 5 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> foreach (@threads);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> sleep_until();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $_tstart = [gettimeofday];</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $time_until_read_only–;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> @threads = get_threads_util( $orig_master_handler->{dbh},</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->{connection_id} );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Setting read_only=1 on the current master so that nobody(except SUPER) can write</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print current_time_us() . " Set read_only=1 on the orig master.. ";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->enable_read_only();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $orig_master_handler->is_read_only() ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print "ok.\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> else {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> die "Failed!\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Waiting for M * 100 milliseconds so that current update queries can complete</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $time_until_kill_threads = 5;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> @threads = get_threads_util( $orig_master_handler->{dbh},</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->{connection_id} );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $time_until_kill_threads % 5 == 0 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> printf</span></p> <p><span style="font-family:微软雅黑;font-size:12px">"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> current_time_us(), $#threads + 1, $time_until_kill_threads * 100;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ( $#threads < 5 ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> foreach (@threads);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> sleep_until();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $_tstart = [gettimeofday];</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $time_until_kill_threads–;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> @threads = get_threads_util( $orig_master_handler->{dbh},</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->{connection_id} );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print "Disabling the VIP on old master: $orig_master_host \n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> &stop_vip(); </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Terminating all threads</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print current_time_us() . " Killing all application threads..\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print current_time_us() . " done.\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #$orig_master_handler->enable_log_bin_local();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $orig_master_handler->disconnect();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $exit_code = 0;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> };</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ($@) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> warn "Got Error: $@\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> elsif ( $command eq "start" ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Activating master ip on the new master</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # 1. Create app user with write privileges</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # 2. Moving backup script if needed</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # 3. Register new master's ip to the catalog database</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"># We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># If exit code is 0 or 10, MHA does not abort</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $exit_code = 10;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> eval {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> my $new_master_handler = new MHA::DBHelper();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # args: hostname, port, user, password, raise_error_or_not</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_handler->connect( $new_master_ip, $new_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_user, $new_master_password, 1 );</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Set read_only=0 on the new master</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #$new_master_handler->disable_log_bin_local();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print current_time_us() . " Set read_only=0 on the new master.\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_handler->disable_read_only();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Creating an app user on the new master</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #print current_time_us() . " Creating app user on the new master..\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #FIXME_xxx_create_app_user($new_master_handler);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> #$new_master_handler->enable_log_bin_local();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $new_master_handler->disconnect();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> ## Update master ip on the catalog database, etc</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print "Enabling the VIP – $vip on the new master – $new_master_host \n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> &start_vip();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> $exit_code = 0;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> };</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> if ($@) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> warn "Got Error: $@\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> elsif ( $command eq "status" ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> # do nothing</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> exit 0;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> else {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> &usage();</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> exit 1;</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> }</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"># A simple system call that enable the VIP on the new master </span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub start_vip() {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># A simple system call that disable the VIP on the old_master</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub stop_vip() {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub usage {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> print</span></p> <p><span style="font-family:微软雅黑;font-size:12px">"Usage: master_ip_online_change –command=start|stop|status –orig_master_host=host –orig_master_ip=ip –orig_master_port=port –new_master_host=host –new_master_ip=ip –new_master_port=port\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> die;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">vim master_ip_failover</span></p> <p><span style="font-family:微软雅黑;font-size:12px">#!/usr/bin/env perl</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use strict;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use warnings FATAL => 'all';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">use Getopt::Long;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my (</span></p> <p><span style="font-family:微软雅黑;font-size:12px">$command, $ssh_user, $orig_master_host, $orig_master_ip,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">$orig_master_port, $new_master_host, $new_master_ip, $new_master_port</span></p> <p><span style="font-family:微软雅黑;font-size:12px">);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $vip = '192.168.23.80'; # Virtual IP</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $gateway = ''; #Gateway IP</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $interface = 'eth1';</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $key = "1";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $ssh_start_vip = "/sbin/ifconfig $interface:$key $vip;/sbin/arping -I $interface -c 3 -s $vip $gateway >/dev/null 2>&1";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $ssh_stop_vip = "/sbin/ifconfig $interface:$key down";</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">GetOptions(</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'command=s' => \$command,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'ssh_user=s' => \$ssh_user,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'orig_master_host=s' => \$orig_master_host,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'orig_master_ip=s' => \$orig_master_ip,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'orig_master_port=i' => \$orig_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'new_master_host=s' => \$new_master_host,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'new_master_ip=s' => \$new_master_ip,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">'new_master_port=i' => \$new_master_port,</span></p> <p><span style="font-family:微软雅黑;font-size:12px">);</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit &main();</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub main {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">if ( $command eq "stop" || $command eq "stopssh" ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># $orig_master_host, $orig_master_ip, $orig_master_port are passed.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># If you manage master ip address at global catalog database,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># invalidate orig_master_ip here.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $exit_code = 1;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">eval {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">print "Disabling the VIP on old master: $orig_master_host \n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">&stop_vip();</span></p> <p><span style="font-family:微软雅黑;font-size:12px">$exit_code = 0;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">};</span></p> <p><span style="font-family:微软雅黑;font-size:12px">if ($@) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">warn "Got Error: $@\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">elsif ( $command eq "start" ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># all arguments are passed.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># If you manage master ip address at global catalog database,</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># activate new_master_ip here.</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># You can also grant write access (create user, set read_only=0, etc) here.</span></p> <p><span style="font-family:微软雅黑;font-size:12px">my $exit_code = 10;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">eval {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">print "Enabling the VIP – $vip on the new master – $new_master_host \n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">&start_vip();</span></p> <p><span style="font-family:微软雅黑;font-size:12px">$exit_code = 0;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">};</span></p> <p><span style="font-family:微软雅黑;font-size:12px">if ($@) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">warn $@;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit $exit_code;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">elsif ( $command eq "status" ) {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">print "Checking the Status of the script.. OK \n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">`ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit 0;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">else {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">&usage();</span></p> <p><span style="font-family:微软雅黑;font-size:12px">exit 1;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># A simple system call that enable the VIP on the new master</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub start_vip() {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"># A simple system call that disable the VIP on the old_master</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub stop_vip() {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px">sub usage {</span></p> <p><span style="font-family:微软雅黑;font-size:12px">print</span></p> <p><span style="font-family:微软雅黑;font-size:12px">"Usage:</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> master_ip_failover –command=start|stop|stopssh|status </span></p> <p><span style="font-family:微软雅黑;font-size:12px">–orig_master_host=host –orig_master_ip=ip –orig_master_port=port </span></p> <p><span style="font-family:微软雅黑;font-size:12px">–new_master_host=host –new_master_ip=ip –new_master_port=port\n";</span></p> <p><span style="font-family:微软雅黑;font-size:12px">}</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <h2><span style="font-family: 微软雅黑;font-size: 16px">测试:</span></h2> <p style="margin-left:28px"><span style="font-family:Wingdings;font-size:12px">l </span><span style="font-family:微软雅黑;font-size:12px">正常访问网站</span></p> <p><span style="font-family:宋体;font-size:14px">web1<span style="font-family:宋体">服务器页面:</span></span></p> <p> <img src="//cto.wang/usr/uploads/2016/07/20160703180425-97.png" title="1439201805408258.png" alt="图片2.png" /></p> <p> </p> <p><span style="font-family:宋体;font-size:14px">web2<span style="font-family:宋体">服务器页面:</span></span></p> <p> <img src="//cto.wang/usr/uploads/2016/07/20160703180425-80.png" title="1439201811672644.png" alt="图片3.png" /></p> <p><span style="font-family:宋体;font-size:14px"> </span></p> <p><span style="font-family:宋体;font-size:14px">lvs + keepalived VIP<span style="font-family:宋体">页面:</span></span></p> <p><span style="font-family:宋体;font-size:14px"> </span></p> <p><span style="font-family:宋体;font-size:14px"> <img src="//cto.wang/usr/uploads/2016/07/20160703180425-42.png" title="1439201818163635.png" alt="图片4.png" /></span></p> <p><span style="font-family:宋体;font-size:14px"> </span></p> <p style="margin-left:28px"><span style="font-family:Wingdings;font-size:12px">l </span><span style="font-family:微软雅黑;font-size:12px">把主数据库关机或关闭网卡</span></p> <p><span style="font-family:微软雅黑;font-size:12px">1,访问页面不受影响,正常访问</span></p> <p><span style="font-family:微软雅黑;font-size:12px">2,数据库数据完整性、一致性、可用性正常</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p style="margin-left:28px"><span style="font-family:Wingdings;font-size:12px">l </span><span style="font-family:微软雅黑;font-size:12px">把web1服务器关机或关闭网卡</span></p> <p><span style="font-family:微软雅黑;font-size:12px">由于web1和web2实现了负载均衡,所以客户打开的页面内容完全一致,访问正常</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p style="margin-left:28px"><span style="font-family:Wingdings;font-size:12px">l </span><span style="font-family:微软雅黑;font-size:12px">把LVS1服务器关机或关闭网卡</span></p> <p><span style="font-family:微软雅黑;font-size:12px">1,由于2台服务器实现前端分发请求的高可用,所以LVS1主机当机,网站仍然正常访问,访问不受影响</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p style="margin-left:28px"><span style="font-family:Wingdings;font-size:12px">l </span><span style="font-family:微软雅黑;font-size:12px">打开网站,登录</span></p> <p><span style="font-family:微软雅黑;font-size:12px">在测试期间,一直刷新页面查看,在LVS1当机的时刻会有少于2秒时间无法访问网站,高可用可以达到 99.99999% ,保证网站访问不因为服务器当机而造成损失。</span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p><span style="font-family:微软雅黑;font-size:12px"> </span></p> <p></p> 最后修改:2021 年 12 月 10 日 10 : 53 AM © 允许规范转载 赞赏 如果觉得我的文章对你有用,请随意赞赏 赞赏作者 支付宝微信