RHCS问题汇总-深圳Eric

博主： cto.wang
发布时间：2014 年 02 月 05 日
507次浏览
暂无评论
6675字数
分类：运维专栏

拓补图：

<img src="//cto.wang/usr/uploads/2016/07/20160703184518-3.jpg" align="" title="1465967173689520.jpg" />

服务器用了4个网卡
两个万兆网卡做了bond连到netgear交换机，交换机端口access 30 对应IP段10.199.16.0/22，网关10.199.16.1做在netgear上
两个千兆网卡做了bond连到cisco 3750交换机，交换机端口truck 30 40 1001-1300 对应IP段10.199.16.0/22、10.176.4.0/22、kvm虚拟机内网段，网关10.176.0.4.1做在cisco 3750上
netgear和cisco 3750均做了port-channel
服务器配置：
1.ISCSI多路径配置
defaults {
udev_dir	/dev
polling_interval 10
path_selector	&quot;round-robin 0&quot;
#	path_grouping_policy	multibus
path_grouping_policy	failover
getuid_callout	&quot;/lib/udev/scsi_id &#8211;whitelisted &#8211;device=/dev/%n&quot;
prio	alua
path_checker	readsector0
rr_min_io	100
max_fds	8192
rr_weight	priorities
failback	immediate
no_path_retry	fail
user_friendly_names	yes
}
multipaths {
multipath {
wwid &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 36000d31003157200000000000000000a
alias &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; primary1
}
multipath {
wwid &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 36000d310031572000000000000000003
alias &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; primary2
}
multipath {
wwid &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 36000d31003157200000000000000000b
alias &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; primary3
}
multipath {
wwid &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 36000d31003157200000000000000001b
alias &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; qdisk
}
}
2.网卡配置
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
TYPE=Bond
ONBOOT=yes
BOOTPROTO=none
BRIDGE=cloudbr0
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth4
DEVICE=eth4
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
MASTER=bond1
SLAVE=yes
USERCTL=no
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth5
DEVICE=eth5
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
TYPE=Bond
ONBOOT=yes
BOOTPROTO=none
NAME=bond1
BRIDGE=cloudbr1
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=none
[root@hmkvm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-cloudbr1
DEVICE=cloudbr1
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.199.16.101
NETMASK=255.255.252.0
GATEWAY=10.199.16.1
DNS1=114.114.114.114
[root@hmkvm01 ~]# tail -f -n 5 /etc/modprobe.d/dist.conf
alias char-major-89-* i2c-dev
alias bond0 bonding
options bond0 mode=0 miimon=100
alias bond1 bonding
options bond1 mode=0 miimon=100
现象：
1.有一台服务器出现卡顿现象，从办公网络ping kvm虚拟机会有丢包现象，ping网关无丢包
2.RHCS集群新建后node是正常的，再添加别的机器不能Join Cluster,luci面板报红色错误，cman和clvmd不能运行，
而且只要手动启动cman服务该节点就会进入无限重启的死循环
3.在luci面板修改Expected votes值不生效，手动修改配置文件设成1，当失败节点再Join Cluster时依然失败，Expected votes值又会改变，
指定network模式为UDP Multicast时地址为239开头的IP，在hmkvm01节点能ping通，在令外的节点ping不通，手动指定Multicast addresses不生效
[root@hmkvm01 ~]# cman_tool status
Version: 6.2.0
Config Version: 28
Cluster Name: hmcloud
Cluster Id: 50417
Cluster Member: Yes
Cluster Generation: 992
Membership state: Cluster-Member
Nodes: 3
Expected votes: 7
Quorum device votes: 3
Total votes: 6
Node votes: 1
Quorum: 4 &nbsp;
Active subsystems: 11
Flags:&nbsp;
Ports Bound: 0 11 177 178 &nbsp;
Node name: hmkvm01
Node ID: 1
Multicast addresses: 255.255.255.255&nbsp;
Node addresses: 10.199.16.101&nbsp;

<img src="//cto.wang/usr/uploads/2016/07/20160703184518-33.png" align="" title="1465967318133441.png" />


<img src="//cto.wang/usr/uploads/2016/07/20160703184519-66.png" align="" title="1465967532725848.png" />

4.启动cman服务&nbsp;Waiting for quorum&#8230; Timed-out waiting for cluster，修改network下的模式为UDP Broadcast或在配置文件加cman broadcast=&quot;yes&quot;，Post Join Delay&nbsp;改成600，
手动修改配置文件Expected votes值为1，重启全部服务器，三台服务器状态都正常了，再看配置文件
[root@hmkvm01 ~]# cat /etc/cluster/cluster.conf&nbsp;
&lt;?xml version=&quot;1.0&quot;?&gt;
&lt;cluster config_version=&quot;28&quot; name=&quot;hmcloud&quot;&gt;
	&lt;clusternodes&gt;
		&lt;clusternode name=&quot;hmkvm01&quot; nodeid=&quot;1&quot;&gt;
			&lt;fence&gt;
				&lt;method name=&quot;hmkvm01&quot;&gt;
					&lt;device name=&quot;hmkvm01&quot;/&gt;
				&lt;/method&gt;
			&lt;/fence&gt;
		&lt;/clusternode&gt;
		&lt;clusternode name=&quot;hmkvm02&quot; nodeid=&quot;2&quot;&gt;
			&lt;fence&gt;
				&lt;method name=&quot;hmkvm02&quot;&gt;
					&lt;device name=&quot;hmkvm02&quot;/&gt;
				&lt;/method&gt;
			&lt;/fence&gt;
		&lt;/clusternode&gt;
		&lt;clusternode name=&quot;hmkvm04&quot; nodeid=&quot;3&quot;&gt;
			&lt;fence&gt;
				&lt;method name=&quot;hmkvm04&quot;/&gt;
			&lt;/fence&gt;
		&lt;/clusternode&gt;
		&lt;clusternode name=&quot;pcs1&quot; nodeid=&quot;4&quot;/&gt;
	&lt;/clusternodes&gt;
	&lt;cman broadcast=&quot;yes&quot; expected_votes=&quot;7&quot;/&gt;
	&lt;fence_daemon post_join_delay=&quot;600&quot;/&gt;
	&lt;fencedevices&gt;
		&lt;fencedevice agent=&quot;fence_idrac&quot; ipaddr=&quot;10.199.2.224&quot; login=&quot;root&quot; name=&quot;hmkvm01&quot; passwd=&quot;HMIDC#88878978&quot;/&gt;
		&lt;fencedevice agent=&quot;fence_idrac&quot; ipaddr=&quot;10.199.2.225&quot; login=&quot;root&quot; name=&quot;hmkvm02&quot; passwd=&quot;HMIDC#88878978&quot;/&gt;
		&lt;fencedevice agent=&quot;fence_idrac&quot; ipaddr=&quot;10.199.2.227&quot; login=&quot;root&quot; name=&quot;hmkvm04&quot; passwd=&quot;HMIDC#88878978&quot;/&gt;
	&lt;/fencedevices&gt;
	&lt;quorumd label=&quot;qdisk&quot; min_score=&quot;1&quot;&gt;
		&lt;heuristic interval=&quot;10&quot; program=&quot;ping -c3 -t2 10.199.16.1&quot; tko=&quot;10&quot;/&gt;
	&lt;/quorumd&gt;
	&lt;logging debug=&quot;on&quot;/&gt;
&lt;/cluster&gt;
5.当集群正常后，在某一节点echo c&gt;/proc/sysrq-trigger，当节点重启后必须重复4现象才能正常加入集群。
6.仲裁磁盘qdisk能在每台机器发现，qdisk配置如下
[root@hmkvm01 ~]# mkqdisk -L
mkqdisk v3.0.12.1
/dev/block/253:5:
/dev/disk/by-id/dm-name-qdisk:
/dev/disk/by-id/dm-uuid-mpath-36000d31003157200000000000000001b:
/dev/dm-5:
/dev/mapper/qdisk:
Magic: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;eb7a62c2
Label: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;qdisk
Created: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Mon Jun 13 16:23:05 2016
Host: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; hmkvm01
Kernel Sector Size: &nbsp; 512
Recorded Sector Size: 512

<img src="//cto.wang/usr/uploads/2016/07/20160703184519-58.png" align="" title="1465967519805093.png" />

6.fence设备是正常的
[root@hmkvm01 ~]# &nbsp;fence_idrac -a 10.199.2.227 -l root -p ****** -o status
Status: ON
7.查看日志没有发现特别的地方
<img src="//cto.wang/usr/uploads/2016/07/20160703184519-19.png" border="0" title="1465967502982541.png" />
8.重启网卡，或中断几秒钟，当前节点就会重启
问题如下：
1.我的网卡绑定是否有有需要修改的地方？
2.多路径配置是否有问题？
2.我的集群有没有配置错误？
3.Multicast addresses是各node都能ping通么？
4.network下的红色方框中的IP地址是什么关系？
5.tcpdump抓包并没有发现个节点有跟Multicast addresses通信，这是正常的么？
6.现象8重启的时间在哪里设置？

<img src="//cto.wang/usr/uploads/2016/07/20160703184519-47.png" align="" title="1465967342608207.png" />


<img src="//cto.wang/usr/uploads/2016/07/20160703184519-61.png" align="" title="1465967383475161.png" />

最后修改：2021 年 12 月 10 日 10 : 53 AM

如果觉得我的文章对你有用，请随意赞赏