首页 » ORACLE » Troubleshooting Oracle RAC a node Fails to Join the Cluster with “no network HB”

Troubleshooting Oracle RAC a node Fails to Join the Cluster with “no network HB”

近日1客户环境的oracle 12cR2 6-nodes RAC多个节点脑裂后无法启动加回cluster, 分析日志又是经典的“has a disk HB, but no network HB“, 最近安全加固需求颇多,当心过度封锁影响到了RAC 间的interconnect 通信。 这里简单记录一下case现象的分析方法。

分析方法:
1, 检查crs状态和资源情况 All nodes

crsctl check crs
crsctl stat res -t
crsctl stat res -t -init

2, 检查问题节点软件环境

 cluvfy stage -post crsinst -n hract21,hract22

3, 检查日志
GI alert log
ocssd.log
crs.log
ASM alert.log
DB alert.log

4, 如果CSSD启动失败,可以开启ocssd debug 日志
#  $GRID_HOME/bin/crsctl set  log css CSSD:3
Set CSSD Module: CSSD  Log Level: 3
#   $GRID_HOME/bin/crsctl get log css CSSD
Get CSSD Module: CSSD  Log Level: 3

5, 检查CSS timeout values
#  $GRID_HOME/bin/crsctl  get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.

6, ocssd log

$ cat $GRID_HOME/log/grac2/cssd/ocssd.log | egrep -i 'Removal|evict|30000|network HB'

alert.log:

2015-02-17 09:42:27.823 [OCSSD(15855)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/hract21/crs/trace/ocssd.trc
2015-02-17 09:42:27.824 [OCSSD(15855)]CRS-1603: CSSD on node hract21 shutdown by user.
2015-02-17 09:42:27.823 [CSSDAGENT(15844)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/grid/diag/crs/hract21/crs/trace/ohasd_cssdagent_root.trc.
Tue Feb 17 09:42:32 2015
Errors in file /u01/app/grid/diag/crs/hract21/crs/trace/ocssd.trc  (incident=2977):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/grid/diag/crs/hract21/crs/incident/incdir_2977/ocssd_i2977.trc

2015-02-17 09:42:33.019 [OCSSD(15855)]CRS-8503: Oracle Clusterware OCSSD process with operating system process ID
 15855 experienced fatal signal or exception code 6
Sweep [inc][2977]: completed
2015-02-17 09:42:38.005 [OHASD(11954)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00163:) {0:0:2} in /u01/app/grid/diag/crs/hract21/crs/trace/ohasd.trc.

ocssd.trc:

2015-02-17 09:42:32.451021 :    CSSD:2417551104: 
   clssnmvDHBValidateNCopy: node 2, hract22, has a disk HB, but no network HB, DHB has rcfg 319544228, wrtcnt, 963949, LATS 92477974, lastSeqNo 963946, uniqueness 1424074596, timestamp 1424162551/21220694
2015-02-17 09:42:32.451113 :    CSSD:2422281984: 
   clssnmvDHBValidateNCopy: node 2, hract22, has a disk HB, but no network HB, DHB has rcfg 319544228, wrtcnt, 963950, LATS 92477974, lastSeqNo 963947, uniqueness 1424074596, timestamp 1424162552/21220904
Trace file /u01/app/grid/diag/crs/hract21/crs/trace/ocssd.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
DDE: Flood control is not active
CLSB:2467473152: Oracle Clusterware infrastructure error in OCSSD (OS PID 15855): Fatal signal 6 has occurred in program ocssd thread 2467473152; nested signal count is 1
Incident 2977 created, dump file: /u01/app/grid/diag/crs/hract21/crs/incident/incdir_2977/ocssd_i2977.trc
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
2015-02-17 09:42:33.108629 :    CSSD:2450904832: clssscWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 1000 with cvtimewait status 4294967186
2015-02-17 09:42:33.451785 :    CSSD:2417551104: clssnmvDHBValidateNCopy: node 2, hract22, has a disk HB, but no network HB, DHB has rcfg 319544228, wrtcnt, 963952, LATS 92478974, lastSeqNo 963949, uniqueness 1424074596, timestamp 1424162552/21221694
2015-02-17 09:42:33.451933 :    CSSD:2422281984: clssnmvDHBValidateNCopy: node 2, hract22, has a disk HB, but no network HB, DHB has rcfg 319544228, wrtcnt, 963953, LATS 92478974, lastSeqNo 963950, uniqueness 1424074596, timestamp 1424162553/21221904
--> Here we know that we have a networking problem

Note:
has a disk HB, but no network HB和OCSSD (OS PID 15855): Fatal signal 6关键字 多是网络不通, (因为现场故障只能截图,所以找了个相同案例日志输出)

7, 检查 gpnp profile.xml

$GRID_HOME/bin/gpnptool get 2>/dev/null | xmllint --format - | egrep 'CSS-Profile|ASM-Profile|Network id'

8, 检查 network

ping     
traceroute     
ifconfig xxx
ip addr

9, 检查 ASM disk

$GRID_HOME/bin/kfod disks=asm  st=true ds=true cluster=true

 

10, 检查OS message

这个故障检查ASM磁盘正常, sqlnet.ora并未配置白名单(可能影响Flex ASM listener通信), OS message日志无硬件错误后, 可以尝试重启crs stack, 如果还是失败,使用traceroute 测试到幸存节点的private IP发现并不通, 结合 no network HB和OCSSD (OS PID 15855): Fatal signal 6关键字初步判断是以下可能:
OS layer: iptables  firewall
Network Layer: 网络防火墙等访问策略

在询问网络工程师确认刚做过网络策略调整, Disabled  firewall especially on the private interconnect. 禁用private interconnect中的所有网络限制后,恢复正常。

打赏

,

目前这篇文章还没有评论(Rss)

我要评论