首页 » ORACLE 9i-23c » Troubleshooting Oracle 19c RAC gipcd start fail after a node reboot

Troubleshooting Oracle 19c RAC gipcd start fail after a node reboot

  • 2023/07/29
  • ORACLE 9i-23c
  • 214 views
  • Troubleshooting Oracle 19c RAC gipcd start fail after a node reboot已关闭评论

前不久某银行客户oracle 19c（19.6） RAC，其中一节点计划性操作系统重新启动后，gipcd无法正常启动，导致CRS无法启动的问题。之前曾经记录过2篇Troubleshooting Oracle 19c GIPC start fail with “EXCEPTION[ ret gipcretBadArg (37) ] “和Exadata X7, RAC gipcd 无法启动，因为Network socket files， 这里简单记录.

针对gipcd无法启动的问题，建议进行网络和socket文件的排查分析。参考 MOS
How to Validate Network and Name Resolution Setup for the Clusterware and RAC (Doc ID 1054902.1)
CRS startup failed with socket file permission issue (Doc ID 2489770.1)

现象
2节点ORACLE rac,重启一个节点操作系统后，gipc无法启动，尝试ping和traceroute节点间私网正常， socket文件正常并且重启也问题依旧，OS内核net.ipv4.ipfrag_high_thresh和

net.ipv4.ipfrag_low_thresh已调整过，kill 两节点gipcd.bin进程无法解决. 因为是7*24生产业务，另一存活节点不允许重启。

查检crs stack

# crsctl stat res -t -init

ora.gpnpd
     1        ONLINE  ONLINE       racnode2
ora.gipcd
     1        ONLINE  OFFLINE

Gi alert log

[ohasd(14353)]CRS-2765:Resource 'ora.gipcd' has failed on server 'racnode2'.

gipcd.log

Trace file /u01/app/grid/diag/crs/racnode2/crs/trace/gipcd.trc
Oracle Database 19c Clusterware Release 19.0.0.0.0 - Production
Version 19.6.0.0.0 Copyright 1996, 2019 Oracle. All rights reserved.
 default:622398720: 1: clskec:has:CLSU:910 4 args[clsdAdr_CLSK_err][mod=clsdadr.c][loc=(:CLSD00302:)][msg=2023-07-26 18:49:08.327 (:CLSD00302:) Trace file size and number of segments fetched from environment variable: ORA_DAEMON_TRACE_FILE_OPTIONS filesize=26214400,numsegments=10    Detected in function clsdAdrGetEnvVar_TFO at line number 6819. ]

    CLSB:622398720: [     INFO] Argument count (argc) for this daemon is 1
    CLSB:622398720: [     INFO] Argument 0 is: /u01/app/19.0.0/grid/bin/gipcd.bin
2023-07-26 18:49:08.648 :   GIPCD:622398720:  gipcdMain: gipcd Started
...
2023-07-26 18:49:09.377 : GIPCTLS:412120832:  gipclibTlsGetActiveCipherList: TLS_RSA_WITH_AES_128_CBC_SHA
2023-07-26 18:49:09.438 :GIPCHGEN:4225754880:  gipchaNodeMarkInfAsTransientF [gipchaDaemonCheckInterfaces : gipchaDaemonThread.c : 8560]: marking infs of node 0x7f80fc0c9c90 { host '', haName 'gipcd_ha_name', srcLuid 1c151faa-00000000, dstLuid 00000000-00000000 numInf 0, sentRegister 0, localMonitor 0, baseStream 0x7f80fc0ca510 type gipchaNodeType12001 (20), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 0, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200001} as TRANSIENT
2023-07-26 18:49:09.438 :GIPCHGEN:4225754880:  gipchaNodeAddInterfaceF [gipchaDaemonCheckInterfaces : gipchaDaemonThread.c : 8598]: adding interface info 0x7f80e81b99d0 { host '', haName 'gipcd_ha_name', local (nil), ip '4.1.1.102', subnet '4.1.1.0', mask '255.255.255.0', mac '00-50-56-b7-57-41', ifname 'priv', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }
2023-07-26 18:49:09.438 :GIPCHDEM:4225754880:  gipchaDaemonProcessClientReq: processing req 0x7f80fc12a710 type gipchaClientReqTypePublish (1)
2023-07-26 18:49:09.439 :GIPCHTHR:4227856128:  gipchaWorkerCreateInterface: created local interface for node 'racnode2', haName 'gipcd_ha_name', inf 'udp://4.1.1.102:51543' inf 0x7f80e81b99d0
2023-07-26 18:49:09.439 :GIPCHTHR:4227856128:  gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'racnode2', haName 'gipcd_ha_name', inf 'mcast://224.0.0.251:42424/4.1.1.102' inf 0x7f80e81b99d0
2023-07-26 18:49:09.439 :GIPCHTHR:4227856128:  gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'racnode2', haName 'gipcd_ha_name', inf 'mcast://230.0.1.0:42424/4.1.1.102' inf 0x7f80e81b99d0
2023-07-26 18:49:09.439 :GIPCHTHR:4227856128:  gipchaWorkerCreateInterface: created local bootstrap broadcast interface for node 'racnode2', haName 'gipcd_ha_name', inf 'udp://4.1.1.255:42424' inf 0x7f80e81b99d0
2023-07-26 18:49:09.439 :GIPCHDEM:4225754880:  gipchaDaemonProcessClientReq: processing req 0x7f80f006d7a0 type gipchaClientReqTypeInfPublish (7)
2023-07-26 18:49:09.451 : GIPCTLS:412120832:  gipcmodTlsGetWalletObjFromCred: using rootCredDomName: NULL, baseCredDomName:nodecert
2023-07-26 18:49:09.454 : GIPCTLS:412120832:  gipcmodTlsGetWalletObjFromCred: found base dom: nodecert
2023-07-26 18:49:09.460 : GIPCTLS:412120832:  gipcmodTlsGetWalletObjFromCred: found one certificate 
2023-07-26 18:49:09.477 :    GIPC:412120832:  gipclibTlsTraceCallback: function: nzosReadCertParams

..
2023-07-26 18:49:25.858 :GIPCDNDE:412120832:  gipcdSendReq: Enqueued the request and waiting for connection to complete with host racnode1
2023-07-26 18:49:25.858 :GIPCHDEM:4225754880:  gipchaDaemonProcessClientReq: processing req 0x7f80f0070540 type gipchaClientReqTypeResolve (4)
2023-07-26 18:49:25.858 : GIPCLIB:4225754880:  gipclibCheckProcessAliveness: ospid 3518, timestamp 2953 is ALIVE
2023-07-26 18:49:25.858 :GIPCHGEN:4225754880:  gipchaNodeCreate: adding new node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 0, sentRegister 0, localMonitor 0, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200000}
2023-07-26 18:49:25.858 :GIPCHDEM:4225754880:  gipchaDaemonCreateResolveResponse: creating resolveResponse for host:racnode1, port:gipcdha_racnode1_, haname:gipcd_ha_name, ret:0
2023-07-26 18:49:25.858 :GIPCHDEM:4225754880:  gipchaDaemonProcessClientReq: processing req 0x7f80e82a7990 type gipchaClientReqTypeMonitor (6)
2023-07-26 18:49:25.858 :GIPCHGEN:4227856128:  gipchaNodeAddInterfaceF [gipchaWorkerAddBootstrap : gipchaWorkerThread.c : 2556]: adding interface info 0x7f80f0070540 { host 'racnode1', haName 'gipcd_ha_name', local (nil), ip '4.1.1.102:51543', subnet '4.1.1.0', mask '255.255.255.0', mac '', ifname 'priv', numRef 0, numFail 0, idxBoot 0, flags 0x1842 }
2023-07-26 18:49:25.859 :GIPCHTHR:4227856128:  gipchaWorkerCreateInterface: created remote bootstrap multicast interface for node 'racnode1', haName 'gipcd_ha_name', inf 'mcast://224.0.0.251:42424/4.1.1.102:51543' inf 0x7f80f0070540
2023-07-26 18:49:25.859 :GIPCHTHR:4227856128:  gipchaWorkerCreateInterface: created remote bootstrap multicast interface for node 'racnode1', haName 'gipcd_ha_name', inf 'mcast://230.0.1.0:42424/4.1.1.102:51543' inf 0x7f80f0070540
2023-07-26 18:49:25.859 :GIPCHTHR:4227856128:  gipchaWorkerCreateInterface: created remote bootstrap broadcast interface for node 'racnode1', haName 'gipcd_ha_name', inf 'udp://4.1.1.255:42424' inf 0x7f80f0070540
2023-07-26 18:49:25.859 :GIPCHGEN:4227856128:  gipchaWorkerAttachInterface: Interface attached inf 0x7f80f0070540 { host 'racnode1', haName 'gipcd_ha_name', local 0x7f80e81b99d0, ip '4.1.1.102:51543', subnet '4.1.1.0', mask '255.255.255.0', mac '', ifname 'priv', numRef 0, numFail 0, idxBoot 0, flags 0x1846 }
2023-07-26 18:49:25.859 :GIPCHALO:4227856128:  gipchaLowerSend: deffering startup of hdr 0x7f80f006ce18 { len 264, seq 0, type gipchaHdrTypeSend (1), lastSeq 0, lastAck 0, minAck 0, flags 0x0, srcLuid 00000000-00000000, dstLuid 00000000-00000000, msgId 0 flags2 0x140014 offset 104 srcStreamId 00000000-00000000 dstStreamId 00000000-00000000 priority 0} , node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200000}, stream 0x7f80f0072510 { host 'racnode1', haName 'gipcd_ha_name' srcStreamId 00000000-00000f4c dstStreamId 00000000-00000000 , hendp 0x7f80fc121990 haNode 0x7f80e829e890 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [0 : 0], priority 0,  duplicate recv 0, completed recv 0, completed send 0, total 
2023-07-26 18:49:25.859 :GIPCHAUP:4227856128:  gipchaUpperConnect: initiated connect for umsg 0x7f80f006ccd0 { msg 0x7f80f006ce80, ret gipcretRequestPending (15), flags 0x6 }, msg 0x7f80f006ce80 { type gipchaMsgTypeConnect (3), srcPort '0251-804c-073e-7355', dstPort 'gipcdha_racnode1_', srcCid 00000000-00000f4c, cookie 00007f80-f006ccd0 } dataLen 0, endp 0x7f80fc121990 [0000000000000f4c] { gipchaEndpoint : port '0251-804c-073e-7355', peer ':', srcCid 00000000-00000f4c,  dstCid 00000000-00000000, numSend 0, maxSend 100, groupListType 1, hagroup 0x558cb27ea400, priority 0, forceAckCount 0, usrFlags 0x4000, flags 0x0 } node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200000}
2023-07-26 18:49:25.859 :GIPCHALO:4227856128:  gipchaLowerSendEstablish: sending establish message for node '0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200004}'
2023-07-26 18:49:25.860 :GIPCHALO:4227856128:  gipchaLowerProcessNode: node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200004} now 4294707980 diff 10 connectTime 90000
2023-07-26 18:49:25.860 :GIPCHALO:4227856128:  gipchaLowerProcessNode: node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200004} now 4294707980 diff 10 connectTime 90000
2023-07-26 18:49:26.861 :GIPCHALO:4227856128:  gipchaLowerProcessPendingQ: GIPCHA_LOWER_PENDINGQ_TIMEOUT: now 4294708980, lmsg->startTime 4294707970, lmsg->maxTime 1000, lmsg 0x7f80f0074a80, hdr0x7f80f0074218 { len 1448, seq 1, type gipchaHdrTypeEstablish (4), lastSeq 0, lastAck 0, minAck 1, flags 0x11, srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000, msgId 1 flags2 0x140014 offset 104 srcStreamId 00000000-00000000 dstStreamId 00000000-00000000 priority 0} 
2023-07-26 18:49:26.862 :GIPCHALO:4227856128:  gipchaLowerSendEstablish: sending establish message for node '0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200004}'
2023-07-26 18:49:26.862 :GIPCHALO:4227856128:  gipchaLowerProcessNode: node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200004} now 4294708980 diff 1010 connectTime 90000
2023-07-26 18:49:27.864 :GIPCHALO:4227856128:  gipchaLowerProcessPendingQ: GIPCHA_LOWER_PENDINGQ_TIMEOUT: now 4294709980, lmsg->startTime 4294708980, lmsg->maxTime 1000, lmsg 0x7f80f0074cf0, hdr0x7f80f007f7b8 { len 1448, seq 1, type gipchaHdrTypeEstablish (4), lastSeq 0, lastAck 0, minAck 1, flags 0x11, srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000, msgId 3 flags2 0x140014 offset 104 srcStreamId 00000000-00000000 dstStreamId 00000000-00000000 priority 0} 
2023-07-26 18:49:27.864 :GIPCHALO:4227856128:  gipchaLowerProcessNode: node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200000} now 4294709980 diff 2010 connectTime 90000
2023-07-26 18:49:28.866 :GIPCHALO:4227856128:  gipchaLowerSendEstablish: sending establish message for node '0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200004}'
2023-07-26 18:49:28.866 :GIPCHALO:4227856128:  gipchaLowerProcessNode: node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200004} now 4294710980 diff 3010 connectTime 90000
2023-07-26 18:49:29.868 :GIPCHALO:4227856128:  gipchaLowerProcessPendingQ: GIPCHA_LOWER_PENDINGQ_TIMEOUT: now 4294711980, lmsg->startTime 4294710980, lmsg->maxTime 1000, lmsg 0x7f80f007b2c0, hdr0x7f80f0074868 { len 1448, seq 1, type gipchaHdrTypeEstablish (4), lastSeq 0, lastAck 0, minAck 1, flags 0x11, srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000, msgId 5 flags2 0x140014 offset 104 srcStreamId 00000000-00000000 dstStreamId 00000000-00000000 priority 0} 
2023-07-26 18:49:29.868 :GIPCHALO:4227856128:  gipchaLowerProcessNode: node 0x7f80e829e890 { host 'racnode1', haName 'gipcd_ha_name', srcLuid 1c151faa-1815c3f6, dstLuid 00000000-00000000 numInf 1, sentRegister 1, localMonitor 1, baseStream 0x7f80e81c3340 type gipchaNodeTypeInvalid (0), nodeIncarnation 00000000-00000000, incarnation 0, cssIncarnation 1, negDigest 4294967295, roundTripTime 4294967295 lastSeenPingAck 0 nextPingId 1 latencySrc 0 latencyDst 0 flags 0x200000} now 4294711980 diff 4010 connectTime 90000

OCSSD LOG

2023-07-26 18:49:24.517 :    CLSF:4203093248: Opened hdl:0x565228832020 for dev:AFD:ORA_CRS3:
2023-07-26 18:49:24.517 :    CSSD:4203093248: [     INFO] clssnmvDHBValidateNCopy: node 1, orafirc1, has a disk HB, but no network HB, DHB has rcfg 513455730, wrtcnt, 216948881, LATS 4294706630, lastSeqNo 0, uniqueness 1633687743, timestamp 1690368564/845925376
2023-07-26 18:49:24.517 :    CSSD:4203093248: [     INFO] clssnmvDHBValidateNCopy: node 2, orafirc2, has a disk HB, but no network HB, DHB has rcfg 513455729, wrtcnt, 216833787, LATS 4294706630, lastSeqNo 0, uniqueness 1668702970, timestamp 1690368343/356734826
2023-07-26 18:49:24.518 :    CSSD:4203093248: [     INFO] clssnmvDiskAvailabilityChange: voting file AFD:ORA_CRS1 now online
2023-07-26 18:49:24.518 :    CSSD:4203093248: [     INFO] clssnmvleaseexpired: lease valid for node 2. expirytime 1690973143 currenttime 1690368564 duration 604800 
2023-07-26 18:49:24.518 :    CSSD:4203093248: [     INFO] clssnmlFindLease: orafirc2 owns the lease for slot 2 in file AFD:ORA_CRS1
2023-07-26 18:49:24.518 :    CSSD:4203093248: [     INFO] clssnmvDHBValidateNcopy: Saving DHB uniqueness for node(1/orafirc1), latestInfo(1633687743), readInfo(1633687743), nodeInfoDHB(0)
2023-07-26 18:49:24.518 :    CSSD:4203093248: [     INFO] clssnmvDHBValidateNCopy: node 1, orafirc1, has a disk HB, but no network HB, DHB has rcfg 513455730, wrtcnt, 216948882, LATS 4294706630, lastSeqNo 0, uniqueness 1633687743, timestamp 1690368564/845925606
2023-07-26 18:49:24.518 :    CSSD:4203093248: [     INFO] clssnmvDiskAvailabilityChange: voting file AFD:ORA_CRS2 now online

...

2023-07-26 18:49:55.830 :    CSSD:3507488512: [     INFO] clssnmvDHBValidateNCopy: node 1, racnode1, has a disk HB, but no network HB, DHB has rcfg 513455730, wrtcnt, 216948974, LATS 4294737950, lastSeqNo 216948971, uniqueness 1633687743, timestamp 1690368595/845956426
2023-07-26 18:49:55.833 :    CSSD:3502757632: [     INFO] clssnmvDHBValidateNCopy: node 1, racnode1, has a disk HB, but no network HB, DHB has rcfg 513455730, wrtcnt, 216948975, LATS 4294737950, lastSeqNo 216948972, uniqueness 1633687743, timestamp 1690368595/845956646
2023-07-26 18:49:55.840 :    CSSD:3547309824: [     INFO] clssscWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 1000 with cvtimewait status 4294967186
2023-07-26 18:49:55.842 :    CSSD:3498026752: [     INFO] clssnmvDHBValidateNCopy: node 1, racnode1, has a disk HB, but no network HB, DHB has rcfg 513455730, wrtcnt, 216948976, LATS 4294737960, lastSeqNo 216948973, uniqueness 1633687743, timestamp 1690368595/845956926
2023-07-26 18:49:55.847 :    CSSD:1409283840: [     INFO] clssscSelect: gipcwait returned with status gipcretTimeout (16)
2023-07-26 18:49:55.920 :GIPCGMOD:1409283840:  gipcmodGipcCompleteRequest: [gipc] completing req 0x7fdb44043b30 [0000000000000f70] { gipcConnectRequest : addr 'gipcha://racnode1:nm2_racnode-cluster', parentEndp 0x7fdb440422f0, ret gipcretTimeout (16), objFlags 0x0, reqFlags 0x2 }
2023-07-26 18:49:55.920 :GIPCGMOD:1409283840:  gipcmodGipcCompleteConnect: [gipc] completed connect on endp 0x7fdb440422f0 [0000000000000f0c] { gipcEndpoint : localAddr 'gipcha://racnode2:513d-c6bd-6b3e-6151', remoteAddr 'gipcha://racnode1:nm2_racnode-cluster', numPend 1, numReady 1, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0x5652286365f0, ready 0, wobj 0x7fdb440390c0, sendp (nil) status 13flags 0x200b8602, flags-2 0x10, usrFlags 0x0 }
2023-07-26 18:49:55.920 : GIPCTLS:1409283840:  gipcmodTlsDisconnect: [tls] disconnect issued on endp 0x7fdb440422f0 [0000000000000f0c] { gipcEndpoint : localAddr 'gipcha://racnode2:513d-c6bd-6b3e-6151', remoteAddr 'gipcha://racnode1:nm2_racnode-cluster', numPend 1, numReady 0, numDone 1, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0x5652286365f0, ready 1, wobj 0x7fdb440390c0, sendp (nil) status 13flags 0x200b8602, flags-2 0x10, usrFlags 0x0 }
2023-07-26 18:49:55.920 :GIPCGMOD:1409283840:  gipcmodGipcDisconnect: [gipc]  Issued endpoint close for endp 0x7fdb440422f0 [0000000000000f0c] { gipcEndpoint : localAddr 'gipcha://racnode2:513d-c6bd-6b3e-6151', remoteAddr 'gipcha://racnode1:nm2_racnode-cluster', numPend 1, numReady 0, numDone 1, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0x5652286365f0, ready 1, wobj 0x7fdb440390c0, sendp (nil) status 13flags 0x200b8602, flags-2 0x10, usrFlags 0x0 }
2023-07-26 18:49:55.920 :    CSSD:1409283840: [     INFO] clssscSelect: conn complete ctx 0x565228637a10 endp 0xf0c

mcasttest多播测试

An Example of Successful output is:

[user@node1]$ ./mcasttest.pl -n node1,node2 -i eth0,eth1

多播测试不通。

解决方法
最终找网络部门，检查网络，调整网络策略后，CRS启动正常， 建议对于ORACLE RAC的internnect network节点间不要增加网络限制，仅做私网心跳对外不提供访问，理论上是安全的。 甚至有做网络限制后CRS当时并不会立即受影响，但下次重启时，可能会出现因网络变更导致RAC无法启动。

打赏

,

对不起，这篇文章暂时关闭评论。

上一篇：

下一篇：