最近一家玻璃基板的制造业客户的oracle 19c RAC 节因光纤交换机故障，后节点2重启失败。手动重启crs， 使用crsctl stat stat -t -init 查看资源，停留在ora.storage starting 状态。
查看crs alert log
2026-08-03 10:43:54.212 [ORAROOTAGENT(69300)]CRS-5019: All OCR locations are on ASM disk groups [CRS], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/anbob2/crs/trace/ohasd_orarootagent_root.trc".查看crs/trace/ohasd_orarootagent_root.trc
2026-08-03 10:43:47.692 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] 9790 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2026-08-03 10:43:47.695 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] 9790 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
2026-08-03 10:43:47.719 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] 9508 Error 4 opening dom root in 0x7f854402e330
2026-08-03 10:43:48.462 : USRTHRD:1939851008: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
-- skip
2026-08-03 10:43:54.152 : CLSNS:1927243520: clsns_SetTraceLevel:trace level set to 1.
2026-08-03 10:43:54.178 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] 9790 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2026-08-03 10:43:54.181 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] 9790 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
2026-08-03 10:43:54.205 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] 9508 Error 4 opening dom root in 0x7f854457f7a0
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] -- trace dump on error exit --
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] Error [kgfoAl06] in [kgfokge] at kgfo.c:3182
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] ORA-15077: could not locate ASM instance serving a required diskgroup
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] Category: 7
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] DepInfo: 15077
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} [ora.storage] -- trace dump end --
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} Thread:kgfoCheckMountExt isRunning is reset to false here
2026-08-03 10:43:54.212 : USRTHRD:1927243520: [ INFO] {0:5:3} Thread:kgfoCheckMountExt isFinished set to true
2026-08-03 10:43:54.212 : USRTHRD:1939851008: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
2026-08-03 10:43:54.212 : USRTHRD:1939851008: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask(kgfoCheckMountExt) executed in 7 seconds
2026-08-03 10:43:54.212 :CLSDYNAM:1939851008: [ora.storage]{0:5:3} [start] StorageAgent::parsekgforetcodes retcode = 7, kgfoCheckMount(CRS), flag 2
2026-08-03 10:43:54.212 :CLSDYNAM:1939851008: [ora.storage]{0:5:3} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 15077, other: ORA-15077: could not locate ASM instance serving a required diskgroup
2026-08-03 10:43:54.212 :CLSDYNAM:1939851008: [ora.storage]{0:5:3} [start] StorageAgent::check 260 kgfo returncode 1
2026-08-03 10:43:54.212 :CLSDYNAM:1939851008: [ora.storage]{0:5:3} [start] (:CLSN00140:)StorageAgent::check 300 parsekgforretcodes OCR dgName CRS state 1
2026-08-03 10:43:54.212 :CLSDYNAM:1939851008: [ora.storage]{0:5:3} [start] Storage::start waiting for check to not return PARTIAL or UNPLANNED_OFFLINE 1
2026-08-03 10:43:54.216 :CLSDYNAM:1939851008: [ora.storage]{0:5:3} [start] CssGroup::regis registration of groupName:ASM_NETWORK mbrid:-1 m_grpNum:1 regType:1 succeeded
2026-08-03 10:43:55.219 :CLSDYNAM:1939851008: [ora.storage]{0:5:3} [start] StorageAgent::check 160 NODEROLE_HUB getOCRdetailsASM密码文件可能已损坏或根本不存在。ASM密码在ASM dg中，获取ASM diskgroup失败，从12c 以后FLEX ASM 通信是通过ASM listener的，检查ASM listener是否存在ASM 服务资源。
[grid@node01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE node01 STABLE
ONLINE ONLINE node02 STABLE
ora.chad
ONLINE OFFLINE node01 STABLE
ONLINE ONLINE node02 STABLE
ora.net1.network
ONLINE ONLINE node01 STABLE
ONLINE ONLINE node02 STABLE
ora.ons
ONLINE ONLINE node01 STABLE
ONLINE ONLINE node02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE node01 STABLE
2 ONLINE ONLINE node02 STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE node01 STABLE
2 ONLINE ONLINE node02 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node01 STABLE
ora.OCR.dg(ora.asmgroup)
1 ONLINE ONLINE node01 STABLE
2 ONLINE ONLINE node02 STABLE
3 OFFLINE OFFLINE STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE node01 Started,STABLE
2 ONLINE ONLINE node02 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE node01 STABLE
2 ONLINE ONLINE node02 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE node01 STABLE
ora.node01.vip
1 ONLINE ONLINE node01 STABLE
ora.node02.vip
1 ONLINE ONLINE node02 STABLE
ora.qosmserver
1 ONLINE ONLINE node02 STABLE
ora.scan1.vip
1 ONLINE ONLINE node01 STABLE
ora.testdb.db
1 ONLINE ONLINE node01 Open,HOME=/u01/app/o
racle/product/19.3.0
.0/dbhome_1,STABLE
2 ONLINE ONLINE node02 Open,HOME=/u01/app/o
racle/product/19.3.0
.0/dbhome_1,STABLE
ora.testdb.testdb_ac.svc
1 ONLINE ONLINE node01 STABLE
ora.testdb.testdb_taf.svc
2 ONLINE ONLINE node01 STABLE
--------------------------------------------------------------------------------
[grid@node01 ~]$
[grid@node01 ~]$ crsctl stat res ora.asm -p
NAME=ora.asm(ora.asmgroup)
TYPE=ora.asm.type
ACL=owner:grid:rwx,pgrp:oinstall:r-x,other::r--
ACTIONS=status_target,user:"grid"
...
DESCRIPTION=CRS resource type for the Cluster ASM instance
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
GEN_RESTART=
GEN_USR_ORA_INST_NAME=
GEN_USR_ORA_INST_NAME@SERVERNAME(node01)=+ASM1
GEN_USR_ORA_INST_NAME@SERVERNAME(node02)=+ASM2
SPFILE=
START_CONCURRENCY=0
START_DEPENDENCIES=hard(type:ora.asm_listener.type) weak(ora.ASMNET1LSNR_ASM.lsnr) pullup(type:ora.asm_listener.type)
START_DEPENDENCIES_RTE_INTERNAL=<xml><Arg name="hardasmlsnr" type="ResList"></Arg><Arg name="weakasmlsnr" type="ResList">ora.ASMNET1LSNR_ASM.lsnr</Arg></xml>
START_DEPENDENCIES_TEMPLATE=<xml>hard(type:ora.asm_listener.type,<ResList>{hardasmlsnr}</ResList>) pullup(type:ora.asm_listener.type,<ResList>{hardasmlsnr}</ResList>) weak(<ResList>{weakasmlsnr}</ResList>)</xml>
START_TIMEOUT=900
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(type:ora.asm_listener.type)
STOP_DEPENDENCIES_RTE_INTERNAL=<xml><Arg name="hardasmlsnr" type="ResList"></Arg></xml>
STOP_DEPENDENCIES_TEMPLATE=<xml>hard(type:ora.asm_listener.type,<ResList>intermediate:{hardasmlsnr}</ResList>)</xml>
STOP_TIMEOUT=600
...
[grid@node01 ~]$ crsctl stat res ora.ASMNET1LSNR_ASM.lsnr -p
NAME=ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
TYPE=ora.asm_listener.type
ACL=owner:grid:rwx,pgrp:oinstall:r-x,other::r--
ACTIONS=
ACTION_SCRIPT=%CRS_HOME%/bin/racgwrap%CRS_SCRIPT_SUFFIX%
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALLOW_RESTART=default
AUTO_START=restore
DESCRIPTION=Oracle ASM Listener resource
ENABLED=1
ENDPOINTS=TCP:1525
FAILOVER_DELAY=0
SERVER_POOLS=*
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.asmnet1.asmnetwork) pullup(ora.asmnet1.asmnetwork)
START_DEPENDENCIES_RTE_INTERNAL=<xml><Arg name="asmnetwork" type="ResList">ora.asmnet1.asmnetwork</Arg></xml>
START_DEPENDENCIES_TEMPLATE=<xml>hard(<ResList>{asmnetwork}</ResList>) pullup(<ResList>{asmnetwork}</ResList>)</xml>
START_TIMEOUT=180
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(intermediate:ora.asmnet1.asmnetwork)
STOP_DEPENDENCIES_RTE_INTERNAL=<xml><Arg name="asmnetwork" type="ResList">ora.asmnet1.asmnetwork</Arg></xml>
STOP_DEPENDENCIES_TEMPLATE=<xml>hard(<ResList>intermediate:{asmnetwork}</ResList>)</xml>
STOP_TIMEOUT=0
SUBNET=192.168.0.0
TARGET_DEFAULT=default
TYPE_VERSION=2.2
...
[grid@node01 ~]$ crsctl stat res ora.asmnet1.asmnetwork -p
NAME=ora.asmnet1.asmnetwork(ora.asmgroup)
TYPE=ora.asm_network.type
ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x
ACTIONS=ipsec_setup,user:"grid"
ACTION_SCRIPT=
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=0
ADDRESS_TYPE=IPV4
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
ALLOW_RESTART=default
AUTO_START=restore
CARDINALITY=3
CHECK_INTERVAL=1
CHECK_TIMEOUT=0
CLEAN_TIMEOUT=60
CSS_CRITICAL=no
DELETE_TIMEOUT=60
DEP_TYPE=optional
DESCRIPTION=Oracle ASMNetwork resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=
IGNORE_TARGET_ON_FAILURE=no
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
NLS_LANG=
OFFLINE_CHECK_INTERVAL=60
PING_TARGET=
PLACEMENT=restricted
RELOCATE_BY_DEPENDENCY=1
RELOCATE_KIND=offline
RESOURCE_GROUP=ora.asmgroup
RESTART_ATTEMPTS=0
RESTART_DELAY=0
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=ora.hub.category
SERVER_POOLS=*
START_CONCURRENCY=0
START_DEPENDENCIES=
START_TIMEOUT=0
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=
STOP_TIMEOUT=0
TARGET_DEFAULT=atleastone
TYPE_VERSION=1.1
UPTIME_THRESHOLD=1h
USER_WORKLOAD=no
USE_STICKINESS=0
USR_ORA_ENV=
USR_ORA_IF=
USR_ORA_NETMASK=
USR_ORA_SUBNET=192.168.0.0
WORKLOAD_CPU=0
WORKLOAD_CPU_CAP=0
WORKLOAD_MEMORY_MAX=0
WORKLOAD_MEMORY_TARGET=0
[grid@node01 ~]$ oifcfg getif
ens192 172.20.0.0 global public
ens224 192.168.0.0 global cluster_interconnect,asm
[grid@node01 ~]$ ip addr|grep 192.168
inet 192.168.99.95/16 brd 192.168.255.255 scope global noprefixroute ens224
如果ASM 实例已启动，检查asm listener是否有服务，如果都ASM和ASM Listener都正常，查看或重建asm password. 查看用户
$ asmcmd lspwusr当然在12c 中也存在Bug 28148720 – Resource ‘ora.storage’ Failed to Star With Error ORA-15077 (Doc ID 28148720.8),
本案例是我手动启动ASM 实例恢复正常。
— over —