首页 » ORACLE, ORACLE [C]系列 » Instance Startup Fails With ORA-00205, ORA-15040 After Applying Patch (安装补丁后实例启动失败)

Instance Startup Fails With ORA-00205, ORA-15040 After Applying Patch (安装补丁后实例启动失败)

最近安装一套12.2.0.1.190417 的数据库安装一个补丁时,安装后数据库无法启动,之前安装过相同的补丁在同平台,发现有时sqlplus/srvctl start instance不同的启动方法结果也是不一样的, 这里不得不吐槽ORACLE 补丁的readme中太不严禁, 而且AIX和suse同样的命令结果也是不一样的。

oracle@anbob1:/home/oracle/opatch/28423598> ps -ef|grep ora_
oracle   24687 19655  0 19:23 pts/7    00:00:00 grep --color=auto ora_
oracle@anbob1:/home/oracle/opatch/28423598> opatch apply
Oracle Interim Patch Installer version 12.2.0.1.14
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /oracle/app/oracle/product/12.2.0/db_1
Central Inventory : /oracle/app/oraInventory
   from           : /oracle/app/oracle/product/12.2.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.14
OUI version       : 12.2.0.1.4
Log file location : /oracle/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatch/opatch2019-06-18_19-23-21PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   28423598  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/oracle/app/oracle/product/12.2.0/db_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '28423598' to OH '/oracle/app/oracle/product/12.2.0/db_1'

Patching component oracle.rdbms, 12.2.0.1.0...
Patch 28423598 successfully applied.
Log file location: /oracle/app/oracle/product/12.2.0/db_1/cfgtoollogs/opatch/opatch2019-06-18_19-23-21PM_1.log

OPatch succeeded.
oracle@anbob1:/home/oracle/opatch/28423598> opatch lsinv|grep 28423598
Patch  28423598     : applied on Tue Jun 18 19:23:43 CST 2019

Note: 从上面看出28423598 patch已经成功安装,接下来使用sqlplus启动数据库。

$ sqlplus / as sysdba
SQL> startup nomount;
SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-00205: error in identifying control file, check alert log for more info

启动失败,检查数据库alert log

NOTE: ASMB0 (9831090) connected to ASM instance +ASM1, osid: 3015490 (Flex mode; client id 0x10001)
NOTE: initiating MARK startup
TMON started with pid=69, OS id=3539800
Starting background process MARK
MARK started with pid=70, OS id=4391266
NOTE: MARK has subscribed
ORACLE_BASE from environment = /oracle/app/oracle
2019-06-18 18:20:36.013000 +08:00
ALTER DATABASE   MOUNT
This instance was first to mount
2019-06-18 18:20:37.031000 +08:00
NOTE: ASMB mounting group 2 (DATADG)
NOTE: ASM background process initiating disk discovery for grp 2 (reqid:0)
WARNING: group 2 (DATADG) has missing disks
ORA-15040: diskgroup is incomplete
WARNING: group 2 is being dismounted.
WARNING: ASMB force dismounting group 2 (DATADG) due to missing disks
SUCCESS: diskgroup DATADG was dismounted
NOTE: ASMB mounting group 2 (DATADG)
NOTE: ASM background process initiating disk discovery for grp 2 (reqid:0)
WARNING: group 2 (DATADG) has missing disks
ORA-15040: diskgroup is incomplete
WARNING: group 2 is being dismounted.
WARNING: ASMB force dismounting group 2 (DATADG) due to missing disks
SUCCESS: diskgroup DATADG was dismounted
NOTE: ASMB mounting group 2 (DATADG)
NOTE: ASM background process initiating disk discovery for grp 2 (reqid:0)
WARNING: group 2 (DATADG) has missing disks
ORA-15040: diskgroup is incomplete
WARNING: group 2 is being dismounted.
WARNING: ASMB force dismounting group 2 (DATADG) due to missing disks
SUCCESS: diskgroup DATADG was dismounted
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATADG/STDSVP/CONTROLFILE/current.268.998251971'
ORA-17503: ksfdopn:2 Failed to open file +DATADG/STDSVP/CONTROLFILE/current.268.998251971
ORA-15001: diskgroup "DATADG" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATADG/STDSVP/CONTROLFILE/current.269.998251971'
ORA-17503: ksfdopn:2 Failed to open file +DATADG/STDSVP/CONTROLFILE/current.269.998251971
ORA-15001: diskgroup "DATADG" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATADG/STDSVP/CONTROLFILE/current.270.998251971'
ORA-17503: ksfdopn:2 Failed to open file +DATADG/STDSVP/CONTROLFILE/current.270.998251971
ORA-15001: diskgroup "DATADG" does not exist or is not mounted
ORA-15040: diskgroup is incomplete

Note: 从db alert log检查是+DATADG ASM diskgroup未MOUNT,因为丢失磁盘。 是真的吗?

grid@anbob1:/home/grid> asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304   2097152  2030188                0         2030188              0             N  ARCHDG/
MOUNTED  EXTERN  N         512             512   4096  4194304  12582912  1265528                0         1265528              0             N  DATADG/
MOUNTED  EXTERN  N         512             512   4096  4194304    102400    66164                0           66164              0             N  MGMT/
MOUNTED  NORMAL  N         512             512   4096  4194304     10240     9348             2048            3650              0             Y  OCRDG/


grid@anbob1:/home/grid> kfod disks=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group   
================================================================================
   1:     524288 MB /dev/rhdisk10                            grid     asmadmin
   2:     524288 MB /dev/rhdisk11                            grid     asmadmin
   3:     524288 MB /dev/rhdisk12                            grid     asmadmin
   4:     524288 MB /dev/rhdisk13                            grid     asmadmin
   5:     524288 MB /dev/rhdisk14                            grid     asmadmin
   6:     524288 MB /dev/rhdisk15                            grid     asmadmin
   7:     524288 MB /dev/rhdisk16                            grid     asmadmin
   8:     524288 MB /dev/rhdisk17                            grid     asmadmin
   9:     524288 MB /dev/rhdisk18                            grid     asmadmin
  10:     524288 MB /dev/rhdisk19                            grid     asmadmin
  11:     524288 MB /dev/rhdisk20                            grid     asmadmin
  12:     524288 MB /dev/rhdisk21                            grid     asmadmin
  13:     524288 MB /dev/rhdisk22                            grid     asmadmin
  14:     524288 MB /dev/rhdisk23                            grid     asmadmin
  15:     524288 MB /dev/rhdisk24                            grid     asmadmin
  16:     524288 MB /dev/rhdisk25                            grid     asmadmin
  17:     524288 MB /dev/rhdisk26                            grid     asmadmin
  18:     524288 MB /dev/rhdisk27                            grid     asmadmin
  19:     524288 MB /dev/rhdisk28                            grid     asmadmin
  20:     524288 MB /dev/rhdisk29                            grid     asmadmin
  21:     524288 MB /dev/rhdisk30                            grid     asmadmin
  22:     524288 MB /dev/rhdisk31                            grid     asmadmin
  23:     102400 MB /dev/rhdisk32                            grid     asmadmin
  24:       2048 MB /dev/rhdisk33                            grid     asmadmin
  25:       2048 MB /dev/rhdisk34                            grid     asmadmin
  26:       2048 MB /dev/rhdisk35                            grid     asmadmin
  27:       2048 MB /dev/rhdisk36                            grid     asmadmin
  28:       2048 MB /dev/rhdisk37                            grid     asmadmin
  29:     524288 MB /dev/rhdisk4                             grid     asmadmin
  30:     524288 MB /dev/rhdisk5                             grid     asmadmin
  31:     524288 MB /dev/rhdisk6                             grid     asmadmin
  32:     524288 MB /dev/rhdisk7                             grid     asmadmin
  33:     524288 MB /dev/rhdisk8                             grid     asmadmin
  34:     524288 MB /dev/rhdisk9                             grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME                                                          
================================================================================

oracle@anbob1:/home/oracle> which oracle
/oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle

oracle@anbob1:/home/oracle> ls -l /oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle
-rwsr-s--x    1 oracle   oinstall  639389170 Jun 18 18:07 /oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle

oracle@anbob1:/home/oracle> exit
root@anbob1:/> cd /oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/
root@anbob1:/oracle/app/oracle/product/12.2.0.1/dbhome_1/bin> chgrp asmadmin oracle
root@anbob1:/oracle/app/oracle/product/12.2.0.1/dbhome_1/bin> ls -l oracle
-rwsr-s--x    1 oracle   asmadmin  639389170 Jun 18 18:07 oracle
root@anbob1:/oracle/app/oracle/product/12.2.0.1/dbhome_1/bin> su - oracle
oracle@anbob1:/home/oracle> sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Tue Jun 18 18:47:36 2019

Copyright (c) 1982, 2017, Oracle.  All rights reserved.

Connected.
SQL> shutdown abort
ORACLE instance shut down.
SQL> startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 6.4425E+10 bytes
Fixed Size                 29967376 bytes
Variable Size            3.2078E+10 bytes
Database Buffers         3.2212E+10 bytes
Redo Buffers              104247296 bytes
Database mounted.
Database opened.

Note:
安装补丁后oracle binary 执行文件的属组从asmadmin变成oinstall了,手动修改后恢复正常, 那为什么之前安装没有问题呢?

上面是在AIX上,这次换个SUSE平台试试

root@anbob1:/root> cd /oracle/app/oracle/product/12.2.0/db_1/bin/
root@anbob1:/root> ls -l /oracle/app/oracle/product/12.2.0/db_1/bin/oracle
-rwsr-s--x  1 oracle oinstall 408547136 Jun 19 16:02 /oracle/app/oracle/product/12.2.0/db_1/bin/oracle

root@anbob1:/oracle/app/oracle/product/12.2.0/db_1/bin> chgrp asmadmin oracle
root@anbob1:/oracle/app/oracle/product/12.2.0/db_1/bin> ls -l oracle
-rwsr-s--x  1 oracle asmadmin 408547136 Jun 18 19:23 oracle

-- 同样要检查S粘滞位, 有些OS执行chgrp后可能会丢失S粘滞位
root@anbob1:/oracle/app/oracle/product/12.2.0/db_1/bin> chmod 6751 oracle
root@anbob1:/oracle/app/oracle/product/12.2.0/db_1/bin> ls -l oracle
-rwsr-s--x 1 oracle asmadmin 408547136 Jun 18 19:23 oracle

SUSE平台发现一样的问题。基于对于手动修改,oracle还提供了另一个工具setasmgidwrap 在$GRID_HOME/bin目录下,它可以自动判断OS ASM组,并修改oracle执行文件的属组.

 
oracle@anbob1:/home/oracle> ls -l /oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle
-rwsr-s--x    1 oracle   oinstall  639380520 Jun 20 14:32 /oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle
oracle@anbob1:/home/oracle> env|grep HOME
GI_HOME=/oracle/app/12.2.0.1/grid
HOME=/home/oracle
ORACLE_HOME=/oracle/app/oracle/product/12.2.0.1/dbhome_1
oracle@anbob1:/home/oracle> $GI_HOME/bin/setasmgidwrap o=/oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle
oracle@anbob1:/home/oracle> ls -l /oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle
-rwsr-s--x    1 oracle   asmadmin  639380520 Jun 20 14:32 /oracle/app/oracle/product/12.2.0.1/dbhome_1/bin/oracle

上次为什么安装相同补丁后,没有手动oracle执行修改权限同样也可以启动数据库呢?区别就在启动方式
–SUSE —

oracle@anbob1:/home/oracle/28423598> ls -l /oracle/app/oracle/product/12.2.0/db_1/bin/oracle
-rwsr-s--x 1 oracle oinstall 408547136 Jun 19 16:42 /oracle/app/oracle/product/12.2.0/db_1/bin/oracle
oracle@anbob1:/home/oracle/28423598>  srvctl start instance -d stdorder -i order2 -o mount
oracle@anbob1:/home/oracle/28423598> ls -l /oracle/app/oracle/product/12.2.0/db_1/bin/oracle
-rwsr-s--x 1 oracle asmadmin 408547136 Jun 19 16:42 /oracle/app/oracle/product/12.2.0/db_1/bin/oracle

racle@anbob1:/home/oracle/28423598> ss

SQL*Plus: Release 12.2.0.1.0 Production on Wed Jun 19 16:43:40 2019
Copyright (c) 1982, 2016, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

oracle@anbob1:/home/oracle/28423598> ss
SQL*Plus: Release 12.2.0.1.0 Production on Wed Jun 19 16:44:30 2019
Copyright (c) 1982, 2016, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup mount
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1.9327E+11 bytes
Fixed Size                 29867600 bytes
Variable Size            1.6052E+11 bytes
Database Buffers         3.2212E+10 bytes
Redo Buffers              507002880 bytes
Database mounted.

Note:
用SRVCTL 启动实例,同样有可能会修改oracle文件的属组, 这里我用srvctl 启动一次实例后oracle执行文件从oinstall变成了asmadmin。 再次用sqlplus 启动就不再有问题了。

Conclusion:
安装PSU或安装one-off patch补丁时,在relink编译oracle执行文件后文件的group ownership属组可能会发生改变,对比可以和oracle 执行文件相同路径下的oracle0文件(上一版oracle的mv)查看。 当oracle用户和oracle执行文件都没有OS ASM属组如asmadmin时,会无法读取ASM disk.
解决方法,手动修改或使用setasmgidwrap 修改;使用srvctl 启动实例,让oracle自动修改,但如果srvctl未修改,同样需要手动修改。

 

注意:

使用srvctl start instance 时如果没有修改为正确的权限,很可能是srvctl add database有不正确的地方,我这有一套库发现没有srvctl add database 时没有指定disk group ,所以无法根据ASM DISKGROUP 权限修改oracle group ownership.

打赏

, , ,

对不起,这篇文章暂时关闭评论。