首页 » ORACLE 9i-23c » Troubleshooting Instance crash ORA-01110 ORA-01208 Due to ORA-63999, Disk Failure

Troubleshooting Instance crash ORA-01110 ORA-01208 Due to ORA-63999, Disk Failure

  • 2021/11/25
  • ORACLE 9i-23c
  • 534 views
  • Troubleshooting Instance crash ORA-01110 ORA-01208 Due to ORA-63999, Disk Failure已关闭评论

最近一起Oracle数据库自动crash, 再次打开是提示需要做datafile recover, 分析alert日志提示ORA-63999 ORA-01122 ORA-01110 ORA-01208， 问题时logfileswitch并不频繁，当时也没有做duplicat 操作(bug 13498382)，ora-1208是因为数据文件头上的ckpt比控制文件上的旧，reover后随后数据文件头被更新，因此可以正常访问而不会发生损坏,判断是IO写丢失导致. 仅记录

2021-11-22 22:25:10.543000 +08:00
Archived Log entry 3751 added for thread 1 sequence 3767 ID 0x83ca5242 dest 1:
ALTER SYSTEM ARCHIVE LOG
Thread 1 advanced to log sequence 3769 (LGWR switch)
  Current log# 1 seq# 3769 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_1_gv0my4b8_.log
Archived Log entry 3752 added for thread 1 sequence 3768 ID 0x83ca5242 dest 1:
2021-11-22 22:26:15.984000 +08:00
ALTER SYSTEM ARCHIVE LOG
Thread 1 advanced to log sequence 3770 (LGWR switch)
  Current log# 2 seq# 3770 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_2_gv0mygsx_.log
Archived Log entry 3753 added for thread 1 sequence 3769 ID 0x83ca5242 dest 1:
2021-11-23 02:00:00.004000 +08:00
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
2021-11-23 02:00:59.344000 +08:00
Thread 1 advanced to log sequence 3771 (LGWR switch)
  Current log# 3 seq# 3771 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_3_gv0mys8m_.log
2021-11-23 02:01:06.292000 +08:00
Archived Log entry 3754 added for thread 1 sequence 3770 ID 0x83ca5242 dest 1:
2021-11-23 03:11:52.148000 +08:00
datafile header validation failure for file /dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf
2021-11-23 03:14:00.276000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
Checker run found 1 new persistent data failures
2021-11-23 03:47:05.194000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
2021-11-23 04:21:58.388000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
2021-11-23 19:30:41.151000 +08:00
Thread 1 advanced to log sequence 3772 (LGWR switch)
  Current log# 4 seq# 3772 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_4_gv0n7srg_.log
2021-11-23 19:30:47.133000 +08:00
Archived Log entry 3755 added for thread 1 sequence 3771 ID 0x83ca5242 dest 1:
2021-11-23 19:35:42.016000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
Errors in file /u01/oracle/diag/rdbms/anbob/anbob/trace/anbob_ckpt_10232.trc:
ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
Errors in file /u01/oracle/diag/rdbms/anbob/anbob/trace/anbob_ckpt_10232.trc:
ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
CKPT (ospid: 10232): terminating the instance due to error 63999        《《《《《《《《《《《《
System state dump requested by (instance=1, osid=10232 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /u01/oracle/diag/rdbms/anbob/anbob/trace/anbob_diag_10216_20211123193542.trc
2021-11-23 19:35:47.118000 +08:00
Instance terminated by CKPT, pid = 10232

提示：
从数据库告警日志同时间可以看到从3:14开始在读取数据文件#22号’/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf’ (fno 22)时，做数据库一致性校验时失败，触发ORA-01208错误，读到了一个老版本的数据. 一直持续到2021-11-23 19:35 数据库的核心后台进程CKPT做检查点时因为相同的问题直接终止了数据库实例。 此日10：04数据库打开，所以提有启动时提示要做recover，前滚数据文件一致性后才能OPEN。

ckpt trace file

*** 2021-11-23 19:35:42.028
Rereading datafile 22 header failed with ORA-01208
 V10 STYLE FILE HEADER:
        Compatibility Vsn = 186647552=0xb200400
        Db ID=2211061570=0x83ca1f42, Db Name='ANBOB'
        Activation ID=0=0x0
        Control Seq=810175=0xc5cbf, File size=3932160=0x3c0000
        File Number=22, Blksiz=8192, File Type=3 DATA
Tablespace #6 - HFMIS_ZHFW  rel_fn:22
Creation   at   scn: 0x0000.e0672b45 08/21/2021 14:35:35
Backup taken at scn: 0x0000.00000000 01/01/1988 00:00:00 thread:0
 reset logs count:0x3cf14dc4 scn: 0x0000.000e2006
 prev reset logs count:0x3121c97a scn: 0x0000.00000001
 recovered at 10/08/2021 09:10:15
 status:0x4 root dba:0x00000000 chkpt cnt: 757 ctl cnt:756
begin-hot-backup file size: 0
Checkpointed at scn:  0x0000.ed6df7ab 11/23/2021 02:00:59
 thread:1 rba:(0xebb.2.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
。。。
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
Backup Checkpointed at scn:  0x0000.00000000
 thread:0 rba:(0x0.0.0)
 enabled  threads:  00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
  00000000 00000000 00000000 00000000 00000000 00000000
External cache id: 0x0 0x0 0x0 0x0
Absolute fuzzy scn: 0x0000.00000000
Recovery fuzzy scn: 0x0000.00000000 08/21/2021 14:35:35
Terminal Recovery Stamp  01/01/1988 00:00:00
Platform Information:    Creation Platform ID: 13
Current Platform ID: 13 Last Platform ID: 13
DATA FILE #22:
  name #30: /dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf
creation size=3932160 block size=8192 status=0xe head=30 tail=30 dup=1
 tablespace 6, index=6 krfil=22 prev_file=21
 unrecoverable scn: 0x0000.ed6a50d0 11/22/2021 22:37:42
 Checkpoint cnt:758 scn: 0x0000.ed6f887f 11/23/2021 03:11:45
 Stop scn: 0xffff.ffffffff 10/08/2021 02:05:50
 Creation Checkpointed at scn:  0x0000.e0672b45 08/21/2021 14:35:35
 thread:1 rba:(0xc4d.74214.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
 Offline scn: 0x0000.00000000 prev_range: 0
 Online Checkpointed at scn:  0x0000.00000000
 thread:0 rba:(0x0.0.0)
 enabled  threads:  00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
  00000000 00000000 00000000 00000000 00000000 00000000
 Hot Backup end marker scn: 0x0000.00000000
 aux_file is NOT DEFINED
 Plugged readony: NO
 Plugin scnscn: 0x0000.00000000
 Plugin resetlogs scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign creation scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign checkpoint scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Online move state: 0
DDE rules only execution for: ORA 1110

ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
error 63999 detected in background process
ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+465<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+63<-ksuitm()+5594<-ksbrdp()+3507<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+250<-ssthrdmain()+265<-main()+201<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----

*** 2021-11-23 19:35:42.048
CKPT (ospid: 10232): terminating the instance due to error 63999
ksuitm: waiting up to [5] seconds before killing DIAG(10216)

提示：
数据文件头记录的Checkpointed at scn: 0x0000.ed6df7ab 11/23/2021 02:00:59，而控制文件记录的是Checkpoint cnt:758 scn: 0x0000.ed6f887f 11/23/2021 03:11:45， 两者不一致，数据文件头丢失了1次ckpt信息，通常发生在磁盘的写丢失或人为的把历史数据覆盖现有数据，但这么近的时间可能性不大。

打赏

对不起，这篇文章暂时关闭评论。

上一篇：

下一篇：