ANBOB™

最近一套oracle 12c R2的数据库日志应用总是中断，并且在standby 节点发现了一些坏块，存储检查正常，并且primary db端并没有发现坏块，standby db alert log中发现了大量的ora-600报错，当前可能为logical corruption（Internal inconsistency in the block while the block may have good header and footer. The block checksum will be correct but the block structures may be corrupt.），ADG的Active Standby Database Automatic Block Corruption Repair (ABMR)并没有自动修复该错误。这里简单记录。

PRn error with ora-600 [kdolkr-2]

ORA-00600 [kdolkr-2] [3] [1] [44]
ORA-07445 [kdxlin()+4328] 

然后开始dump redo。

ORA-07445 [kdxlin()+4328]

ORA-01110: data file 973: ''
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 76740
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: Internal programming exception
ORA-07445 [kdxlin()+4328] 
Standby crash recovery aborted due to error 10879

日志中出现在该错误，ORA-10562 ORA-10564 ORA-1110 ORA-10561 ORA-607 ORA-600 [ktbair1] Errors in the Alert Log (Doc ID 2811646.1)记录存在已知bug.右one off 的RU并不多。
Bug 30819340 – ORA-600[kcbsbt_1] Causes Media Recovery Process Shutdown on Standby Database。

ORA-00600:[6102]

ORA-00600: internal error code, arguments: [6102], [2], [7], [], [], [], [], [], [], [], [], []

ora -600 6000系列一般是在data layer，比如索引相关，也和上面的ora-7445 kdxlin 索引匹配。

ORA-600 kdblkcheckerror

ORA-10562: Error occurred while applying redo to data block (file# 973, block# 3178921)
ORA-10564: tablespace 
ORA-01110: data file 973: ''
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 76740
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdblkcheckerror], [973], [3178921], [], [], [], [], [], [], [], [], []
PR00 (PID:43695): MRP0: Background Media Recovery terminated with error 448 

v$database_block_corruption中显示存在一些Corrupt block。 存在的已知 Bug 25886433 – ORA-600 [kdBlkCheckError] [<file#>] [<block#>] [77002] by Media Recovery on Big File Tablespace (Doc ID 25886433.8)， 当前参数db_block_checking =medium， db_block_checksum=typical。 这种情况可以尝试增量备份或从生产恢复新的文件。

重新同步数据文件

select distinct to_char(fhscn, '99999999999999999999') as "Stby_datafile_hdr_SCN" from X$KCVFH;

select min(to_char(fhscn, '99999999999999999999')) as Standby_Datafile_Header_SCN from X$KCVFH;

select file#, name, to_char(checkpoint_change#, '99999999999999999999') from v$datafile_header where checkpoint_change# <
RMAN> restore datafile xxx from service pri_xxxxx;

在当前的12c版本，提供了更方便的恢复方法，可以在standby 直接restore datafile from service xx;

如果RMAN 配置了磁带会提示下面的报错

ORA-19554 error allocating device, device type: SBT_TAPE, device name: %s
ORA-27211  Failed to load Media Management Library

备份配置清理当前配置即可

RMAN> SHOW ALL
...
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT ..........
...

RMAN > CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE’ CLEAR;

在恢复了一些报错文件后，应用日志还可能报如下错误
DBWn crash database with ORA-600 [kcbb_kcfqaio_1]

dbwN process error 
ORA-00600: internal error code, arguments: [kcbb_kcfqaio_1], [3], [], [], [], [], [], [], [], [], [], []
DB crash

Known Bug 32101419 DBWRS causes DB to crash with ORA-600 [kcbb_kcfqaio_1]

目前除了升级没有太好的方法，建议禁用db block check相关，或做增量备份跳过当前归档。 目前12c r2版本的bug确实较多，该环境还发生过Multi-Instance Redo Apply (MIRA)相关的错误，所以生产建议当前选用19c版本。

MRP hang

SQL>shutdown abort;
SQL>startup mount;

SQL> recover managed standby database using current logfile disconnect;
select process, thread#, sequence#, status from v$managed_standby where process='MRP0';

-- Stop the managed recovery and start the manual recovery.

SQL> recover managed standby database cancel;


SQL> recover automatic standby database allow 10 corruption;
-- 检查一但标记为坏块后，重启恢复,指定归档文件 alter database recover automatic from '/<path>/' standby database;

SQL> alter database recover cancel;

SQL> alter database recover managed standby database using current logfile disconnect;

Note:
除了像BCT/NFS v4相关，MRP hang有好多的bug,有时因为坏块导致无法应用，MRP hang，可以尝试增加allow N corruption跳过，标记为坏块继续应用。如果以上方法都不行，可以考虑使用
RMAN incremental backup 去前滚physical database.
识别坏块的方法
1) RMAN

1.1) dump logfile validate
You could also verify whether the archive log file is corrupted on either standby or primary site by

SQL>alter system dump logfile '' validate;

2) DBVerify (DBV)

dbv file='+DATAC1/DB_s/datafile/APEX_2150625023474721.7764.933145699' blocksize=8192
dbv FILE=/fk03/DB/oradata/fnd_1.dbf LOGFILE=verify.log BLOCKSIZE=8192 FEEDBACK=100

3) LEGACY EXPORT METHOD
4) ANALYZE VALIDATE STRUCTURE

ANALYZE TABLE EMPLOYEE VALIDATE STRUCTURE [CASCADE];

打赏

微信扫一扫，打赏作者吧～