首页 » ORACLE » Instance Crash in 11.2.0.3 RAC and ORA-600 [kcbo_switch_cq_1]

Instance Crash in 11.2.0.3 RAC and ORA-600 [kcbo_switch_cq_1]

最近有套新割接的环境DB 11203 2NODE RAC与原环境唯一的区别是新环境未修改任何DB参数,结果数据库两次crash,同时alert日志中显示ORA-600 [kcbo_switch_cq_1] , 该错误与UNDO 相关,还不确认是否是_undo_autotune=FALSE在以前的环境中规避了该问题,可以明确的应该是一bug。

# db alert log

Sat Nov 05 01:31:04 2016
Thread 1 advanced to log sequence 3738 (LGWR switch)
  Current log# 2 seq# 3738 mem# 0: +DATADG/anbob/onlinelog/group_2.258.923576167
Sat Nov 05 01:31:31 2016
Thread 1 advanced to log sequence 3739 (LGWR switch)
  Current log# 5 seq# 3739 mem# 0: +DATADG/anbob/onlinelog/group_5.268.923581181
Sat Nov 05 01:31:44 2016
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc  (incident=242865):
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242865/anbob1_ora_29818922_i242865.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242865/anbob1_ora_29818922_i242865.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
Sat Nov 05 01:31:52 2016
Dumping diagnostic data in directory=[cdmp_20161105013152], requested by (instance=1, osid=29818922), summary=[incident=242865].
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc  (incident=242866):
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242866/anbob1_ora_29818922_i242866.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sat Nov 05 01:31:53 2016
Sweep [inc][242865]: completed
Sweep [inc2][242865]: completed
Sat Nov 05 01:31:53 2016
Sweep [inc][242866]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242866/anbob1_ora_29818922_i242866.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
Some DDE async actions failed or were cancelled
Dumping diagnostic data in directory=[cdmp_20161105013154], requested by (instance=1, osid=29818922), summary=[incident=242866].
Sat Nov 05 01:31:54 2016
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc:
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
Sat Nov 05 01:31:54 2016
Sweep [inc2][242866]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc:
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
Dumping diagnostic data in directory=[cdmp_20161105013202], requested by (instance=1, osid=29818922), summary=[abnormal process termination].
Sat Nov 05 01:32:03 2016
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc  (incident=252001):
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_252001/anbob1_ora_29818922_i252001.trc
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_252001/anbob1_ora_29818922_i252001.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
Sat Nov 05 01:32:03 2016
Dumping diagnostic data in directory=[cdmp_20161105013203], requested by (instance=1, osid=29818922), summary=[abnormal process termination].
Sat Nov 05 01:32:06 2016
ERROR: Unable to normalize symbol name for the following short stack (at offset 457):
dbgexProcessError()+180<-dbgeExecuteForError()+72<-dbgePostErrorKGE()+2048<-dbkePostKGE_kgsf()+68<-kgeadse()+380<-kgerinv_internal()+48<-kgerinv()+48<-kserin()+76<-kcbo_switch_cq1()+748<-kcbso1()+5252<-kcbpsod()+288<-kcbsod()+92

# trace file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc

----------------------------------------------------------
LEVEL PAYLOAD
----- ------------------------------------------------------------------------------------------------------------------------------------------------
      Trace file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
      Data Mining and Real Application Testing options
      ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
      System name:      AIX
      Node name:        qdanbob1
      Release:  1
      Version:  6
      Machine:  00F739554C00
      Instance name: anbob1
      Redo thread mounted by this instance: 1
      Oracle process number: 358
      Unix process pid: 29818922, image: oracle@qdanbob1 (TNS V1-V3)
      *** 2016-11-05 01:31:44.460
      *** SESSION ID:(5540.25401) 2016-11-05 01:31:44.460
      *** CLIENT ID:() 2016-11-05 01:31:44.460
      *** SERVICE NAME:(SYS$USERS) 2016-11-05 01:31:44.460
      *** MODULE NAME:(SQL*Plus) 2016-11-05 01:31:44.460
      *** ACTION NAME:() 2016-11-05 01:31:44.460
1>     ***** Incident 242865 created, dump file:  *****
       /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242865/anbob1_ora_29818922_i242865.trc
1< ***** incident_file ***** 1>     ***** Error Stack *****
       ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
1< ***** Error Stack ***** *** 2016-11-05 01:31:52.830 1>     ***** Incident 242866 created, dump file:  *****
       /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242866/anbob1_ora_29818922_i242866.trc
1< ***** incident_file ***** 1>     ***** Error Stack *****
       ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
       ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
1< ***** Error Stack ***** *** 2016-11-05 01:31:54.449 dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0) 1>     ***** Error Stack Dump *****
       ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
       ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
1< ***** Error Stack ***** 1>     ***** Current SQL Statement for this session (sql_id=28gq1caugn5dc) *****
       insert into sm_tab
        select REGION,TELNUM,TEMPLATE_NO,PART,CONTENT,SFROM,FORMNUM,RELATEDFORMNUM,STATUS,MSGID,INTIME,SENDDATE,REALDATE,SHOULDTIMES,LEFTTIMES,TRYTIME
       S,NOTIFYTYPE,RETCODE,RETMSG,YWFORMNUM from sm_tab@lnkd2
        where REGION=:Region
         and ( INTIME>=:StartTime and INTIME      ***** PL/SQL Stack *****
        ----- PL/SQL Call Stack -----
          object      line  object
          handle    number  name
        700000bb4975e10       190  package body TBCS.PKG_REPLX_DAILY
        700000bc41cd358         1  anonymous block
2<      ***** PL/SQL Stack *****
1< ***** current_sql_statement ***** 1>     ***** Call Stack Trace *****
       calling              call     entry               
       location             type     point               
       -------------------- -------- --------------------
       skdstdst()+40        bl       107ca55c0           
       ksedst1()+112        call     skdstdst()          
       ksedst()+40          call     ksedst1()           
       dbkedDefDump()+1516  call     ksedst()            
       ksedmp()+72          call     dbkedDefDump()      
       ksupop()+4012        call     ksedmp()             
       opiodr()+4048        call     ksupop()            
       rpidrus()+216        call     opiodr()            
       skgmstack()+120      call     rpidrus()           
       rpidru()+132         call     skgmstack()         
       rpiswu2()+480        call     rpidru()            
       rpidrv()+1060        call     rpiswu2()           
       psddr0()+260         call     rpidrv()            
       psdnal()+472         call     psddr0()            
       pevm_EXIM()+240      call     psdnal()            
       pfrinstr_EXIM()+60   call     pevm_EXIM()         
       pfrrun_no_tool()+96  call     pfrinstr_EXIM()     
       pfrrun()+1032        call     pfrrun_no_tool()    
       plsql_run()+616      call     pfrrun()            
       peicnt()+320         call     plsql_run()         
       kkxexe()+508         call     peicnt()            
       opiexe()+17224       call     kkxexe()            
       kpoal8()+4616        call     opiexe()            
       opiodr()+720         call     kpoal8()            
       ttcpip()+1028        call     opiodr()            
       opitsk()+1508        call     ttcpip()            
       opiino()+940         call     opitsk()            
       opiodr()+720         call     opiino()            
       opidrv()+1132        call     opiodr()            
       sou2o()+136          call     opidrv()            
       opimai_real()+608    call     sou2o()             
       ssthrdmain()+268     call     opimai_real()       
       main()+204           call     ssthrdmain()        
       __start()+112        call     main()              
1<     ***** call_stack_dump *****

# trace file anbob1_ora_29818922_i242865.trc

Dump continued from file: /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
========= Dump for incident 242865 (ORA 600 [kcbo_switch_cq_1]) ========
----- Beginning of Customized Incident Dump(s) -----
BH (0x70000084fafdb20) file#: 176 rdba: 0x2c1626fc (176/1451772) class: 1 ba: 0x7000008485c0000
  set: 199 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0
  dbwrid: 6 obj: 35154 objn: 26924 tsn: 19 afn: 176 hint: f
  hash: [0x700000c24aa51e0,0x700000847b0eb88] lru: [0x700000847b029e0,0x70000083fb0e8d0]
  obj-flags: object_write_list
  ckptq: [0x700000c254d0728,0x700000847b04db0] fileq: [0x700000c254d1d28,0x700000c254d1d28] objq: [0x70000084fafba68,0x700000847b02cf8] objaq: [0x700000847b090e8,0x70000084faff0d0]
  use: [0x700000bfc0dd180,0x700000bfc0dd180] wait: [NULL]
  st: XCURRENT md: EXCL fpin: 'kdiwh22: kdifind' tch: 1 atm: 1236014467,1236494533
  flags: mod_started block_written_once redo_since_read affinity_lock
  change state: NOT_ACTIVE_YET     
  change count: 1
  LRBA: [0xe9b.bbb6d.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [4]
Dump of buffer cache at level 10 for tsn=19 rdba=739649276
BH (0x70000084fafdb20) file#: 176 rdba: 0x2c1626fc (176/1451772) class: 1 ba: 0x7000008485c0000
  set: 199 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0
  dbwrid: 6 obj: 35154 objn: 26924 tsn: 19 afn: 176 hint: f
  hash: [0x700000c24aa51e0,0x700000847b0eb88] lru: [0x700000847b029e0,0x70000083fb0e8d0]
  obj-flags: object_write_list
  ckptq: [0x700000c254d0728,0x700000847b04db0] fileq: [0x700000c254d1d28,0x700000c254d1d28] objq: [0x70000084fafba68,0x700000847b02cf8] objaq: [0x700000847b090e8,0x70000084faff0d0]
  use: [0x700000bfc0dd180,0x700000bfc0dd180] wait: [NULL]
  st: XCURRENT md: EXCL fpin: 'kdiwh22: kdifind' tch: 1 atm: 1236014467,1236499550
  flags: mod_started block_written_once redo_since_read affinity_lock
  change state: NOT_ACTIVE_YET
  change count: 1
  LRBA: [0xe9b.bbb6d.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [4]
  Using State Objects
 ... 
 ...
 ...
Main Stack:
kcbo_switch_cq1

# trace file /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_252001/anbob1_ora_29818922_i252001.trc

*** 2016-11-05 01:32:03.618
Dump continued from file: /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [],
========= Dump for incident 252001 (ORA 603) ========
*** 2016-11-05 01:32:03.636
dbkedDefDump(): Starting incident default dumps (flags=0x0, level=3, mask=0x0)
----- Error Stack Dump -----
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
----- SQL Statement (None) -----
Current SQL information unavailable - no SGA.
----- Call Stack Trace -----
calling              call     entry                
location             type     point                
-------------------- -------- -------------------- 
skdstdst()+40        bl       107ca55c0            
ksedst1()+112        call     skdstdst()           
ksedst()+40          call     ksedst1()            
dbkedDefDump()+1516  call     ksedst()             
ksedmp()+72          call     dbkedDefDump()       
ksfdmp()+100         call     ksedmp()             
dbgexPhaseII()+1940  call     ksfdmp()             
dbgexProcessError()  call     dbgexPhaseII()       
+932                                               
dbgeExecuteForError  call     dbgexProcessError()  
()+72                                              
dbgePostErrorKGE()+  call     dbgeExecuteForError  
2048                          ()                   
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   
68                                                 
kgeade()+364         call     dbkePostKGE_kgsf()    ### kgeade
kgefec()+204         call     kgeade()             
kgefic()+104         call     kgefec()             
ksefic()+200         call     kgefic()             
ksupop()+4176        call     ksefic()             
opiodr()+4048        call     ksupop()             
rpidrus()+216        call     opiodr()             
skgmstack()+120      call     rpidrus()            
rpidru()+132         call     skgmstack()          
rpiswu2()+480        call     rpidru()             
rpidrv()+1060        call     rpiswu2()            
psddr0()+260         call     rpidrv()             
psdnal()+472         call     psddr0()             
pevm_EXIM()+240      call     psdnal()             
pfrinstr_EXIM()+60   call     pevm_EXIM()          
pfrrun_no_tool()+96  call     pfrinstr_EXIM()      
pfrrun()+1032        call     pfrrun_no_tool()     
plsql_run()+616      call     pfrrun()             
peicnt()+320         call     plsql_run()          
kkxexe()+508         call     peicnt()             
opiexe()+17224       call     kkxexe()             
kpoal8()+4616        call     opiexe()             
opiodr()+720         call     kpoal8()             
ttcpip()+1028        call     opiodr()             
opitsk()+1508        call     ttcpip()             
opiino()+940         call     opitsk()             
opiodr()+720         call     opiino()             
opidrv()+1132        call     opiodr()             
sou2o()+136          call     opidrv()             
opimai_real()+608    call     sou2o()              
ssthrdmain()+268     call     opimai_real()        
main()+204           call     ssthrdmain()         
__start()+112        call     main()         

在MOS中可以匹配上Bug 15883525

解决方法:
1, 安装对应版本中的ONE-OFF PATCH 15883525
2, 升级到11.2.0.4, 12.1.0.2 and later, and 12.2
3, Disable undo autotune(Maybe) set _undo_autotune=FALSE –I’m not sure

打赏

,

对不起,这篇文章暂时关闭评论。