今天无意执行了一个脚本，后来发现kill 了j000和m000的进程，oracle被僵死在那里，j000是job的守护进程，m000是MMON进程启动的从属平行进程，都是轻量级但不容忽视！
[oracle@orazhang bdump]$ echo $ORACLE_SID orcl [oracle@orazhang bdump]$ sqlplus / as sysdba SQL*Plus: Release 10.2.0.1.0 - Production on 星期三 6月 15 14:04:33 2011 Copyright (c) 1982, 2005, Oracle. All rights reserved. Connected to an idle instance. idle> exit Disconnected [oracle@orazhang bdump]$ ps -ef|grep ora_ oracle 3453 1 0 Jun13 ? 00:00:00 ora_pmon_orcl oracle 3455 1 0 Jun13 ? 00:00:00 ora_psp0_orcl oracle 3457 1 0 Jun13 ? 00:00:00 ora_mman_orcl oracle 3459 1 0 Jun13 ? 00:00:01 ora_dbw0_orcl oracle 3461 1 0 Jun13 ? 00:00:02 ora_lgwr_orcl oracle 3463 1 0 Jun13 ? 00:00:00 ora_ckpt_orcl oracle 3465 1 0 Jun13 ? 00:00:03 ora_smon_orcl oracle 3467 1 0 Jun13 ? 00:00:00 ora_reco_orcl oracle 3469 1 0 Jun13 ? 00:00:01 ora_cjq0_orcl oracle 3471 1 0 Jun13 ? 00:00:00 ora_mmon_orcl oracle 3473 1 0 Jun13 ? 00:00:00 ora_mmnl_orcl oracle 3475 1 0 Jun13 ? 00:00:00 ora_d000_orcl oracle 3477 1 0 Jun13 ? 00:00:00 ora_s000_orcl oracle 3485 1 0 Jun13 ? 00:00:00 ora_arc0_orcl oracle 3487 1 0 Jun13 ? 00:00:00 ora_arc1_orcl oracle 3491 1 0 Jun13 ? 00:00:00 ora_qmnc_orcl oracle 3673 1 0 Jun13 ? 00:00:00 ora_q000_orcl oracle 3677 1 0 Jun13 ? 00:00:00 ora_q001_orcl oracle 29600 29164 0 14:04 pts/0 00:00:00 grep ora_ [oracle@orazhang bdump]$ tail alert_orcl.log -n 100 Thread 1 advanced to log sequence 20 Thread 1 opened at log sequence 20 Current log# 5 seq# 20 mem# 0: /u01/app/oracle/oradata/ORCL/onlinelog/redo5.log Current log# 5 seq# 20 mem# 1: /u01/app/oracle/flash_recovery_area/ORCL/onlinelog/redo5.log .........省略...... Wed Jun 15 13:59:57 2011 Process J000 died, see its trace file Wed Jun 15 13:59:57 2011 kkjcre1p: unable to spawn jobq slave process Wed Jun 15 13:59:57 2011 Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_cjq0_3469.trc: Process J000 died, see its trace file Wed Jun 15 14:00:03 2011 kkjcre1p: unable to spawn jobq slave process Wed Jun 15 14:00:03 2011 Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_cjq0_3469.trc: Wed Jun 15 14:00:55 2011 Process m000 died, see its trace file Wed Jun 15 14:00:55 2011 ksvcreate: Process(m000) creation failed Wed Jun 15 14:01:39 2011 Process J000 died, see its trace file Wed Jun 15 14:01:39 2011 kkjcre1p: unable to spawn jobq slave process Wed Jun 15 14:01:39 2011 Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_cjq0_3469.trc: Wed Jun 15 14:01:56 2011 Process m000 died, see its trace file Wed Jun 15 14:01:56 2011 ksvcreate: Process(m000) creation failed Wed Jun 15 14:02:57 2011 Process m000 died, see its trace file Wed Jun 15 14:02:57 2011 ksvcreate: Process(m000) creation failed Wed Jun 15 14:03:58 2011 Process m000 died, see its trace file Wed Jun 15 14:03:58 2011 ksvcreate: Process(m000) creation failed Wed Jun 15 14:04:59 2011 Process m000 died, see its trace file Wed Jun 15 14:04:59 2011 ksvcreate: Process(m000) creation failed [oracle@orazhang bdump]$ [oracle@orazhang bdump]$ cat /u01/app/oracle/admin/orcl/bdump/orcl_cjq0_3469.trc /u01/app/oracle/admin/orcl/bdump/orcl_cjq0_3469.trc Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production With the Partitioning, OLAP and Data Mining options ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1 System name: Linux Node name: orazhang Release: 2.6.18-53.el5xen Version: #1 SMP Wed Oct 10 17:06:12 EDT 2007 Machine: i686 Instance name: orcl Redo thread mounted by this instance: 1 Oracle process number: 10 Unix process pid: 3469, image: oracle@orazhang (CJQ0) *** SERVICE NAME:(SYS$BACKGROUND) 2011-06-15 13:59:57.946 *** SESSION ID:(162.1) 2011-06-15 13:59:57.946 *** 2011-06-15 13:59:57.946 Process J000 is dead (pid=29552, state=3): *** 2011-06-15 14:00:03.954 Process J000 is dead (pid=29556, state=3): *** 2011-06-15 14:01:39.960 Process J000 is dead (pid=29574, state=3): [oracle@orazhang bdump]$ ipcs -sa ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 0 oracle 640 1258295296 18 dest 0x00000000 32769 oracle 600 393216 2 dest 0x00000000 65538 oracle 600 393216 2 dest 0x00000000 98307 oracle 600 393216 2 dest 0x00000000 131076 oracle 600 393216 2 dest 0x00000000 163845 oracle 600 393216 2 dest 0x00000000 196614 oracle 600 393216 2 dest 0x00000000 229383 oracle 600 393216 2 dest 0x00000000 262152 oracle 600 393216 2 dest 0x00000000 294921 oracle 600 393216 2 dest 0x00000000 393226 oracle 600 393216 2 dest 0x00000000 360459 oracle 600 393216 2 dest ------ Semaphore Arrays -------- key semid owner perms nsems 0xf5d4b884 98304 oracle 640 154 ------ Message Queues -------- key msqid owner perms used-bytes messages [root@orazhang ~]# killall -9 -r "ora_*" [root@orazhang ~]# ps -ef|grep ora_ root 29973 29760 0 15:06 pts/0 00:00:00 grep ora_ [root@orazhang ~]# ipcs -as ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 32769 oracle 600 393216 2 dest 0x00000000 65538 oracle 600 393216 2 dest 0x00000000 98307 oracle 600 393216 2 dest 0x00000000 131076 oracle 600 393216 2 dest 0x00000000 163845 oracle 600 393216 2 dest 0x00000000 196614 oracle 600 393216 2 dest 0x00000000 229383 oracle 600 393216 2 dest 0x00000000 262152 oracle 600 393216 2 dest 0x00000000 294921 oracle 600 393216 2 dest 0x00000000 393226 oracle 600 393216 2 dest 0x00000000 360459 oracle 600 393216 2 dest ------ Semaphore Arrays -------- key semid owner perms nsems 0xf5d4b884 98304 oracle 640 154 ------ Message Queues -------- key msqid owner perms used-bytes messages sql > startup ORACLE instance started. Total System Global Area 1258291200 bytes Fixed Size 1219184 bytes Variable Size 385877392 bytes Database Buffers 855638016 bytes Redo Buffers 15556608 bytes Database mounted. Database opened.
Many thanks for sharing this first-class write-up. Very interesting ideas! (as always, btw)