首页 » ORACLE, ORACLE [C]系列 » Troubleshooting ASM allocation is failed due to ORA-4030 though OS has enough free memories.

Troubleshooting ASM allocation is failed due to ORA-4030 though OS has enough free memories.

某客户一套Oracle 11.2.0.4 4-node RAC ON RHEL 7.6 环境 ,ASM High冗余Diskgroup 有600TB存储(没错是个超级大库), 其中有1个1TB的ACFS DG. 一日突然节点1个节点ASM和DB实例crash, 重启后正常, 分析当时的日志是ASM 实例的VDBG后台进程出现的ora-4030错误,目前需要分析一下原因。 简单记录。

DB alert log

Thu Aug 04 19:45:05 2022
WARNING: ASM communication error: op 18 state 0x50 (3113)
ERROR: slave communication error with ASM
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_47083.trc:
ORA-03113: end-of-file on communication channel
...

NOTE: Deferred communication with ASM instance
Thu Aug 04 19:45:05 2022
NOTE: ASMB terminating
Thu Aug 04 19:45:05 2022
WARNING: ASM communication error: op 17 state 0x50 (3113)NOTE: deferred map free for map id 7572857

Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_65011.trc:
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 2380 Serial number: 31
ERROR: slave communication error with ASM
...
Session ID: 1404 Serial number: 149
ASMB (ospid: 56359): terminating the instance due to error 15064
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_65017.trc:
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-03113: end-of-file on communication channel
Session ID: 2014 Serial number: 2539
NOTE: deferred map free for map id 7572856
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_47096.trc  (incident=198037):
ORA-00600: internal error code, arguments: [2618], [8], [], [], [], [], [], [], [], [], [], []
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_65011.trc:
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-15077: could not locate ASM instance serving a required diskgroup
NOTE: deferred map free for map id 7572859

Note:
db instance crash是因为ASM intance请求失败.

ASM alert log

Thu Aug 04 19:44:52 2022
Errors in file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+ASM4_vdbg_9668.trc  (incident=2638776):
ORA-04030: out of process memory when trying to allocate 118584 bytes (pga heap,kfvsuIOStatOp)
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM4/incident/incdir_2638776/+ASM4_vdbg_9668_i2638776.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Aug 04 19:45:03 2022
Dumping diagnostic data in directory=[cdmp_20220804194503], requested by (instance=4, osid=9668 (VDBG)), summary=[incident=2638776].
Thu Aug 04 19:45:05 2022
Errors in file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+ASM4_vdbg_9668.trc:
ORA-04030: out of process memory when trying to allocate 118584 bytes (pga heap,kfvsuIOStatOp)
System state dump requested by (instance=4, osid=9668 (VDBG)), summary=[abnormal instance termination].
VDBG (ospid: 9668): terminating the instance due to error 4030
System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+ASM4_diag_3689_20220804194505.trc
Thu Aug 04 19:45:06 2022
ORA-1092 : opitsk aborting process
Thu Aug 04 19:45:06 2022
ORA-1092 : opitsk aborting process
Thu Aug 04 19:45:06 2022
License high water mark = 45

vdbg trace file

Dump file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+asm4_vdbg_9668.trc
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
..
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
DDE: Flood control is not active
========= Dump for critical error (no incident) (DIA 48002) ========
Starting a Diag Context default dump (level=3)
DIA-48002: thread terminated by fatal error
DIA-48003: out of process memory when trying to allocate 56 bytes (diag pga,dbgr entry)
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
..
..
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
kgdsdstsg()+53       call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFC14E0A000 ? 7FFC14E0A0D8 ?
                                                   7FFC14E0EB80 ? 000000003 ?
00000000077CEDDD     call     kgdsdstsg()          7F067C909878 ? 7FFC14E0EE80 ?
                                                   000000003 ? 7FFC14E0A0D8 ?
                                                   7FFC14E0EB80 ? 000000003 ?
000000000770FF80     call     00000000077CEDDD     7F067C909878 ? 000000003 ?
                                                   000000003 ? 7FFC14E0A0D8 ?
                                                   7FFC14E0EB80 ? 000000003 ?
dbgexProcessError()  call     000000000770FF80     7F067C8FE728 ? 7F067C938E58 ?
+2680                                              7FFC14E12FF8 ? 7FFC14E0A0D8 ?
                                                   7FFC14E0EB80 ? 000000003 ?
00000000076CF596     call     dbgexProcessError()  7F067C8FE728 ? 7F067C938E58 ?
                                                   000000001 ? 000000000 ?
                                                   7FFC14E0EB80 ? 000000003 ?
dbgePostErrorKGE()+  call     00000000076CF596     7F067C8FE728 ? 7F067C938E58 ?
2136                                               000000001 ? 000000001 ?
                                                   000000000 ? 000000003 ?
00000000077CEF95     call     dbgePostErrorKGE()   7F067C909878 ? 7F067C92CA98 ?
                                                   00000BB82 ? 000000001 ?
                                                   000000000 ? 000000003 ?
0000000002A28E05     call     00000000077CEF95     7F067C909878 ? 7F067C92CA98 ?
                                                   00000BB82 ? 000000001 ?
                                                   000000000 ? 000000003 ?
kgefec()+176         call     0000000002A28E05     7F067C909878 ? 7F067C909A38 ?
                                                   7F067C92CA98 ? 00000BB82 ?
                                                   000000000 ? 000000003 ?
0000000002A19192     call     kgefec()             7F067C909878 ? 7F067C92CA98 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 000000003 ?
kghalf()+1139        call     0000000002A19192     7F067C909878 ? 7F067C8FE7F8 ?
                                                   000000038 ? 000000001 ?
                                                   000000000 ? 000000003 ?
dbgric_init_cache()  call     kghalf()             7F067C909878 ? 7F067C8FE7F8 ?
+527                                               000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
dbgriap_init_adr_pg  call     dbgric_init_cache()  7F067C8FE728 ? 7FFC14E144F0 ?
a()+198                                            000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
dbgc_init_all()+848  call     dbgriap_init_adr_pg  7F067C8FE728 ? 000000003 ?
                              a()                  000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
0000000002D5FFCE     call     dbgc_init_all()      000000003 ? 7F067C8FE728 ?
                                                   000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
0000000002D5FB37     call     0000000002D5FFCE     7F067C8FE728 ? 7F067C8FE728 ?
                                                   000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
dbgc_rls_diagctx()+  call     0000000002D5FB37     000000002 ? 00AC3BA20 ?
58                                                 000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
0000000001F9A23F     call     dbgc_rls_diagctx()   000000002 ? 00AC3BA20 ?
                                                   000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
dbkc_sga_unmap()+75  call     0000000001F9A23F     000000002 ? 00AC3BA20 ?
                                                   000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
ksmsdes()+111        call     dbkc_sga_unmap()     000000000 ? 00AC3BA20 ?
                                                   000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
ksuitm()+2165        call     ksmsdes()            7FFC14E14AF0 ? 00AC3BA20 ?
                                                   000000020 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
ksbrdp()+3507        call     ksuitm()             000000000 ? 000000FBE ?
                                                   154484DD0 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
opirip()+623         call     ksbrdp()             000000000 ? 000000FBE ?
                                                   154484DD0 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
opidrv()+603         call     opirip()             000000032 ? 000000004 ?
                                                   7FFC14E15FA8 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
sou2o()+103          call     opidrv()             000000032 ? 000000004 ?
                                                   7FFC14E15FA8 ? 000000001 ?
                                                   000000000 ? 0098F08C8 ?
mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4
opimai_real()+250    call     sou2o()              7FFC14E15F80 ? 000000032 ?
                                                   000000004 ? 7FFC14E15FA8 ?
                                                   000000000 ? 0098F08C8 ?
ssthrdmain()+265     call     opimai_real()        000000000 ? 7FFC14E16170 ?
                                                   000000004 ? 7FFC14E15FA8 ?
                                                   000000000 ? 0098F08C8 ?
main()+201           call     ssthrdmain()         000000003 ? 7FFC14E16170 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 0098F08C8 ?
__libc_start_main()  call     main()               000000003 ? 7FFC14E16310 ?
+245                                               000000001 ? 000000000 ?
                                                   000000000 ? 0098F08C8 ?
00000000008489F9     call     __libc_start_main()  000848AC0 ? 000000001 ?
                                                   7FFC14E16308 ? 000000000 ?
                                                   000000000 ? 0098F08C8 ?
												   
...


----- Process Map Dump -----
******************* Dumping process map ****************
00400000-0a8ac000 r-xp 00000000 fd:03 67451950                           /u01/app/11.2.0/grid/bin/oracle
0aaac000-0aaad000 r--p 0a4ac000 fd:03 67451950                           /u01/app/11.2.0/grid/bin/oracle
0aaad000-0ac15000 rw-p 0a4ad000 fd:03 67451950                           /u01/app/11.2.0/grid/bin/oracle
0ac15000-0ac5e000 rw-p 00000000 00:00 0 
0bd44000-0bdea000 rw-p 00000000 00:00 0                                  [heap]
60000000-60001000 r--s 00000000 00:13 450040                             /dev/shm/ora_+ASM4_131075_0
60001000-61000000 rw-s 00001000 00:13 450040                             /dev/shm/ora_+ASM4_131075_0
61000000-62000000 rw-s 00000000 00:13 450041                             /dev/shm/ora_+ASM4_131075_1
...
7f04bd20c000-7f04bd21c000 rw-p 00000000 00:05 1030                       /dev/zero
7f04bd21c000-7f04bd23c000 rw-p 00000000 00:05 1030                       /dev/zero
7f04bd23c000-7f04bd25c000 rw-p 00000000 00:05 1030                       /dev/zero
7f04bd25c000-7f04bd27c000 rw-p 00000000 00:05 1030                       /dev/zero
7f04bd27c000-7f04bd29c000 rw-p 00000000 00:05 1030                       /dev/zero
7f04bd29c000-7f04bd2ac000 rw-p 00000000 00:05 1030                       /dev/zero
7f04bd2ac000-7f04bd2cc000 rw-p 00000000 00:05 1030                       /dev/zero
7f04bd2cc000-7f04bd2ec000 rw-p 00000000 00:05 1030                       /dev/zero
...
7ffc14f8e000-7ffc14f90000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
                                                                        #######  ---- total number of 65532 lines 
******************* End of process map dump ************
----- Process Resource Limits -----
***************** Dumping Resource Limits(s/h) *****************
core file size                    UNLIMITED/UNLIMITED         
data seg size                     UNLIMITED/UNLIMITED         
file size                         UNLIMITED/UNLIMITED         
pending signals                     2010 KB/2010 KB           
max locked memory                 UNLIMITED/UNLIMITED         
max memory size                   UNLIMITED/UNLIMITED         
open files                            64 KB/64 KB             
POSIX message queues                 800 KB/800 KB            
stack size                            32 MB/UNLIMITED         
cpu time                          UNLIMITED/UNLIMITED         
max user processes                    16 KB/16 KB             
virtual memory                    UNLIMITED/UNLIMITED         
file locks                        UNLIMITED/UNLIMITED         
***************** End of Resource Limits Dump ******************


Note:
call stack中有mmap(offset=210198528, len=4096) failed with errno=12报错, 当时的data和stack 在当前用户已经是unlimited, MAX memory size和virtual memory 也是unlimited . 因为mmap失败导致asm intance的VDBG进程出现的ora-4030错误 。

也可以手动收集maps

$ps -ef|grep vdbg
$ pmap -p <vdbg进程号>

mmap failed with errno=12

[root@oel7db1 ~]# perror 12
OS error code  12:  Cannot allocate memory

Note:
that 12 is exactly ENOMEM error code. Investigation for mmap() syscall shows that ENOMEM stands for “No memory is available, or the process’s maximum number of mappings would have been exceeded.”

VDBG Process

Volume Driver Process
Forwards Oracle ASM requests to perform various volume-related tasks
VDBG handles requests to lock or unlock an extent for rebalancing, volume resize, disk offline, add or drop a disk, force and dismount disk group to the Dynamic Volume Manager driver.

Note:
判断与ASM ACFS相关。

OSW vmstat

zzz ***Thu Aug 4 19:41:05 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
25  2 436116 118542424  81556 42647252    0    0  2605   406    0    0 10  3 85  1  0
19  4 436116 118563664  81556 42648388    0    0 684500  2338 478437 189366 17  7 74  2  0
21  4 436116 118491496  81556 42648276    0    0 570272  2741 459318 203020 13  6 79  2  0
zzz ***Thu Aug 4 19:41:36 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
24  7 436116 118174656  81556 42654560    0    0  2605   406    0    0 10  3 85  1  0
17  7 436116 118107056  81556 42654244    0    0 960114  4220 488187 295850 16  7 73  3  0
23  5 436116 118005568  81556 42654516    0    0 1095528  9780 476740 248006 15  7 75  4  0
zzz ***Thu Aug 4 19:42:07 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
36  1 436116 118660784  81556 42661968    0    0  2605   406    0    0 10  3 85  1  0
29  2 436116 118633664  81556 42662728    0    0 621297  8233 565649 258656 17  9 73  1  0
29  3 436116 118634256  81556 42663212    0    0 707392  2908 590466 315953 15  9 75  1  0
zzz ***Thu Aug 4 19:42:38 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
25  4 436116 118511632  81556 42665980    0    0  2605   406    0    0 10  3 85  1  0
21  3 436116 118320824  81556 42666804    0    0 758968 11114 420537 277405 16  5 75  4  0
19  3 436116 118262856  81556 42667052    0    0 774818 13257 439119 296825 13  7 77  3  0
zzz ***Thu Aug 4 19:43:09 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
23  0 436116 118024848  81556 42677192    0    0  2605   406    0    0 10  3 85  1  0
23  1 436116 117908224  81556 42678480    0    0 828708  7031 528760 157467 15  8 76  1  0
21  0 436116 117911840  81556 42678396    0    0 829886  3924 544309 201965 16  7 76  1  0
zzz ***Thu Aug 4 19:43:40 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
24  6 436116 118217664  81556 42685468    0    0  2605   406    0    0 10  3 85  1  0
16  6 436116 118186256  81556 42685844    0    0 1247490  1470 472355 210836 14  7 76  3  0
20  4 436116 118188568  81556 42686296    0    0 1252882  9776 467208 207419 13  6 78  3  0
zzz ***Thu Aug 4 19:44:11 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
23  3 436116 117897696  81556 42694212    0    0  2605   406    0    0 10  3 85  1  0
26  0 436116 117777872  81552 42695332    0    0 665390 11447 539412 167680 17 10 72  1  0
25  1 436116 117754168  81500 42695716    0    0 715547  4452 549530 185636 17  8 73  2  0
zzz ***Thu Aug 4 19:44:43 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
27  4 436116 117627000  81556 42701572    0    0  2605   406    0    0 10  3 85  1  0
21  7 436116 118095872  81556 42702520    0    0 1182373  2582 493360 193122 19  8 69  4  0
23  5 436116 118133888  81556 42702364    0    0 1095942  2632 473514 196193 16  6 74  4  0
zzz ***Thu Aug 4 19:45:14 CST 2022
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6  0 436116 125675744  44588 45139048    0    0  2605   406    0    0 10  3 85  1  0
 8  0 436116 125555808  44588 45296892    0    0  2873 772219 344671 316581  2  6 93  0  0
 4  0 436116 125464688  44588 45400196    0    0  3830 1466162 162413 196481  1  3 96  0  0

Note:
可见swap和free memory 在操作系统层还有空间。

OSW PS

ps  RSS kb为单位
USER        PID   PPID PRI %CPU %MEM    VSZ   RSS WCHAN  S  STARTED     TIME COMMAND
zzz ***Thu Aug 4 19:00:11 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308376 SYSC_s S   Jul 09 1-22:52:09 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:00:42 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308496 SYSC_s S   Jul 09 1-22:52:09 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:01:13 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308496 SYSC_s S   Jul 09 1-22:52:09 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:01:44 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308648 SYSC_s S   Jul 09 1-22:52:09 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:02:15 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308648 SYSC_s S   Jul 09 1-22:52:10 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:02:46 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308648 SYSC_s S   Jul 09 1-22:52:10 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:03:17 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308648 SYSC_s S   Jul 09 1-22:52:10 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:03:48 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308912 SYSC_s S   Jul 09 1-22:52:11 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:04:20 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308912 SYSC_s S   Jul 09 1-22:52:11 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:04:51 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308912 SYSC_s S   Jul 09 1-22:52:11 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:05:22 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7308912 SYSC_s S   Jul 09 1-22:52:12 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:05:53 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309032 SYSC_s S   Jul 09 1-22:52:12 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:06:24 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309032 SYSC_s S   Jul 09 1-22:52:12 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:06:55 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309160 SYSC_s S   Jul 09 1-22:52:13 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:07:26 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309280 SYSC_s S   Jul 09 1-22:52:13 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:07:57 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309280 SYSC_s S   Jul 09 1-22:52:13 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:08:28 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309280 SYSC_s S   Jul 09 1-22:52:14 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:08:59 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309420 SYSC_s S   Jul 09 1-22:52:14 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:09:30 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309420 SYSC_s S   Jul 09 1-22:52:15 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:10:01 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309540 SYSC_s S   Jul 09 1-22:52:15 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:10:32 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309540 SYSC_s S   Jul 09 1-22:52:15 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:11:03 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309540 SYSC_s S   Jul 09 1-22:52:16 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:11:34 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309540 SYSC_s S   Jul 09 1-22:52:16 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:12:05 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309724 SYSC_s S   Jul 09 1-22:52:17 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:12:36 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309724 SYSC_s S   Jul 09 1-22:52:17 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:13:08 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309848 -  S   Jul 09 1-22:52:17 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:13:39 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309848 SYSC_s S   Jul 09 1-22:52:18 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:14:10 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309992 SYSC_s S   Jul 09 1-22:52:18 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:14:41 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7309992 SYSC_s S   Jul 09 1-22:52:18 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:15:12 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310112 SYSC_s S   Jul 09 1-22:52:19 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:15:43 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310112 SYSC_s S   Jul 09 1-22:52:19 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:16:14 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310248 SYSC_s S   Jul 09 1-22:52:20 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:16:45 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310248 SYSC_s S   Jul 09 1-22:52:20 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:17:16 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310368 SYSC_s S   Jul 09 1-22:52:20 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:17:47 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310368 SYSC_s S   Jul 09 1-22:52:21 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:18:18 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310368 SYSC_s S   Jul 09 1-22:52:21 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:18:49 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310368 SYSC_s S   Jul 09 1-22:52:21 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:19:20 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310368 SYSC_s S   Jul 09 1-22:52:22 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:19:51 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310556 SYSC_s S   Jul 09 1-22:52:22 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:20:22 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310556 SYSC_s S   Jul 09 1-22:52:22 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:20:54 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310692 SYSC_s S   Jul 09 1-22:52:22 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:21:25 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310824 SYSC_s S   Jul 09 1-22:52:23 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:21:56 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310944 SYSC_s S   Jul 09 1-22:52:23 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:22:27 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310944 SYSC_s S   Jul 09 1-22:52:23 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:22:58 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310944 SYSC_s S   Jul 09 1-22:52:23 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:23:29 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7310944 SYSC_s S   Jul 09 1-22:52:24 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:24:00 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311128 SYSC_s S   Jul 09 1-22:52:24 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:24:31 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311128 SYSC_s S   Jul 09 1-22:52:24 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:25:02 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311248 SYSC_s S   Jul 09 1-22:52:24 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:25:33 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311248 SYSC_s S   Jul 09 1-22:52:25 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:26:04 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311248 SYSC_s S   Jul 09 1-22:52:25 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:26:35 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311248 SYSC_s S   Jul 09 1-22:52:26 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:27:06 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311404 SYSC_s S   Jul 09 1-22:52:26 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:27:37 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311404 SYSC_s S   Jul 09 1-22:52:27 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:28:08 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311404 SYSC_s S   Jul 09 1-22:52:27 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:28:39 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311600 SYSC_s S   Jul 09 1-22:52:28 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:29:10 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311720 SYSC_s S   Jul 09 1-22:52:28 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:29:41 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311720 SYSC_s S   Jul 09 1-22:52:29 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:30:12 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311720 SYSC_s S   Jul 09 1-22:52:29 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:30:43 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311720 SYSC_s S   Jul 09 1-22:52:29 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:31:14 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311920 SYSC_s S   Jul 09 1-22:52:30 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:31:45 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7311920 SYSC_s S   Jul 09 1-22:52:30 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:32:17 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312040 SYSC_s S   Jul 09 1-22:52:31 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:32:48 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312040 SYSC_s S   Jul 09 1-22:52:31 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:33:19 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312040 SYSC_s S   Jul 09 1-22:52:32 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:33:50 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312216 SYSC_s S   Jul 09 1-22:52:32 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:34:21 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312216 SYSC_s S   Jul 09 1-22:52:32 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:34:52 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312216 SYSC_s S   Jul 09 1-22:52:33 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:35:23 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312360 SYSC_s S   Jul 09 1-22:52:33 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:35:54 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312360 SYSC_s S   Jul 09 1-22:52:34 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:36:25 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312488 SYSC_s S   Jul 09 1-22:52:34 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:36:56 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312488 SYSC_s S   Jul 09 1-22:52:34 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:37:27 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312488 SYSC_s S   Jul 09 1-22:52:35 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:37:58 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312616 SYSC_s S   Jul 09 1-22:52:35 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:38:29 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312616 SYSC_s S   Jul 09 1-22:52:35 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:39:00 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312736 SYSC_s S   Jul 09 1-22:52:36 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:39:32 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312736 SYSC_s S   Jul 09 1-22:52:36 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:40:03 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312864 -  S   Jul 09 1-22:52:36 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:40:34 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312864 SYSC_s S   Jul 09 1-22:52:36 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:41:05 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7312864 SYSC_s S   Jul 09 1-22:52:37 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:41:37 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7313048 SYSC_s S   Jul 09 1-22:52:37 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:42:08 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7313048 SYSC_s S   Jul 09 1-22:52:37 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:42:39 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7313048 SYSC_s S   Jul 09 1-22:52:37 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:43:10 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7313180 SYSC_s S   Jul 09 1-22:52:38 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:43:41 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7313180 SYSC_s S   Jul 09 1-22:52:38 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:44:12 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7313320 SYSC_s S   Jul 09 1-22:52:38 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:44:43 CST 2022
grid       9668      1  19  0.4  1.3 11730252 7313320 SYSC_s S   Jul 09 1-22:52:38 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:45:14 CST 2022
grid       9668      1  19  0.4  1.3 11730372 7339608 hrtime S   Jul 09 1-22:52:50 asm_vdbg_+ASM4
zzz ***Thu Aug 4 19:45:45 CST 2022
zzz ***Thu Aug 4 19:46:16 CST 2022

Note:
持续关注进程是否有内存泄露是否过大或持续增长,当前如果正在做asm Operater有操作,我认为vdbg增长也是正常现象。

TFA OS report

# more /proc/sys/vm/max_map_count
65530

Note:
如果有tfactl 收集日志在os report中有OS 内核参数信息。

max_map_count

max_map_count 文件允许限制特定进程可以拥有的 VMA(虚拟内存区域)的数量。虚拟内存区是虚拟地址空间的连续区域。当程序尝试对文件进行内存映射、链接到共享内存段或分配堆空间时,会在进程的生命周期中创建这些区域。调整此值会限制进程可以拥有的这些 VMA 的数量。限制进程可以拥有的 VMA 数量可能会导致有问题的应用程序行为,因为当进程达到其 VMA 限制时系统将返回内存不足错误,但可以释放低内存以供其他内核使用。如果您的系统在 NORMAL 区域内存不足,则降低此值将有助于释放内存供内核使用.

mmap,OS error code  12:  Cannot allocate memory 这些错误通常是由于从操作系统中耗尽了map entries而出现的。

每个进程默认有64k map entries(max_map_count),默认的realfree allocator pagesize 是64 kB (65536), 所以64K entries可以分配4GB.加大map entries可以分配更多的内存,如 256kB (262144) pages, 限制增加到16GB.

查看realfree pages

select i.ksppinm name , v.ksppstvl cur_val,
       v.ksppstdf default_val, v.ksppstvf
  from x$ksppi i, x$ksppcv v
 where i.indx = v.indx
   and i.ksppinm in ('_realfree_heap_pagesize_hint', '_use_realfree_heap')

Diag event
ora-4030 发生时没有生成PGA heapdump信息,可以开启error TRACE

ALTER SYSTEM SET EVENTS '4030 trace name heapdump level 536870917;name errorstack level 3';

解决方案建议

1,增加map entries记录数或增加realfree pagesize

 Change the page count at the OS level: 
$ more /proc/sys/vm/max_map_count
$ sysctl -w vm.max_map_count=262144

-- OR --

Adjust the realfree heap pagesize within the database by setting the following parameters in the init/spfile and restart the database.

ALTER SYSTEM SET "_use_realfree_heap"=TRUE SCOPE=SPFILE;
ALTER SYSTEM SET "_realfree_heap_pagesize_hint"=262144 SCOPE=SPFILE;

NOTE: In Oracle 12c, the parameter _realfree_heap_pagesize_hint is changed to _realfree_heap_pagesize. In 12C, _realfree_heap_pagesize = 262144

2, 增加MEMROY_TARGET
当前系统OS free 内存较多,ASM DG较大,建议也增加一些MEMORY_TARGET(SGA+PGA AUTO). ASM 默认使用AMM, 在ASM实例中SGA的shared_pool占用较大, 所以我们的memory_target要给shared_pool预留外建议再增加2-4GB空间。

# FOR SHARED POOL

SELECT SUM(bytes)/(1024*1024*1024) FROM V$DATAFILE;
SELECT SUM(bytes)/(1024*1024*1024) FROM V$LOGFILE a, V$LOG b
WHERE a.group#=b.group#;
SELECT SUM(bytes)/(1024*1024*1024) FROM V$TEMPFILE
WHERE status='ONLINE';


o For disk groups using external redundancy, every 100 GB of space needs 1 MB of extra shared pool plus 2 MB.
o For disk groups using normal redundancy, every 50 GB of space needs 1 MB of extra shared pool plus 4 MB.
o For disk groups using high redundancy, every 33 GB of space needs 1 MB of extra shared pool plus 6 MB.
打赏

,

目前这篇文章还没有评论(Rss)

我要评论