ANBOB™

Insufficent memory related errors (i.e: ORA-27xxx).ORA-27102 error may occur because of insufficient shared memory segments.

ORA-27102 on Startup

A] Ensure that the physical RAM is greater than the sga_max_size / sga_target
B] Ensure SHMMAX kernel setting is equal or larger than the sga max size Usually occurs with  Error: 22: Invalid argument,Shmall is the total amount of shared memory
C] Ensure Ulimit settings are unlimited for the memory related parameters
C.1] The “ulimit -l” parameter is not set
D] Ensure that the swap size Usually occurs with Error: 12: Cannot allocate memory
E] If oracle version is 32bit, the maximum SGA_TARGET / SGA_MAX_SIZE is 1.75gb
unix> file $ORACLE_HOME/bin/oracle
F] Automatic Memory Management MEMORY_TARGET and MEMORY_MAX_TARGET cannot be used when LOCK_SGA is enabled or with huge pages on Linux.

ORA-27102 To Connect To Database

A] Bug 13691225 – File handle leak for processes accessing /proc/stat
B] Bug 24921392 Linux:Background process fails to start under memory/swap pressure
C] BUG 24304753 – RESTORE SHOULD CHECK AVAILABLE RAM AGAINST REQUIRED MEMORY FROM PFILE has been filed and fix will be released with AVDF 12.2BP4.
D] Not enough space for Oracle memory under /dev/shm
E] NUMA, Setting numa=off in the kernel for x86_64 machines.

A CASE CAUSED BY NUMA
For versions prior to 11gR2 (where NUMA is disabled by default), ensure the NUMA (Non Uniform Memory Architecture) feature is turned OFF unless explicitly required and tested, as there have been issues reported with NUMA enabled. Refer to Document 759565.1 for more details.Oracle 最佳实践中建议禁用NUMA,.On NUMA Enabled Systems, the value of vm.min_free_kbytes = n * 0.4% of total Physical Memory. Here ‘n’ is the number of NUMA nodes.MySQL采用了线程模式，对于NUMA特性的支持并不好，如果单机只运行一个MySQL实例，建议关闭NUMA

zzz ***Thu Jun 4 17:01:36 CST 2020
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
4 1 9903044 4558384 288372 4925348 0 0 54 16 0 0 2 1 97 0 0
1 0 9903044 4552780 288372 4925388 0 0 387 258 7637 7453 8 3 89 0 0
1 0 9903036 4554308 288376 4925508 32 0 5907 105 7707 8353 8 1 90 1 0<<<<<
zzz ***Thu Jun 4 17:01:41 CST 2020
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 0 9903032 4827400 288376 4657420 0 0 54 16 0 0 2 1 97 0 0
3 0 9903032 4819560 288376 4657424 0 0 35 182 7782 7220 8 3 89 0 0<<<<4706M
2 0 9903028 7826248 288380 4657556 0 0 51 382 6676 6921 7 2 91 0 0
zzz ***Thu Jun 4 17:01:46 CST 2020
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 0 9903028 19882220 288380 4648484 0 0 54 16 0 0 2 1 97 0 0<<<<<19449M
0 0 9903028 19915868 288380 4646208 0 0 19 18 6634 6836 2 3 94 0 0
0 0 9903028 19915364 288388 4646112 0 0 51 658 5707 6528 1 1 98 0 0

[root@anbob01 proc]# free -g
             total       used       free     shared    buffers     cached
Mem:            31         12         18          0          0          4
-/+ buffers/cache:          7         23
Swap:           15          9          6
[root@anbob01 proc]# free -g
             total       used       free     shared    buffers     cached
Mem:            31         22          8          0          0          4
-/+ buffers/cache:         17         13
Swap:           15          9          6


[oracle@anbob01 ~]$ sqlplus anbob/anbob@10.129.1.1:1521/JKDB

SQL*Plus: Release 11.2.0.4.0 Production on Thu Jun 4 18:17:39 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

ERROR:
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 26
Additional information: 294919
Additional information: 1744830464
Process ID: 0
Session ID: 0 Serial number: 0

[root@anbob01 proc]# numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 1
node 0 size: 4095 MB
node 0 free: 26 MB
node 1 cpus: 2 3
node 1 size: 4096 MB
node 1 free: 7 MB
node 2 cpus: 4 5
node 2 size: 4096 MB
node 2 free: 576 MB
node 3 cpus: 6 7
node 3 size: 4096 MB
node 3 free: 2302 MB
node 4 cpus: 8 9
node 4 size: 4096 MB
node 4 free: 1899 MB
node 5 cpus: 10 11
node 5 size: 4096 MB
node 5 free: 7 MB <<<<<<<<<<<
node 6 cpus: 12 13
node 6 size: 4096 MB
node 6 free: 7 MB  <<<<<<<<<<<<<<<
node 7 cpus: 14 15
node 7 size: 4096 MB
node 7 free: 7 MB   <<<<<<<<<<
node distances:
node   0   1   2   3   4   5   6   7 
  0:  10  20  20  20  20  20  20  20 
  1:  20  10  20  20  20  20  20  20 
  2:  20  20  10  20  20  20  20  20 
  3:  20  20  20  10  20  20  20  20 
  4:  20  20  20  20  10  20  20  20 
  5:  20  20  20  20  20  10  20  20 
  6:  20  20  20  20  20  20  10  20 
  7:  20  20  20  20  20  20  20  10 

you can check zoneinfo also.

In the case, We found that it was caused by a single process of NBU using 12G of memory.

Refrences
Summary of Causes for ORA-27102 on Startup (Doc ID 1080317.1)

打赏

微信扫一扫，打赏作者吧～