近期有客户的oracle数据库启动失败，之前有过存储掉电，而启动后提示ORA-01242: data file suffered media failure和ORA-01114: IO error writing block to file错误，关键是Linux-x86_64 Error: 5: Input/output error，表明问题在OS层，但是从OS 看文件存在，简单记录分析流程。
数据库alert log
Current log# 3 seq# 46656 mem# 0: /oradata/anbob/redo03.log
2026-02-10 13:05:38.908000 +08:00
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob/trace/anbob_ckpt_43797.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01114: IO error writing block to file 345 (block # 1)
ORA-01110: data file 345: '/oradata/anbob/tbs0341.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1
Additional information: -1
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob/trace/anbob_ckpt_43797.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01114: IO error writing block to file 345 (block # 1)
ORA-01110: data file 345: '/oradata/anbob/tbs0341.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1
Additional information: -1
CKPT (ospid: 43797): terminating the instance due to error 1242提示是数据文件在写时失败。
检查数据文件
$ ls -l tbs*.dbf
...
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 4 15:32 tbs0335.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 4 15:32 tbs0336.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 4 15:32 tbs0337.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 4 15:32 tbs0338.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 4 15:32 tbs0339.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 4 15:32 tbs0340.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 4 15:32 tbs0341.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0342.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0343.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0344.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0345.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0346.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0347.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0348.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0349.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0350.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Feb 10 09:05 tbs0351.dbf
...
有些情况是因为存储 100% 使用，无法扩展
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 50G 11G 40G 21% /
devtmpfs 126G 0 126G 0% /dev
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 126G 85M 126G 1% /run
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/sda2 509M 235M 275M 46% /boot
/dev/sda1 512M 9.8M 502M 2% /boot/efi
/dev/mapper/oradata-lv_oradata 15T 12T 3.1T 80% /oradata
...显示80%， 且尝试在该文件系统touch 新文件和，rm 一些无效文件后，问题并未解决
使用dbv 校验文件
$ for i in $(seq 340 406); do echo "/oradata/anbob/tbs0$i.dbf"; dbv file="/oradata/anbob/tbs0$i.dbf"; done
/oradata/anbob/tbs0340.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Wed Mar 4 16:30:13 2026
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBV-00102: File I/O error on FILE (/oradata/anbob/tbs0340.dbf) during end read operation (-2)
/oradata/anbob/tbs0341.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Wed Mar 4 16:30:13 2026
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBV-00102: File I/O error on FILE (/oradata/anbob/tbs0341.dbf) during end read operation (-2)
/oradata/anbob/tbs0342.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Wed Mar 4 16:30:13 2026
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBV-00102: File I/O error on FILE (/oradata/anbob/tbs0342.dbf) during end read operation (-2)
/oradata/anbob/tbs0343.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Wed Mar 4 16:30:14 2026
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBV-00102: File I/O error on FILE (/oradata/anbob/tbs0343.dbf) during end read operation (-2)
/oradata/anbob/tbs0344.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Wed Mar 4 16:30:14 2026
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBV-00102: File I/O error on FILE (/oradata/anbob/tbs0344.dbf) during end read operation (-2)
/oradata/anbob/tbs0345.dbf遍历表空间新创建的文件，发现从340# 文件开始，后面的全部dbv无法读取。
dbv验证前部分数据文件
$ dbv file=/oradata/anbob/tbs0001.dbf
DBVERIFY: Release 11.2.0.4.0 - Production on Wed Mar 4 16:15:24 2026
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = /oradata/anbob/tbs0001.dbf
DBVERIFY - Verification complete
Total Pages Examined : 3932160
Total Pages Processed (Data) : 123137
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 18132
Total Pages Failing (Index): 0
Total Pages Processed (Other): 3504298
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 286593
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 2483362263 (2.2483362263)使用dd 读取文件
$ dd if=/oradata/anbob/tbs0341.dbf bs=8192 count=10|hexdump|more
10+0 records in
10+0 records out
81920 bytes (82 kB) copied, 0.000491537 s, 167 MB/s
0000000 a200 0000 0000 ffc0 0000 0000 0000 0000
0000010 fa5a 0000 2000 0000 0000 003c 7c7d 7a7b
0000020 81a0 0000 0000 0000 0000 0000 0000 0000
0000030 0000 0000 0000 0000 0000 0000 0000 0000
*
0002000 a20b 0000 0001 5640 0000 0000 0000 0401
0002010 156d 0000 0000 0000 0400 0b20 4e41 748f
0002020 4948 5353 5847 5a50 8b66 0009 0000 003c
0002030 2000 0000 0159 0003 0000 0000 0000 0000
0002040 0000 0000 0000 0000 0000 0000 0000 0000
*
0002060 0000 0000 822b 9066 0002 0000 919d 4746
0002070 b937 4745 2006 000e 0000 0000 0000 0000
0002080 0000 0000 0000 0000 0000 0004 3a80 0000
0002090 dff1 4922 3a7c 0000 0000 0000 0000 0000
00020a0 0000 0000 0000 0000 0000 0000 0000 0000
*验证文件系统 可用
$ touch c
[oracle@dbhost01 anbob]$ ls -l c
-rw-r--r-- 1 oracle oinstall 0 Mar 4 16:55 c
[oracle@dbhost01 anbob]$ touch -d "yesterday" c
[oracle@dbhost01 anbob]$ ls -l c
-rw-r--r-- 1 oracle oinstall 0 Mar 3 16:55 c
[oracle@dbhost01 anbob]$ rm c
[oracle@dbhost01 anbob]$ touch -d "yesterday" tbs0406.dbf
[oracle@dbhost01 anbob]$ ls -l tbs0406.dbf
-rwxr-x--- 1 oracle oinstall 32212262912 Mar 3 16:55 tbs0406.dbf
[oracle@dbhost01 anbob]$ stat tbs0406.dbf
File: ‘tbs0406.dbf’
Size: 32212262912 Blocks: 62914576 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 12883951492 Links: 1
Access: (0750/-rwxr-x---) Uid: ( 1002/ oracle) Gid: ( 1001/oinstall)
Access: 2026-03-03 16:55:51.514540836 +0800
Modify: 2026-03-03 16:55:51.514540836 +0800
Change: 2026-03-04 16:55:51.513147512 +0800检查操作系统日志和存储日志
# dmesg
# vi message如果操作系统确认正常，可尝试做datafile recover
SQL> recover datafile xx;
-- alter database open检查存储
lsblk
lsscsi
dlnkmgr
Note: 有一半的存储光纤链路离线。
修复光纤链路后恢复， 后确认创建vg卷组时，用的是单链路设备名/dev/sdb, 而没有使用聚合多路径后的设备名，导致在损坏的链路(sdb) 不可用时，出现了该问题。
— over —