Linux 文件系统mount失败

by

Linux上文件系统df 显示有未mount的文件系统，手动mount 不报错，但也没成功,当 df 显示了一个设备，但你手动挂载时感觉“没成功”，通常意味着挂载点被“隐藏”或“覆盖”了。或是udev或dm的原因，导致设备未正常挂载，系统日志（如 /var/log/messages 或 journalctl）可能会提供挂载失败的详细原因。最近遇到一个案例简单记录

现象

客户一套RHEL数据库环境，近日因为断电重启，操作系统重启后，发现部分文件系统未mount，手动mount未提示错误，但也未挂载，有使用xfs_repair检查并修重文件系统，但问题依旧存在。

df 仅显示以下文件系统
/
/dev
/dev/shm
# lsblk  -- or  blkid 显示
sda1 sda2 sda3 sda6
# cat /etc/fstab 
...
/dev/sda6  mount /
/dev/sda2  mount /boot
/dev/sda1  mount /boot/efi
/dev/sda3   mount /u02

Note: 如果在fstab中使用uuid挂载，uuid可以对比blkid的输出对应device name。

使用手动mount返回0，但实际也没有挂载上。

mount /dev/sda3  /u02

Note: df 仍不显示/u02文件系统

查看message 日志

Sep 23 23:02:23 zdb002 kernel: sd 0:2:17:0: [sdr] Attached SCSI disk
Sep 23 23:02:23 zdb002 kernel: sd 0:2:15:0: [sdp] Attached SCSI disk
Sep 23 23:02:23 zdb002 kernel: sd 0:2:19:0: [sdt] Attached SCSI disk
Sep 23 23:02:23 zdb002 kernel: sda: sda1 sda2 sda3 sda5 sda6
Sep 23 23:02:23 zdb002 kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Sep 23 23:02:23 zdb002 kernel: sd 0:2:1:0: [sdb] Attached SCSI disk
...

Sep 23 23:02:47 zdb002 systemd: Started /usr/sbin/lvm pvscan --cache 8:192.
Sep 23 23:02:47 zdb002 systemd: Starting /usr/sbin/lvm pvscan --cache 8:192...
Sep 23 23:02:47 zdb002 systemd: Started /usr/sbin/lvm pvscan --cache 8:208.
Sep 23 23:02:47 zdb002 systemd: Starting /usr/sbin/lvm pvscan --cache 8:208...
Sep 23 23:02:47 zdb002 systemd-udevd: inotify_add_watch(7, /dev/sda5, 10) failed: No such file or directory
Sep 23 23:02:47 zdb002 systemd: Stopped target Local File Systems.
Sep 23 23:02:47 zdb002 systemd: Stopping Local File Systems.
Sep 23 23:02:47 zdb002 systemd: Unmounting /boot/efi...
Sep 23 23:02:47 zdb002 systemd: Unmounting /u02...
Sep 23 23:02:47 zdb002 systemd: Unmounted /boot/efi.
Sep 23 23:02:47 zdb002 systemd: Unmounting /boot...
Sep 23 23:02:47 zdb002 kernel: XFS (sda3): Unmounting Filesystem
Sep 23 23:02:47 zdb002 kernel: XFS (sda2): Unmounting Filesystem
Sep 23 23:02:47 zdb002 systemd: Unmounted /u02.
Sep 23 23:02:47 zdb002 systemd: Unmounted /boot.

查看分区

cat /proc/partitions

fdisk -l /dev/sda

parted -l

显示确实有/dev/sda5分区，但是lsblk不显示，ls -l /dev/sda5 提示设备文件不存在。可能是udev挂载sda5失败，导致又自动unmount了其他文件系统。

解决方法

刷新分区表 

partprobe /dev/sda
 or
kpartx -u /dev/sda5

后恢复正常， 如果还不能解决，尝试

systemctl daemon-reload

或重启操作系统。

网上相关问题

https://unix.stackexchange.com/questions/204906/linux-mount-command-returns-zero-0-but-not-working

Finally found the answer here.

Actually the problem were came from systemd-udevd which succeeded original udev. The systemd-udevd creates it’s own mirror of root file system, when the ‘udev’ rule mount the device, it gets mounted and is accessible from:

/proc/{PID of systemd-udevd service}/root/{path to mount point}
but it is not visible from main root filesystem /.

archlinux’s wiki (here) suggests:

Warning: To mount removable drives, do not call mount from udev rules. In case of FUSE filesystems, you will get Transport endpoint not connected errors. Instead, you could use udisks that handles automount correctly or to make mount work inside udev rules, copy /usr/lib/systemd/system/systemd-udevd.service to /etc/systemd/system/systemd-udevd.service and replace MountFlags=slave to MountFlags=shared.[3] Keep in mind though that udev is not intended to invoke long-running processes.

Solution:

I copied /usr/lib/systemd/system/systemd-udevd.service to the /etc/systemd/sytem/ directory and replaced MountFlags=slave by MountFlags=shared. then restarted the system, and now everything works fine.

RHEL Underlying paths of dm-multipath devices showing error:”systemd-udevd: inotify_add_watch(6, /dev/sdg4, 10) failed: Nosuch file or directory”

Resolution
This issue is a race between KERNEL!=”sr*”, OPTIONS+=”watch” run from /lib/udev/rules.d/60-persistent-storage.rules and the /sbin/partx -d –nr 1-1024 $env{DEVNAME} run from/lib/udev/rules.d/62-multipath.rules .The
partx command in this udev rule removes the kernel mapping of the partition device before worker executing
udev_watch_begin is able to complete.
The kernel mappings for partitions on scsi devices controlled by dm-multipath are expected to be gone. This error isbenign and could be ignored safely.


Root Cause
From testing, it seems this race was possible in earlier versions, but it’s prevalance was drastically increased at the additionof patch 0168-RHBZ-1347769-shared-lock.patch within the device-mapper-multipath package. This allowed udevd anddm-multipath to share a lock on the device.

https://bugzilla.redhat.com/show_bug.cgi?id=1402426

— OVER —

Leave a Comment