Issue
Offlining a virtual disk (VDisk) using onappstore offline fails because the corresponding device-mapper mapping does not disappear.
Example error:
result=FAILURE error=onappstore offlineVDisk rxik65m89072uh failed on frontend 2660499106 with error map: [] and optional error: API call failed for a subset of nodes. Failures: [('2660499106', u'Failed to detect device mapper node [/dev/mapper/rxik65m89072uh] disappear. out:, err:None')]A related error may appear when attempting to remove a mapping manually:
device-mapper: remove ioctl on <device> failed: Device or resource busy Command failedCause
The VDisk cannot be offlined because a device-mapper mapping remains present or busy. This can happen when another mapping depends on the VDisk, a process is using the device, a backup snapshot remains mounted, or an NBD connection is still active.
A suspended device-mapper mapping may also require investigation.
Resolution
1. Try to offline the vdisk manually from the CLI
[root@ ~]# onappstore offline uuid=rxik65m89072uh
It could be succesfull or fail
Result=FAILURE error=onappstore offlineVDisk rxik65m89072uh failed on frontend 2660499106 with error map: [] and optional error: API call failed for a subset of nodes. Failures: [('2660499106', u'Failed to detect device mapper node [/dev/mapper/rxik65m89072uh] disappear. out:, err:None')] completion_time=502. Inspect the device-mapper hierarchy
Find and inspect the mapping hierarchy and after remove the dependent mappings first, following the actual hierarchy.
[root@ ~]# dmsetup ls --tree | grep rxik65m89072uh
rxik65m89072uhX1 (253:30)
`-rxik65m89072uh (253:29)
|-1_rxik65m89072uh (253:28)
`-0_rxik65m89072uh (253:3)
[root@~]# If there is Xn suffix (X1in this case) at the end of the mapper, it means that it was created with kpartx tool. Then proceed with removing it with same tool:
[root@ ~]# kpartx -s -d -p X /dev/mapper/rxik65m89072uhIf there is no Xnsuffix at the end of the mapper proceed with removing using dmsetup tool
[root@10.0.0.42 ~]# dmsetup ls --tree | grep c5km9wj0821rxa
c5km9wj0821rxa (253:197)
|-1_c5km9wj0821rxa (253:224)
`-0_c5km9wj0821rxa (253:223)
[root@10.0.0.42 ~]#
[root@10.0.0.42 ~]# dmsetup remove c5km9wj0821rxa
[root@10.0.0.42 ~]# dmsetup remove 1_c5km9wj0821rxa
[root@10.0.0.42 ~]# dmsetup remove 0_c5km9wj0821rxa
[root@10.0.0.42 ~]# If any of these methods are succesfull you can proceed with offlining the disk as in step 1.
If above methods fail to remove mappers with below error, proceed with further checks.
device-mapper: remove ioctl on <device> failed: Device or resource busy Command failed
3. Check for a mounted backup snapshot
If the VDisk is associated with a backup operation, check the backup server (BS) for a mountpoint referencing the VDisk:
[root@BS ~]# mount |grep rxik65m89072uh /dev/sdgn1 on /mnt/onapp-backup-rxik65m89072uh type ext4 (rw,noatime) [root@BS ~]#
If the mount belongs to the affected VDisk, determine whether the backup operation is still active and whether the filesystem is still in use.
If the backup is no longer active and the mount is safe to remove, unmount it:
[root@BS ~]# umount /mnt/onapp-backup-rxik65m89072uh
[root@BS ~]#
If the unmount succeeds, return to step 2, remove any remaining stale mappings in dependency order, and retry offlining (step 1).
If no relevant mountpoint exists, continue to step 4.
4. Check the device-mapper state
A mapping may be suspended. Inspect the state of the affected device and its related mappings
Example:
[root@cs101 ~]# dmsetup info | grep -C3 "3rmsebkpl0c1fy" Major, minor: 253, 314 Number of targets: 1 Name: 0_3rmsebkpl0c1fy State: ACTIVE Read Ahead: 256 Tables present: LIVE -- Major, minor: 253, 151 Number of targets: 1 Name: 1_3rmsebkpl0c1fy State: ACTIVE Read Ahead: 256 Tables present: LIVE -- Major, minor: 253, 92 Number of targets: 1 Name: 3rmsebkpl0c1fy State: SUSPENDED Read Ahead: 256 Tables present: LIVE -- Major, minor: 253, 123 Number of targets: 1 [root@cs101 ~]#
For resolving the issue device mapper should be resumed:
[root@cs101 ~]# dmsetup resume 3rmsebkpl0c1fy
After the mapping is recovered, retry the cleanup and offlining procedure.
If no suspended mapping is found, continue to step 5.
5. Identify processes and NBD connections associated with the VDisk
Check for processes referencing the affected VDisk UUID:
[root@ ~]# ps aux | grep rxik65m89072uh root 10103 6.5 0.0 1172 80 ? Ss 14:14 20:16 /onappstore/sbin/bdevclient 10.200.5.1 26088 /dev/nbd49 -b 65536 -U rxik65m89072uh -timeout 60 root 10118 6.2 0.0 1172 76 ? Ss 14:14 19:22 /onappstore/sbin/bdevclient 10.200.6.2 18705 /dev/nbd50 -b 65536 -U rxik65m89072uh -timeout 60 root 10124 6.9 0.0 1172 76 ? Ss 14:14 21:32 /onappstore/sbin/bdevclient 10.200.5.2 28367 /dev/nbd51 -b 65536 -U rxik65m89072uh -timeout 60 root 11249 0.0 0.0 6440 772 pts/6 S+ 19:23 0:00 grep 02frn1hyl37v8k [root@ ~]#
disconnect it using the OnApp utility:
[root@ ~]# /onappstore/sbin/bdevclient -d /dev/nbd49 Disconnecting: que, disconnect, sock, done [root@ ~]# /onappstore/sbin/bdevclient -d /dev/nbd50 Disconnecting: que, disconnect, sock, done [root@1 ~]# /onappstore/sbin/bdevclient -d /dev/nbd51 Disconnecting: que, disconnect, sock, done
Verify that the relevant connections have disappeared:
[root@ ~]# ps aux | grep '<VDISK_UUID>' [root@ ~]#
6. Handle stale device-mapper mappings after disconnecting NBD sessions
Check if the disk is offline
If the NBD connections have been disconnected but the VDisk still cannot be offlined, inspect the device-mapper hierarchy again:
dmsetup ls --tree | grep -B5 -A10 <VDISK_UUID> rxik65m89072uh (253:29) |-1_rxik65m89072uh (253:28) `-0_rxik65m89072uh (253:3)
If the mappings remain after the NBD sessions have been disconnected, the device-mapper structure may not have been cleaned up correctly during offlining.
First, confirm that the VDisk is no longer used by a running VM, backup operation, or other process, and verify that the remaining mappings are stale.
If the mappings cannot be removed normally with dmsetup remove, rename the stale mappings using dmsetup rename
Example:
dmsetup rename rxik65m89072uh rxik65m89072uh.bad