| We moved this page to our Documentation Portal. You can find the latest updates here. |
Question
What can I do if "device-mapper: remove ioctl failed: Device or resource busy" shows when running dmsetup remove?
Environment
OnApp 5.x,6.x
Answer
Follow these steps to find the process preventing the dm entry from being removed:
1) SSH into HV shown in the error log.
2) Check the device mapper hierarchy to see which backup is utilizing the -real snapshot in the error log.
- Run: dmsetup ls --tree | grep -B5 onapp--xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real
- Please keep in mind that if you don't see the root backup device, you may need to increase the value for the -B switch to something higher than 5.
Example:
[root@10.0.0.97 ~]# dmsetup ls --tree | grep -B5 onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real
onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq (253:12)
|-onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq-cow (253:21)
| `-mpath2 (253:6)
| |- (8:64)
| `- (8:32)
`-onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real (253:20)
3) Determine what process is using the top dm entry in the hierarchy with lsof.
- onapp--xxxxxxxxxxxxxx-backup--yyyyyyyyyyyyyy
- Run: lsof /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy
Example:
[root@10.0.0.97]# lsof /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tgtd 5991 root 15u BLK 253,12 0t0 331375079 /dev/mapper/onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq
Note: If there is no backup server, lsof might show nothing here. Check "/proc/mounts/" to see if the backup is mounted. If it is mounted, umount it, and proceed with clearing the dmsetup entries.
4) Since the backup is being utilized by tgtd, check if there are any iSCSI targets active.
- Run: tgtadm --lld iscsi --mode target --op show
Example:
[root@10.0.0.97]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 644245 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq
Backing store flags:
Account information:
ACL information:
10.0.50.99
5) Check the Backup Server for related ISCSI sessions and logout of them.
- Login to the backup server shown under "ACL information" from the previous step and run iscsiadm -m session.
- Compare the target name from the previous step and logout of the corresponding ISCSI sessions using iscsiadm -m node -u -T <target name>.
- If there are no corresponding sessions on the backup server, you can skip to step 6.
Example:
[root@10.0.0.99]# iscsiadm -m session
tcp: [171] X.X.X.X:XXXX,1 iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq
[root@10.0.0.99]# iscsiadm -m node -u -T iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq
Important: Make sure the -T command is -T. If the iscsiadm command does not receive the -T after the -u, it will logout of all iscsi targets!
6) Delete the iSCSI target.
- Go back to the Hypervisor and run tgtadm --lld iscsi --op delete --mode target --tid *
- tid stands for Target ID, which is shown when running the command in step 4.
Example:
[root@10.0.0.97]# tgtadm --lld iscsi --op delete --mode target --tid 1
7) Remove the dm entries for the backup and the -real snapshot.
For entries that end with X1, you should run this first: kpartx -d -p X /dev/mapper/backup-XXXXXXXXXX
- Run: dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy
- Run: dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy-cow
- Run: dmsetup remove /dev/mapper/onapp--xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real
Example:
[root@10.0.0.97]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq
[root@10.0.0.97]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq-cow
[root@10.0.0.97]# dmsetup remove /dev/mapper/onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real
Important: Do not run dmsetup remove on a non-backup or non-real device. That can cause data loss for the VM.