| We moved this page to our Documentation Portal. You can find the latest updates here. |
Question
How do I remove backup snapshots on a hypervisor without getting an LVM I/O error?
Environment
All OnApp versions
LVM Datastore
Answer
First, we will show how OnApp performs backups to show the process and commands used. This does not cover incremental backups. Three methods depending on the VM's operating system and the HV type are described:
**** Linux backups on XEN HV (LVM logical volume is formatted with filesystem)
1) LVM snapshot is created and the devices in /dev/mapper appear:
/dev/mapper/onapp-aaaaaaaa/ccccccccccc-real /dev/mapper/onapp-aaaaaaaa/backup-dddddddd /dev/mapper/onapp-aaaaaaaa/backup-dddddddd-cow
2) LVM snapshot is mounted to some temporary directory
3) The data is 'tar'-ed to a backup file (rsync'ed for incremental backups)
4) Umount
5) LVM snapshot is deleted
**** Linux backups on KVM HV (partition is created on the LVM logical volume, and this partition is formatted with filesystem)
1) LVM snapshot is created and the devices in /dev/mapper appear:
/dev/mapper/onapp-aaaaaaaa/ccccccccccc-real /dev/mapper/onapp-aaaaaaaa/backup-dddddddd /dev/mapper/onapp-aaaaaaaa/backup-dddddddd-cow
2) 'kpartx -a -p X ...' is used to create a device for partition and the device in /dev/mapper appears:
/dev/mapper/backup-ddddddddX1
3) The partition is mounted to some temporary directory
4) The data is 'tar'-ed to a backup file (rsync'ed for incremental backups)
5) The partition is unmounted
6) 'kpartx -d -p X ...' is used to create device for partition
7) LVM snapshot is deleted
**** Windows backups on XEN and KVM HV (partition is created on LVM logical volume and this partition is formatted with filesystem)
1) LVM snapshot is created and the devices in /dev/mapper appear:
/dev/mapper/onapp-aaaaaaaa/ccccccccccc-real /dev/mapper/onapp-aaaaaaaa/backup-dddddddd /dev/mapper/onapp-aaaaaaaa/backup-dddddddd-cow
2) 'kpartx -a -p X ...' is used to create a device for partition and the device in /dev/mapper appears:
/dev/mapper/backup-ddddddddX1
3) 'ntfsclone' backs up data to file
4) 'kpartx -d -p X ...' is used to create device for partition
5) LVM snapshot is deleted
Now, we will show how to detect and remove the backup snapshots.
**** Detection of used snapshots
Running this command on the HV or backup server will show the running processes using tar or ntfsclone. As mentioned above, those are the main processes for the backup:
# ps ax|grep -e 'tar' -e 'ntfsclone' -e 'rsync'
Note: To avoid LVM I/O error after backup snapshot is removed, the snapshot should be deleted on the HV where it is active.
**** Detection of active snapshots
Running this command on the HV or backup server will show all snapshots that are currently marked active on that server:
# lvscan|grep Active|grep Snapshot
The snapshots can be removed with:
lvremove /dev/onapp-aaaaaaa/backup-bbbbbbb
As a note, the 'lvremove' can fail if the device is busy. This is addressed below.
Sometimes 'lvscan|grep Active|grep Snapshot' doesn't show active snapshots. Run:
ls -al /dev/mapper
and look for ...X1 and backup-.... files to get 100% result.
If lvremove failed because of a busy device error, you can do the following:
**** Linux VM & XEN HV:
The snapshot /dev/onapp-aaaaaaa/backup-bbbbbbb is mounted. To check this, use the command:
# mount
If the snapshot appears in the output, try unmounting it with:
# umount /dev/onapp-aaaaaaa/backup-bbbbbbb
Then, run lvremove again.
**** Linux&KVM / Windows
The partition device wasn't deleted (/dev/mapper/backup-bbbbbbX1 still exists). Delete it with:
# kpartx -d -p X /dev/onapp-aaaaaaa/backup-bbbbbbb
Then, /dev/mapper/backup-ddddddddX1 should disappear, and you can run lvremove again.
Removing backup snapshots I/O errors.
Note: The steps below should be used ONLY if the steps above don't work.
Typically, these errors look like:
/dev/onapp-kagaedbrcmye6q/backup-tl3h1shnd3cre9: read failed after 0 of 4096 at 53687025664: Input/output error /dev/onapp-kagaedbrcmye6q/backup-tl3h1shnd3cre9: read failed after 0 of 4096 at 53687083008: Input/output error /dev/onapp-kagaedbrcmye6q/backup-tl3h1shnd3cre9: read failed after 0 of 4096 at 0: Input/output error /dev/onapp-kagaedbrcmye6q/backup-tl3h1shnd3cre9: read failed after 0 of 4096 at 4096: Input/output error /dev/onapp-kagaedbrcmye6q/backup-t6izl9vaqcar0o: read failed after 0 of 4096 at 751619211264: Input/output error /dev/onapp-kagaedbrcmye6q/backup-t6izl9vaqcar0o: read failed after 0 of 4096 at 751619268608: Input/output error /dev/onapp-kagaedbrcmye6q/backup-t6izl9vaqcar0o: read failed after 0 of 4096 at 0: Input/output error /dev/onapp-kagaedbrcmye6q/backup-t6izl9vaqcar0o: read failed after 0 of 4096 at 4096: Input/output error /dev/mapper/backup-t6izl9vaqcar0oX1: read failed after 0 of 4096 at 751616065536: Input/output error /dev/mapper/backup-t6izl9vaqcar0oX1: read failed after 0 of 4096 at 751616122880: Input/output error /dev/mapper/backup-t6izl9vaqcar0oX1: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/backup-t6izl9vaqcar0oX1: read failed after 0 of 4096 at 4096: Input/output error
To remove the errors, run:
# dmsetup remove backup-bbbbbbbbbX1 # dmsetup remove onapp--aaaaaaaaa-backup--bbbbbbbbbb # dmsetup remove onapp--aaaaaaaaa-backup--bbbbbbbbbb-cow
Make sure that you only run dmsetup remove on the backup snapshots and not on any other lines. Doing so can cause data loss or corruption. If you are unsure, please contact the Support team.