Failed Backup: Too Many Links, Cannot Create Link

Issue

Backups are failing with an error like:

Running: cp -al /onapp/backups/templates/ly7g0sjfbtmwn3_20140227112659/ 

/onapp/backups/b/j/bj2zk7p3no3xv2/ cp: cannot create link 

`/onapp/backups/b/j/bj2zk7p3no3xv2/var/lib/yum/yumdb/a/828dc883fa148e03f699e5dd3815dc6e46dd536c-alt-

php53-recode-5.3.28-2.el6-x86_64/checksum_type': Too many links Executing Rollback...

OR in later versions of OnApp:

Running: rsync --link-dest /onapp/backups/s/h/shuulpqdauqgrf/ --acls --archive --hard-links --delete --numeric-ids --sparse --exclude '/dev/**' --exclude '/proc/**' --exclude '/sys/**' --exclude '/selinux/**' --exclude '/tmp/**' /mnt/onapp-backup-nqhlfndoimwfkz/ /onapp/backups/n/q/nqhlfndoimwfkz/
rsync: link 

"/onapp/backups/n/q/nqhlfndoimwfkz/var/lib/yum/yumdb/n/5473c3fdbb2b0a81a15a2a7391bc02eeb6360495-nss-softokn-freebl-3.28.3-8.el7_4-x86_64/releasever" => 

var/lib/yum/yumdb/G/5d1d730c5f843277da5723b9d79e05fa10cd8019-GeoIP-1.5.0-11.el7-x86_64/releasever failed: Too many links (31)

Environment

OnApp 3.x and above using incremental backups

Resolution

This is best resolved from INSIDE the VM:

1. First clear

var/lib/yum/yumdb/

You can do this with:

yum clean dbcache

or:

yum clean all

2. If the VM is using ext4, this may need to be done as well:

https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#The_Contents_of_inode.i_block