| We moved this page to our Documentation Portal. You can find the latest updates here. |
Issue
After upgrade to version 6.0 hot migration fails for VSs with an error message:
Running: virsh migrate --live --unsafe byyjnxgstgywrm qemu+ssh://192.168.163.136:22/system?no_verify=1 tcp:192.168.163.136
error: internal error: process exited while connecting to monitor: 2018-10-30T11:02:45.478516Z qemu-kvm: unable to map backing store for guest RAM: Cannot allocate memory
Fatal: Execution of virsh migrate --live --unsafe byyjnxgstgywrm qemu+ssh://192.168.163.136:22/system?no_verify=1 tcp:192.168.163.136 failed
Environment
OnApp version 6.0
Resolution 1
Case 1. If you do not have any VSs migrated, the issue can be fixed in the following way:
1. Check the value of Huge Pages at the destination compute resource:
# cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 0
These values indicate that Huge Pages are disabled and hot migration is impossible.
2. Reserve some resources for Huge Pages. Before VS migration you should reserve Huge Pages for qemu process.
For that, do the following:
- Identify the size of a huge page by viewing the Hugepagesize entry in the /proc/meminfo file. For example:
# cat /proc/meminfo | grep Hugepagesize
Hugepagesize: 2048 kB
- Calculate the number of Huge Pages that you need to reserve, using the formula:
HPAGES = $((($GUESTs_SIZE_IN_MB)/ 2) + 10)
Where:
GUEST_SIZE_IN_MB - the amount of memory size in MB, required for VSs migration.
|
For example: You need to migrate 2 virtual servers with 1GB of RAM. You divide the amount of Huge Page memory that you need by the size of a Huge Page. The calculation will look in the following way: HPAGES = $(((1024+1024)/ 2) + 10) = 1029
|
- Allocate the memory as Huge Pages by writing the number of Huge Pages to the /proc/sys/vm/nr_hugepages file.
# echo 1029 >/proc/sys/vm/nr_hugepages
3. Verify that Huge Pages values are configured accordingly:
# cat /proc/meminfo | grep -i huge
AnonHugePages: 112640 kB
HugePages_Total: 1029
HugePages_Free: 1029
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Case 2. You have some VSs already migrated, but need to migrate one more and need to check what is already used and to add that number to it. In this case, do the following:
1. Check for the current free Huge Pages:
#cat /proc/meminfo | grep HugePages_Free
HugePages_Free: 802
2. Multiply the HugePages_Free by 2 (2 is the number of Mb of a single page).
Example:
802*2=1604
The virtual server will be migrated with no issues if RAM does not exceed 1600Mb.
If virtual server RAM exceeds this value, the number of HugePages_Total must be increased.
# cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 1029
#cat /proc/meminfo | grep HugePages_Free
HugePages_Free: 802
3. Calculate new HugePages_Total value.
Example:
VS with 2Gb RAM, require 1024 of HugePages_Free but we have 802
1024-802=222 - is the number of pages that we need to add to our reserve
4. Set the new value:
New value for HugePages_Total = HugePages_Total + 222 +10 = 1261
Where 10 HugePages reserved for some factors.
5. Run the following commands:
# echo 1261 >/proc/sys/vm/nr_hugepages
# cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 1261
Resolution 2
You can specify how large the pool of huge pages can grow in /proc/sys/vm/nr_overcommit_hugepages file, if more huge pages than /proc/sys/vm/nr_hugepages are requested by applications. Writing any non-zero value into this file indicates the number of "surplus" Huge Pages from the kernel's normal page pool that the hugetlb subsystem is allowed to obtain when the persistent huge page pool is exhausted. As these surplus Huge Pages become unused, they are freed back to the kernel's normal page pool.
1. Check the value of Huge Pages at the destination compute resource:
# cat /proc/sys/vm/nr_overcommit_hugepages
0
These values indicate that Huge Pages overcommit is disabled.
2. Identify the size of a huge page by viewing the Hugepagesize entry in the /proc/meminfo file.
Example:
# cat /proc/meminfo | grep Hugepagesize
Hugepagesize: 2048 kB
3. Set the Huge Page overcommit to a large value (large enough to meet your peak Huge Page requests). Then, you can have a dynamic Huge Page allocation in your environments.
|
For example: You have a compute resource with 32Gb (32768Mb) of RAM, so you can set the Huge Page overcommit value to almost 80% of free RAM (approximately 26000Mb) Divide this value by 2 (2 is the number of Mb of a single page) So your command should be: # echo 13000 > /proc/sys/vm/nr_overcommit_hugepage
|
Cause
In previous versions, the VSs could be booted with Huge Pages enabled. In OnApp version 5.10, Huge Pages were removed and so the resources cannot be allocated on the destination compute resource. To enable Transparent Huge Pages for CentOS 6 and CentOS 7 CloudBoot KVM compute resources, run the following command in the custom config file:
echo always > /sys/kernel/mm/transparent_hugepage/enabled