Issue
VMs have a slow performance when using CPU intensive tasks.
Symptoms
- Task execution on the VM seems slower after migrating the VM from one HV to the other.
- Poor performance on the VM.
- Applications running on the VM take a long time to launch.
- Multi-user services have long transaction times or can handle less simultaneous users than expected
- Machine becomes unresponsive
Throubeshooting
Issues regarding the VM performance, unless very obvious, are very difficult to identify as perception plays a big role on how things should work. In order to really compare, you need to gather some information and create a satisfactory base-mark/bench-mark.
Here is an example on how to gather this information:
1. Execute a CPU intensive task and track the time to see how long it takes for the task to complete:
#!/bin/bash
COUNTER=100000
until [ $COUNTER -lt 10 ]; do
echo $((333**999)) 1>/dev/null 2>&1
let COUNTER-=1
done
2. In this case, I am using this little script called cpuperf. It is used along with time to "time" the execution:
On HV1:
vmtest:~ $ time ./cpuperf
real 0m4.588s
user 0m3.446s
sys 0m1.143s
On HV2:
vmtest:~ $ time ./cpuperf
real 0m10.588s
user 0m9.446s
sys 0m6.143s
As you can see the same script on the same VM in each Hypervisor has a different performance. That is clearly an issue with the execution times, as it takes almost double the time in HV2.
Resolution
In order to mitigate the issue, we have a check list that seems to be solving the issue at the hardware level:
- Update your BIOS firmware to the latest.
- Disable the feature for "automatically downscaling the cpus."
Deactivating this feature has proven to boost the performance.
Also, other features identified for downscaling:
Core performance boost mode (disabled)
Processor HPC Mode (disabled)
Please also talk to your manufacturer in order to understand if you need to enable/disable any other features.
Cause
We have identified some performance issues when running XEN 3.x.x. This seems to be down to the BIOS CPU scaling configuration.
As per Xen.org wiki:
NB: Many systems have features that will scale down the computing power of a CPU that is not 100% utilized. This can be in the operating system, but can also sometimes be below the operating system, in the BIOS. If you set a cap such that individual cores are running at less than 100%, this may have an impact on the performance of your workload and above the impact of the cap. For example, if your processor runs at 2GHz, and you cap a VM at 50%, the power management system may also reduce the clock speed to 1GHz; the effect will be that your VM gets 25% of the available power (50% of 1GHz) rather than 50% (50% of 2GHz). If you are not getting the performance you expect, look at the performance and cpufreq options in your operating system and your BIOS.