| We moved this page to our Documentation Portal. You can find the latest updates here. |
Issue
After installing a new Hypervisor, very poor TCP performance was experienced on the VMs residing on that Hypervisor.
Environment
Any OnApp version
Troubleshooting
1. Download speed tests where 4 Kb/s, a 500MB file estimated a 62-hour download time:
[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 500M 0 35944 0 0 2324 0 62:39:57 0:00:15 62:39:42 2350^C
2. The same test from the Hypervisor showed much faster download speed - 16MB/s.
3. While checking the logs, the following was found on /var/log/dmesg and dmesg:
bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled
Resolution
1. Disable LRO on each NIC:
ethtool -K eth2 lro off
2. After disabling LRO, the TCP performance was fine on the VMs:
[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 500M 100 500M 0 0 13.2M 0 0:00:37 0:00:37 --:--:-- 17.2M
3. To have the config persist across reboots, use ETHTOOL_OPTS (initscripts-9.03.27-1 or later)on the rc.local of the HV.
#Disabling LRO after poor TCP performance identified:
ETHTOOL_OPTS="-K eth0 lro off" ETHTOOL_OPTS="-K eth1 lro off" ETHTOOL_OPTS="-K eth2 lro off" ETHTOOL_OPTS="-K eth3 lro off"
4. If the version of initscripts is earlier than initscripts-9.03.27-1, then on rc.local:
/sbin/ethtool -K eth0 lro off /sbin/ethtool -K eth1 lro off /sbin/ethtool -K eth2 lro off /sbin/ethtool -K eth3 lro off ..... etc