<p dir="ltr">Try enabling the gso and the tso but keeping the gro disabled</p>
<div class="gmail_quote">El 16/12/2014 19:38, "Georgios Dimitrakakis" <<a href="mailto:giorgis@acmac.uoc.gr">giorgis@acmac.uoc.gr</a>> escribió:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have changed that on both the node and the VMs and actually made things worse.<br>
<br>
I did that on both eth1 and br100 interfaces on the physical node.<br>
<br>
The transfer speed now is 15MB/s half of the one I had before!<br>
<br>
Have I missed something? I believe that this is not an expected behaviour?<br>
<br>
<br>
Here are the outputs for both br100 and eth1 on the node in case I have missed something:<br>
<br>
<br>
# ethtool -k br100<br>
Features for br100:<br>
rx-checksumming: off [fixed]<br>
tx-checksumming: on<br>
tx-checksum-ipv4: off<br>
tx-checksum-unneeded: off<br>
tx-checksum-ip-generic: on<br>
tx-checksum-ipv6: off<br>
tx-checksum-fcoe-crc: off [fixed]<br>
tx-checksum-sctp: off [fixed]<br>
scatter-gather: on<br>
tx-scatter-gather: on<br>
tx-scatter-gather-fraglist: on [fixed]<br>
tcp-segmentation-offload: off<br>
tx-tcp-segmentation: off<br>
tx-tcp-ecn-segmentation: on<br>
tx-tcp6-segmentation: on<br>
udp-fragmentation-offload: on [fixed]<br>
generic-segmentation-offload: off<br>
generic-receive-offload: off<br>
large-receive-offload: off [fixed]<br>
rx-vlan-offload: off [fixed]<br>
tx-vlan-offload: off [fixed]<br>
ntuple-filters: off [fixed]<br>
receive-hashing: off [fixed]<br>
highdma: on [fixed]<br>
rx-vlan-filter: off [fixed]<br>
vlan-challenged: off [fixed]<br>
tx-lockless: on [fixed]<br>
netns-local: on [fixed]<br>
tx-gso-robust: off [fixed]<br>
tx-fcoe-segmentation: off [fixed]<br>
tx-gre-segmentation: on [fixed]<br>
tx-udp_tnl-segmentation: on [fixed]<br>
fcoe-mtu: off [fixed]<br>
loopback: off [fixed]<br>
<br>
<br>
# ethtool -k eth1<br>
Features for eth1:<br>
rx-checksumming: on<br>
tx-checksumming: on<br>
tx-checksum-ipv4: off<br>
tx-checksum-unneeded: off<br>
tx-checksum-ip-generic: on<br>
tx-checksum-ipv6: off<br>
tx-checksum-fcoe-crc: off [fixed]<br>
tx-checksum-sctp: off [fixed]<br>
scatter-gather: on<br>
tx-scatter-gather: on<br>
tx-scatter-gather-fraglist: off [fixed]<br>
tcp-segmentation-offload: off<br>
tx-tcp-segmentation: off<br>
tx-tcp-ecn-segmentation: off<br>
tx-tcp6-segmentation: off<br>
udp-fragmentation-offload: off [fixed]<br>
generic-segmentation-offload: off<br>
generic-receive-offload: off<br>
large-receive-offload: off [fixed]<br>
rx-vlan-offload: on [fixed]<br>
tx-vlan-offload: on [fixed]<br>
ntuple-filters: off [fixed]<br>
receive-hashing: off [fixed]<br>
highdma: on [fixed]<br>
rx-vlan-filter: off [fixed]<br>
vlan-challenged: off [fixed]<br>
tx-lockless: off [fixed]<br>
netns-local: off [fixed]<br>
tx-gso-robust: off [fixed]<br>
tx-fcoe-segmentation: off [fixed]<br>
tx-gre-segmentation: off [fixed]<br>
tx-udp_tnl-segmentation: off [fixed]<br>
fcoe-mtu: off [fixed]<br>
loopback: off [fixed]<br>
<br>
<br>
<br>
<br>
Best,<br>
<br>
<br>
George<br>
<br>
<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Disabling it only on the nodes should boost the speed but disabling it<br>
in the vms too improves greatly the speed<br>
El 16/12/2014 19:13, "Georgios Dimitrakakis" escribió:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ooops...It seems that I have been confused..<br>
<br>
The pasted part is indeed from the node when I was looking<br>
somewhere else....<br>
<br>
Thanks a lot for noticing that Adrian!!!!<br>
<br>
I will turn it off on the nodes and test again!<br>
<br>
Should it be off on both the nodes and the VMs?<br>
<br>
Regards,<br>
<br>
George<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
That shows that those 3 offload settibgs are enabled.<br>
El 16/12/2014 19:01, "Georgios Dimitrakakis" escribió:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I believe that they are already disabled.<br>
<br>
Here is the ethtool output:<br>
<br>
# ethtool --show-offload eth1<br>
Features for eth1:<br>
rx-checksumming: on<br>
tx-checksumming: on<br>
tx-checksum-ipv4: off<br>
tx-checksum-unneeded: off<br>
tx-checksum-ip-generic: on<br>
tx-checksum-ipv6: off<br>
tx-checksum-fcoe-crc: off [fixed]<br>
tx-checksum-sctp: off [fixed]<br>
scatter-gather: on<br>
tx-scatter-gather: on<br>
tx-scatter-gather-fraglist: off [fixed]<br>
tcp-segmentation-offload: on<br>
tx-tcp-segmentation: on<br>
tx-tcp-ecn-segmentation: off<br>
tx-tcp6-segmentation: on<br>
udp-fragmentation-offload: off [fixed]<br>
generic-segmentation-offload: on<br>
generic-receive-offload: on<br>
large-receive-offload: off [fixed]<br>
rx-vlan-offload: on [fixed]<br>
tx-vlan-offload: on [fixed]<br>
ntuple-filters: off [fixed]<br>
receive-hashing: off [fixed]<br>
highdma: on [fixed]<br>
rx-vlan-filter: off [fixed]<br>
vlan-challenged: off [fixed]<br>
tx-lockless: off [fixed]<br>
netns-local: off [fixed]<br>
tx-gso-robust: off [fixed]<br>
tx-fcoe-segmentation: off [fixed]<br>
tx-gre-segmentation: off [fixed]<br>
tx-udp_tnl-segmentation: off [fixed]<br>
fcoe-mtu: off [fixed]<br>
loopback: off [fixed]<br>
<br>
Regards,<br>
<br>
George<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Disable offloading on the nodes with: ethtool -K interfaceName<br>
gro<br>
off<br>
gso off tso off<br>
<br>
And then try it again<br>
El 16/12/2014 18:36, "Georgios Dimitrakakis" escribió:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all!<br>
<br>
In my OpenStack installation (Icehouse and use nova legacy<br>
networking) the VMs are talking to each other over a 1Gbps<br>
network<br>
link.<br>
<br>
My issue is that although file transfers between physical<br>
(hypervisor) nodes can saturate that link transfers between<br>
VMs<br>
reach very lower speeds e.g. 30MB/s (approx. 240Mbps).<br>
<br>
My tests are performed by scping a large image file<br>
(approx.<br>
4GB)<br>
between the nodes and between the VMs.<br>
<br>
I have updated my images to use e1000 nic driver but the<br>
results<br>
remain the same.<br>
<br>
What are any other limiting factors?<br>
<br>
Does it has to do with the disk driver I am using? Does it<br>
play<br>
significant role the filesystem of the hypervisor node?<br>
<br>
Any ideas on how to approach the saturation of the 1Gbps<br>
link?<br>
<br>
Best regards,<br>
<br>
George<br>
<br>
______________________________<u></u>_________________<br>
Mailing list:<br>
<br>
</blockquote>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[1]<br>
[1] [1]<br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a> [2] [2] [2]<br>
Unsubscribe :<br>
<br>
</blockquote>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[3]<br>
[3] [3]<br>
</blockquote>
<br>
Links:<br>
------<br>
[1]<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[4]<br>
[4]<br>
[2] mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a> [5] [5]<br>
[3]<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[6]<br>
[6]<br>
[4] mailto:<a href="mailto:giorgis@acmac.uoc.gr" target="_blank">giorgis@acmac.uoc.gr</a> [7] [7]<br>
</blockquote></blockquote>
<br>
Links:<br>
------<br>
[1] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[8]<br>
[2] mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a> [9]<br>
[3] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[10]<br>
[4] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[11]<br>
[5] mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a> [12]<br>
[6] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[13]<br>
[7] mailto:<a href="mailto:giorgis@acmac.uoc.gr" target="_blank">giorgis@acmac.uoc.gr</a> [14]<br>
[8] mailto:<a href="mailto:giorgis@acmac.uoc.gr" target="_blank">giorgis@acmac.uoc.gr</a> [15]<br>
</blockquote></blockquote>
<br>
<br>
Links:<br>
------<br>
[1] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[2] mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a><br>
[3] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[4] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[5] mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a><br>
[6] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[7] mailto:<a href="mailto:giorgis@acmac.uoc.gr" target="_blank">giorgis@acmac.uoc.gr</a><br>
[8] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[9] mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a><br>
[10] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[11] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[12] mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a><br>
[13] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
[14] mailto:<a href="mailto:giorgis@acmac.uoc.gr" target="_blank">giorgis@acmac.uoc.gr</a><br>
[15] mailto:<a href="mailto:giorgis@acmac.uoc.gr" target="_blank">giorgis@acmac.uoc.gr</a><br>
[16] mailto:<a href="mailto:giorgis@acmac.uoc.gr" target="_blank">giorgis@acmac.uoc.gr</a><br>
</blockquote>
<br>
</blockquote></div>