Changing gso on tso on gro off got me back to the initial status. Although now it starts with approximately 65-70MB/s for a few seconds but then it drops down to 30MB/s Regards, George > Try enabling the gso and the tso but keeping the gro disabled > El 16/12/2014 19:38, "Georgios Dimitrakakis" escribió: > >> I have changed that on both the node and the VMs and actually made >> things worse. >> >> I did that on both eth1 and br100 interfaces on the physical node. >> >> The transfer speed now is 15MB/s half of the one I had before! >> >> Have I missed something? I believe that this is not an expected >> behaviour? >> >> Here are the outputs for both br100 and eth1 on the node in case I >> have missed something: >> >> # ethtool -k br100 >> Features for br100: >> rx-checksumming: off [fixed] >> tx-checksumming: on >> tx-checksum-ipv4: off >> tx-checksum-unneeded: off >> tx-checksum-ip-generic: on >> tx-checksum-ipv6: off >> tx-checksum-fcoe-crc: off [fixed] >> tx-checksum-sctp: off [fixed] >> scatter-gather: on >> tx-scatter-gather: on >> tx-scatter-gather-fraglist: on [fixed] >> tcp-segmentation-offload: off >> tx-tcp-segmentation: off >> tx-tcp-ecn-segmentation: on >> tx-tcp6-segmentation: on >> udp-fragmentation-offload: on [fixed] >> generic-segmentation-offload: off >> generic-receive-offload: off >> large-receive-offload: off [fixed] >> rx-vlan-offload: off [fixed] >> tx-vlan-offload: off [fixed] >> ntuple-filters: off [fixed] >> receive-hashing: off [fixed] >> highdma: on [fixed] >> rx-vlan-filter: off [fixed] >> vlan-challenged: off [fixed] >> tx-lockless: on [fixed] >> netns-local: on [fixed] >> tx-gso-robust: off [fixed] >> tx-fcoe-segmentation: off [fixed] >> tx-gre-segmentation: on [fixed] >> tx-udp_tnl-segmentation: on [fixed] >> fcoe-mtu: off [fixed] >> loopback: off [fixed] >> >> # ethtool -k eth1 >> Features for eth1: >> rx-checksumming: on >> tx-checksumming: on >> tx-checksum-ipv4: off >> tx-checksum-unneeded: off >> tx-checksum-ip-generic: on >> tx-checksum-ipv6: off >> tx-checksum-fcoe-crc: off [fixed] >> tx-checksum-sctp: off [fixed] >> scatter-gather: on >> tx-scatter-gather: on >> tx-scatter-gather-fraglist: off [fixed] >> tcp-segmentation-offload: off >> tx-tcp-segmentation: off >> tx-tcp-ecn-segmentation: off >> tx-tcp6-segmentation: off >> udp-fragmentation-offload: off [fixed] >> generic-segmentation-offload: off >> generic-receive-offload: off >> large-receive-offload: off [fixed] >> rx-vlan-offload: on [fixed] >> tx-vlan-offload: on [fixed] >> ntuple-filters: off [fixed] >> receive-hashing: off [fixed] >> highdma: on [fixed] >> rx-vlan-filter: off [fixed] >> vlan-challenged: off [fixed] >> tx-lockless: off [fixed] >> netns-local: off [fixed] >> tx-gso-robust: off [fixed] >> tx-fcoe-segmentation: off [fixed] >> tx-gre-segmentation: off [fixed] >> tx-udp_tnl-segmentation: off [fixed] >> fcoe-mtu: off [fixed] >> loopback: off [fixed] >> >> Best, >> >> George >> >>> Disabling it only on the nodes should boost the speed but >>> disabling it >>> in the vms too improves greatly the speed >>> El 16/12/2014 19:13, "Georgios Dimitrakakis" escribió: >>> >>>> Ooops...It seems that I have been confused.. >>>> >>>> The pasted part is indeed from the node when I was looking >>>> somewhere else.... >>>> >>>> Thanks a lot for noticing that Adrian!!!! >>>> >>>> I will turn it off on the nodes and test again! >>>> >>>> Should it be off on both the nodes and the VMs? >>>> >>>> Regards, >>>> >>>> George >>>> >>>>> That shows that those 3 offload settibgs are enabled. >>>>> El 16/12/2014 19:01, "Georgios Dimitrakakis" escribió: >>>>> >>>>>> I believe that they are already disabled. >>>>>> >>>>>> Here is the ethtool output: >>>>>> >>>>>> # ethtool --show-offload eth1 >>>>>> Features for eth1: >>>>>> rx-checksumming: on >>>>>> tx-checksumming: on >>>>>> tx-checksum-ipv4: off >>>>>> tx-checksum-unneeded: off >>>>>> tx-checksum-ip-generic: on >>>>>> tx-checksum-ipv6: off >>>>>> tx-checksum-fcoe-crc: off [fixed] >>>>>> tx-checksum-sctp: off [fixed] >>>>>> scatter-gather: on >>>>>> tx-scatter-gather: on >>>>>> tx-scatter-gather-fraglist: off [fixed] >>>>>> tcp-segmentation-offload: on >>>>>> tx-tcp-segmentation: on >>>>>> tx-tcp-ecn-segmentation: off >>>>>> tx-tcp6-segmentation: on >>>>>> udp-fragmentation-offload: off [fixed] >>>>>> generic-segmentation-offload: on >>>>>> generic-receive-offload: on >>>>>> large-receive-offload: off [fixed] >>>>>> rx-vlan-offload: on [fixed] >>>>>> tx-vlan-offload: on [fixed] >>>>>> ntuple-filters: off [fixed] >>>>>> receive-hashing: off [fixed] >>>>>> highdma: on [fixed] >>>>>> rx-vlan-filter: off [fixed] >>>>>> vlan-challenged: off [fixed] >>>>>> tx-lockless: off [fixed] >>>>>> netns-local: off [fixed] >>>>>> tx-gso-robust: off [fixed] >>>>>> tx-fcoe-segmentation: off [fixed] >>>>>> tx-gre-segmentation: off [fixed] >>>>>> tx-udp_tnl-segmentation: off [fixed] >>>>>> fcoe-mtu: off [fixed] >>>>>> loopback: off [fixed] >>>>>> >>>>>> Regards, >>>>>> >>>>>> George >>>>>> >>>>>>> Disable offloading on the nodes with: ethtool -K >>>>>>> interfaceName >>>>>>> gro >>>>>>> off >>>>>>> gso off tso off >>>>>>> >>>>>>> And then try it again >>>>>>> El 16/12/2014 18:36, "Georgios Dimitrakakis" escribió: >>>>>>> >>>>>>>> Hi all! >>>>>>>> >>>>>>>> In my OpenStack installation (Icehouse and use nova >>>>>>>> legacy >>>>>>>> networking) the VMs are talking to each other over a >>>>>>>> 1Gbps >>>>>>>> network >>>>>>>> link. >>>>>>>> >>>>>>>> My issue is that although file transfers between >>>>>>>> physical >>>>>>>> (hypervisor) nodes can saturate that link transfers >>>>>>>> between >>>>>>>> VMs >>>>>>>> reach very lower speeds e.g. 30MB/s (approx. 240Mbps). >>>>>>>> >>>>>>>> My tests are performed by scping a large image file >>>>>>>> (approx. >>>>>>>> 4GB) >>>>>>>> between the nodes and between the VMs. >>>>>>>> >>>>>>>> I have updated my images to use e1000 nic driver but >>>>>>>> the >>>>>>>> results >>>>>>>> remain the same. >>>>>>>> >>>>>>>> What are any other limiting factors? >>>>>>>> >>>>>>>> Does it has to do with the disk driver I am using? Does >>>>>>>> it >>>>>>>> play >>>>>>>> significant role the filesystem of the hypervisor node? >>>>>>>> >>>>>>>> Any ideas on how to approach the saturation of the >>>>>>>> 1Gbps >>>>>>>> link? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> George >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: >>>>>>> >>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> [2] >>>>>>> >>>>>>>> [1] >>>>>>>> [1] [1] >>>>>>>> Post to : openstack at lists.openstack.org [1] [2] >>>>>>>> [2] [2] >>>>>>>> Unsubscribe : >>>>>>> >>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> [3] >>>>>>> >>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> [4] >>>>>>> [4] >>>>>>> [2] mailto:openstack at lists.openstack.org [4] [5] [5] >>>>>>> [3] >>>>>>> >>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> [5] >>>>>>> [6] >>>>>>> [6] >>>>>>> [4] mailto:giorgis at acmac.uoc.gr [6] [7] [7] >>>>> >>>>> Links: >>>>> ------ >>>>> [1] >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>> [7] >>>>> [8] >>>>> [2] mailto:openstack at lists.openstack.org [8] [9] >>>>> [3] >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>> [9] >>>>> [10] >>>>> [4] >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>> [10] >>>>> [11] >>>>> [5] mailto:openstack at lists.openstack.org [11] [12] >>>>> [6] >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>> [12] >>>>> [13] >>>>> [7] mailto:giorgis at acmac.uoc.gr [13] [14] >>>>> [8] mailto:giorgis at acmac.uoc.gr [14] [15] >>> >>> Links: >>> ------ >>> [1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> [15] >>> [2] mailto:openstack at lists.openstack.org [16] >>> [3] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> [17] >>> [4] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> [18] >>> [5] mailto:openstack at lists.openstack.org [19] >>> [6] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> [20] >>> [7] mailto:giorgis at acmac.uoc.gr [21] >>> [8] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> [22] >>> [9] mailto:openstack at lists.openstack.org [23] >>> [10] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [24] >>> [11] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [25] >>> [12] mailto:openstack at lists.openstack.org [26] >>> [13] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [27] >>> [14] mailto:giorgis at acmac.uoc.gr [28] >>> [15] mailto:giorgis at acmac.uoc.gr [29] >>> [16] mailto:giorgis at acmac.uoc.gr [30] > > > Links: > ------ > [1] mailto:openstack at lists.openstack.org > [2] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [3] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [4] mailto:openstack at lists.openstack.org > [5] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [6] mailto:giorgis at acmac.uoc.gr > [7] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [8] mailto:openstack at lists.openstack.org > [9] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [10] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [11] mailto:openstack at lists.openstack.org > [12] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [13] mailto:giorgis at acmac.uoc.gr > [14] mailto:giorgis at acmac.uoc.gr > [15] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [16] mailto:openstack at lists.openstack.org > [17] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [19] mailto:openstack at lists.openstack.org > [20] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [21] mailto:giorgis at acmac.uoc.gr > [22] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [23] mailto:openstack at lists.openstack.org > [24] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [25] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [26] mailto:openstack at lists.openstack.org > [27] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > [28] mailto:giorgis at acmac.uoc.gr > [29] mailto:giorgis at acmac.uoc.gr > [30] mailto:giorgis at acmac.uoc.gr > [31] mailto:giorgis at acmac.uoc.gr --