Open Stack

Hi Jon -

From what I understand, while you might have gone to the trouble of configuring a lossless data centre ethernet, that guarantee of packet loss ends at the hypervisor. OVS (and other virtual switches) will drop packets rather than exert back pressure.

I saw a useful paper from IBM Zurich on developing a flow-controlled virtual switch:

http://researcher.ibm.com/researcher/files/zurich-DCR/Got%20Loss%20Get%20zOVN.pdf <http://researcher.ibm.com/researcher/files/zurich-DCR/Got%20Loss%20Get%20zOVN.pdf>

It’s a bit dated (2013) but may still apply.

If you figure out a way of preventing this with modern OVS, I’d be very interested to know.

Best wishes,
Stig


> On 21 Jun 2017, at 16:24, Jonathan Proulx <jon at csail.mit.edu> wrote:
> 
> On Wed, Jun 21, 2017 at 02:39:23AM -0700, Kevin Benton wrote:
> :Are there any events going on during these outages that would cause
> :reprogramming by the Neutron agent? (e.g. port updates) If not, it's likely
> :an OVS issue and you might want to cross-post to the ovs-discuss mailing
> :list.
> 
> Guess I'll have to wander deeper into OVS land.
> 
> No agent updates and nothing in ovs logs (at INFO), flipping to Debug
> and there's so many messages they get dropped:
> 
> 017-06-21T15:15:36.972Z|00794|dpif(handler12)|DBG|Dropped 35 log messages in last 0 seconds (most recently, 0 seconds ago) due to excessive rate
> 
> /me wanders over to ovs-discuss
> 
> Thanks,
> -Jon
> 
> :Can you check the vswitch logs during the packet loss to see if there are
> :any messages indicating a reason? If that doesn't show anything and it can
> :be reliably reproduced, it might be worth increasing the logging for the
> :vswitch to debug.
> :
> :
> :
> :On Tue, Jun 20, 2017 at 12:36 PM, Jonathan Proulx <jon at csail.mit.edu> wrote:
> :
> :> Hi All,
> :>
> :> I have a very busy VM (well one of my users does I don't have access
> :> but do have cooperative and copentent admin to interact with on th
> :> eother end).
> :>
> :> At peak times it *sometimes* misses packets.  I've been didding in for
> :> a bit ant it looks like they get dropped in OVS land.
> :>
> :> The VM's main function in life is to pull down webpages from other
> :> sites and analyze as requested.  During peak times ( EU/US working
> :> hours ) it sometimes hangs some requests and sometimes fails.
> :>
> :> Looking at traffic the out bound SYN request from VM is always good
> :> and returning ACK always gets to physical interface of the hypervisosr
> :> (on a provider vlan).
> :>
> :> When packets get dropped they do not make it to the qvoXXXXXXXX-XX on
> :> the integration bridge.
> :>
> :> My suspicion is that OVS isn't keeping up eth1-br flow rules remaping
> :> from external to internal vlan-id but neither quite sure how to prove
> :> that or what to do about it.
> :>
> :> My initial though had been to blame contrack but drops are happening
> :> before the iptables rules and while there's a lot of connections on
> :> this hypervisor:
> :>
> :> net.netfilter.nf_conntrack_count = 351880
> :>
> :> There should be plent of overhead to handle:
> :>
> :> net.netfilter.nf_conntrack_max = 1048576
> :>
> :> Anyone have thought son where to go with this?
> :>
> :> version details:
> :> Ubuntu 14.04
> :> OpenStack Mitaka
> :> ovs-vsctl (Open vSwitch) 2.5.0
> :>
> :> Thanks,
> :> -Jon
> :>
> :> --
> :>
> :> _______________________________________________
> :> OpenStack-operators mailing list
> :> OpenStack-operators at lists.openstack.org
> :> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> :>
> 
> -- 
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20170621/4473e82a/attachment.html>