<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Jon -<div class=""><br class=""></div><div class="">From what I understand, while you might have gone to the trouble of configuring a lossless data centre ethernet, that guarantee of packet loss ends at the hypervisor. OVS (and other virtual switches) will drop packets rather than exert back pressure.</div><div class=""><br class=""></div><div class="">I saw a useful paper from IBM Zurich on developing a flow-controlled virtual switch:</div><div class=""><br class=""></div><div class=""><a href="http://researcher.ibm.com/researcher/files/zurich-DCR/Got Loss Get zOVN.pdf" class="">http://researcher.ibm.com/researcher/files/zurich-DCR/Got%20Loss%20Get%20zOVN.pdf</a></div><div class=""><br class=""></div><div class=""><div class="">It’s a bit dated (2013) but may still apply.</div></div><div class=""><br class=""></div><div class="">If you figure out a way of preventing this with modern OVS, I’d be very interested to know.</div><div class=""><br class=""></div><div class="">Best wishes,</div><div class="">Stig</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On 21 Jun 2017, at 16:24, Jonathan Proulx <<a href="mailto:jon@csail.mit.edu" class="">jon@csail.mit.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">On Wed, Jun 21, 2017 at 02:39:23AM -0700, Kevin Benton wrote:<br class="">:Are there any events going on during these outages that would cause<br class="">:reprogramming by the Neutron agent? (e.g. port updates) If not, it's likely<br class="">:an OVS issue and you might want to cross-post to the ovs-discuss mailing<br class="">:list.<br class=""><br class="">Guess I'll have to wander deeper into OVS land.<br class=""><br class="">No agent updates and nothing in ovs logs (at INFO), flipping to Debug<br class="">and there's so many messages they get dropped:<br class=""><br class="">017-06-21T15:15:36.972Z|00794|dpif(handler12)|DBG|Dropped 35 log messages in last 0 seconds (most recently, 0 seconds ago) due to excessive rate<br class=""><br class="">/me wanders over to ovs-discuss<br class=""><br class="">Thanks,<br class="">-Jon<br class=""><br class="">:Can you check the vswitch logs during the packet loss to see if there are<br class="">:any messages indicating a reason? If that doesn't show anything and it can<br class="">:be reliably reproduced, it might be worth increasing the logging for the<br class="">:vswitch to debug.<br class="">:<br class="">:<br class="">:<br class="">:On Tue, Jun 20, 2017 at 12:36 PM, Jonathan Proulx <<a href="mailto:jon@csail.mit.edu" class="">jon@csail.mit.edu</a>> wrote:<br class="">:<br class="">:> Hi All,<br class="">:><br class="">:> I have a very busy VM (well one of my users does I don't have access<br class="">:> but do have cooperative and copentent admin to interact with on th<br class="">:> eother end).<br class="">:><br class="">:> At peak times it *sometimes* misses packets. I've been didding in for<br class="">:> a bit ant it looks like they get dropped in OVS land.<br class="">:><br class="">:> The VM's main function in life is to pull down webpages from other<br class="">:> sites and analyze as requested. During peak times ( EU/US working<br class="">:> hours ) it sometimes hangs some requests and sometimes fails.<br class="">:><br class="">:> Looking at traffic the out bound SYN request from VM is always good<br class="">:> and returning ACK always gets to physical interface of the hypervisosr<br class="">:> (on a provider vlan).<br class="">:><br class="">:> When packets get dropped they do not make it to the qvoXXXXXXXX-XX on<br class="">:> the integration bridge.<br class="">:><br class="">:> My suspicion is that OVS isn't keeping up eth1-br flow rules remaping<br class="">:> from external to internal vlan-id but neither quite sure how to prove<br class="">:> that or what to do about it.<br class="">:><br class="">:> My initial though had been to blame contrack but drops are happening<br class="">:> before the iptables rules and while there's a lot of connections on<br class="">:> this hypervisor:<br class="">:><br class="">:> net.netfilter.nf_conntrack_count = 351880<br class="">:><br class="">:> There should be plent of overhead to handle:<br class="">:><br class="">:> net.netfilter.nf_conntrack_max = 1048576<br class="">:><br class="">:> Anyone have thought son where to go with this?<br class="">:><br class="">:> version details:<br class="">:> Ubuntu 14.04<br class="">:> OpenStack Mitaka<br class="">:> ovs-vsctl (Open vSwitch) 2.5.0<br class="">:><br class="">:> Thanks,<br class="">:> -Jon<br class="">:><br class="">:> --<br class="">:><br class="">:> _______________________________________________<br class="">:> OpenStack-operators mailing list<br class="">:> <a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class="">:> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br class="">:><br class=""><br class="">-- <br class=""><br class="">_______________________________________________<br class="">OpenStack-operators mailing list<br class=""><a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators<br class=""></div></div></blockquote></div><br class=""></div></body></html>