<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I got some answers from the openvswitch mailing list, essentially indicating the issue is in the connection between neutron-openvswitch-agent and ovs.<div class=""><br class=""></div><div class="">Here’s an output of ovs-vsctl list controller:</div><div class=""><br class=""></div><div class="">_uuid : ff2dca74-9628-43c8-b89c-8d2f1242dd3f<br class="">connection_mode : out-of-band<br class="">controller_burst_limit: []<br class="">controller_rate_limit: []<br class="">enable_async_messages: []<br class="">external_ids : {}<br class="">inactivity_probe : []<br class="">is_connected : false<br class="">local_gateway : []<br class="">local_ip : []<br class="">local_netmask : []<br class="">max_backoff : []<br class="">other_config : {}<br class="">role : other<br class="">status : {last_error="Connection timed out", sec_since_connect="22", sec_since_disconnect="1", state=BACKOFF}<br class="">target : "tcp:127.0.0.1:6633 »</div><div class=""><br class=""></div><div class="">So OVS is still working but the connection between neutron-openvswitch-agent and OVS gets interrupted somehow. It may also be linked to the HA vrrp switching host at random as the connection between both network nodes get severed. We also see SSH lagging momentarily. I’m starting to think that a limit of some kind in linux is reached, preventing connections from happening. However, I don’t think it’s max open file since the number of open files is nowhere close to what I’ve set it.</div><div class=""><br class=""></div><div class="">Ideas?</div><div class=""> <br class=""><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">Jean-Philippe Méthot</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">Openstack system administrator</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">Administrateur système Openstack</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">PlanetHoster inc.</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><br class=""></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><br class="Apple-interchange-newline">
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">Le 26 sept. 2018 à 15:16, Jean-Philippe Méthot <<a href="mailto:jp.methot@planethoster.info" class="">jp.methot@planethoster.info</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Yes, I notice that every time that message appears, at least a few packets get dropped and some of our instances pop up in nagios, even though they are reachable 1 or 2 seconds after. It’s really causing us some issues as we can’t ensure proper network quality for our customers. Have you noticed the same?<div class=""><br class=""></div><div class="">By that point I think it may be best to contact openvswitch directly since it seems to be an issue with their component. I am about to do that and hope I don’t get sent back to the openstack mailing list. I would really like to know what this probe is and why it disconnects constantly under load.</div><div class=""><br class=""><div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Jean-Philippe Méthot</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Openstack system administrator</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Administrateur système Openstack</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">PlanetHoster inc.</div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><br class="Apple-interchange-newline">
</div>
<div class=""><br class=""><blockquote type="cite" class=""><div class="">Le 26 sept. 2018 à 11:48, Simon Leinen <<a href="mailto:simon.leinen@switch.ch" class="">simon.leinen@switch.ch</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><div class="">Jean-Philippe Méthot writes:<br class=""><blockquote type="cite" class="">This particular message makes it sound as if openvswitch is getting overloaded.<br class="">Sep 23 03:54:08 network1 ovsdb-server: ovs|01253|reconnect|ERR|tcp:127.0.0.1:50814: no response to inactivity probe after 5.01 seconds, disconnecting<br class=""></blockquote><br class="">We get these as well :-(<br class=""><br class=""><blockquote type="cite" class="">A lot of those keep appear, and openvswitch always reconnects almost<br class="">instantly though. I’ve done some research about that particular<br class="">message, but it didn’t give me anything I can use to fix it.<br class=""></blockquote><br class="">Would be interested in solutions as well. But I'm sceptical whether<br class="">kernel settings can help here, because the timeout/slowness seems to be<br class="">located in the user-space/control-plane parts of Open vSwitch,<br class="">i.e. OVSDB.<br class="">-- <br class="">Simon.<br class=""><br class=""><blockquote type="cite" class="">Jean-Philippe Méthot<br class="">Openstack system administrator<br class="">Administrateur système Openstack<br class="">PlanetHoster inc.<br class=""></blockquote><br class=""><blockquote type="cite" class=""> Le 25 sept. 2018 à 19:37, Erik McCormick <<a href="mailto:emccormick@cirrusseven.com" class="">emccormick@cirrusseven.com</a>> a écrit :<br class=""></blockquote><br class=""><blockquote type="cite" class=""> Ate you getting any particular log messages that lead you to conclude your issue lies with OVS? I've hit lots of kernel limits under those conditions before OVS itself ever<br class=""> noticed. Anything in dmesg, journal or neutron logs of interest? <br class=""></blockquote><br class=""><blockquote type="cite" class=""> On Tue, Sep 25, 2018, 7:27 PM Jean-Philippe Méthot <<a href="mailto:jp.methot@planethoster.info" class="">jp.methot@planethoster.info</a>> wrote:<br class=""></blockquote><br class=""><blockquote type="cite" class=""> Hi,<br class=""></blockquote><br class=""><blockquote type="cite" class=""> Are there some recommendations regarding kernel settings configuration for openvswitch? We’ve just been hit by what we believe may be an attack of some kind we<br class=""> have never seen before and we’re wondering if there’s a way to optimize our network nodes kernel for openvswitch operation and thus minimize the impact of such an<br class=""> attack, or whatever it was.<br class=""></blockquote><br class=""><blockquote type="cite" class=""> Best regards,<br class=""></blockquote><br class=""><blockquote type="cite" class=""> Jean-Philippe Méthot<br class=""> Openstack system administrator<br class=""> Administrateur système Openstack<br class=""> PlanetHoster inc.<br class=""></blockquote><br class=""><blockquote type="cite" class=""> _______________________________________________<br class=""> OpenStack-operators mailing list<br class=""> <a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class=""> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br class=""></blockquote><br class=""><blockquote type="cite" class="">_______________________________________________<br class="">OpenStack-operators mailing list<br class=""><a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br class=""></blockquote><br class=""></div></div></blockquote></div><br class=""></div></div>_______________________________________________<br class="">OpenStack-operators mailing list<br class=""><a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators<br class=""></div></blockquote></div><br class=""></div></body></html>