<div dir="auto">No prob. Thanks for replying. </div><div class="gmail_extra"><br><div class="gmail_quote">On May 31, 2017 10:11 AM, "Gustavo Randich" <<a href="mailto:gustavo.randich@gmail.com">gustavo.randich@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Kevin, I confirm that applying the patch the problem is fixed.<div><br></div><div>Sorry for the inconvenience.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 30, 2017 at 9:36 PM, Kevin Benton <span dir="ltr"><<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Do you have that patch already in your environment? If not, can you confirm it fixes the issue?</div><div class="m_-3681328053216849904HOEnZb"><div class="m_-3681328053216849904h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 30, 2017 at 9:49 AM, Gustavo Randich <span dir="ltr"><<a href="mailto:gustavo.randich@gmail.com" target="_blank">gustavo.randich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">While dumping OVS flows as you suggested, we finally found the cause of the problem: our br-ex OVS bridge lacked the secure fail mode configuration.<div><br></div><div>May be the issue is related to this: <a href="https://bugs.launchpad.net/neutron/+bug/1607787" target="_blank">https://bugs.launchpad.net/neu<wbr>tron/+bug/1607787</a></div><div><br></div><div>Thank you</div><div><br></div></div><div class="m_-3681328053216849904m_-169860072123775951HOEnZb"><div class="m_-3681328053216849904m_-169860072123775951h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 26, 2017 at 6:03 AM, Kevin Benton <span dir="ltr"><<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sorry about the long delay.<div><br></div><div>Can you dump the OVS flows before and after the outage? This will let us know if the flows Neutron setup are getting wiped out.</div></div><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405HOEnZb"><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 2, 2017 at 12:26 PM, Gustavo Randich <span dir="ltr"><<a href="mailto:gustavo.randich@gmail.com" target="_blank">gustavo.randich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Kevin, here is some information aout this issue:</div><div><br></div><div>- if the network outage lasts less than ~1 minute, then connectivity to host and instances is automatically restored without problem<br></div></div></div><div><br></div><div>- otherwise:</div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>- upon outage, "ovs-vsctl show" reports "is_connected: true" in all bridges (br-ex / br-int / br-tun)</div><div><br></div><div>- after about ~1 minute, "ovs-vsctl show" ceases to show "is_connected: true" on every bridge</div><div><br></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>- upon restoring physical interface (fix outage)</div><div><br></div><div> - "ovs-vsctl show" now reports "is_connected: true" in all bridges (br-ex / br-int / br-tun)<br></div><div><br></div><div> - access to host and VMs is NOT restored, although some pings are sporadically answered by host (~1 out of 20)</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div>- to restore connectivity, we:<br></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div> - execute "ifdown br-ex; ifup br-ex" -> access to host is restored, but not to VMs<br></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> - restart neutron-openvswitch-agent -> access to VMs is restored</div><div><br></div></blockquote>Thank you!</div><div><br></div><div><br></div><div><div><div><br></div></div></div></div><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750HOEnZb"><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 28, 2017 at 5:07 PM, Kevin Benton <span dir="ltr"><<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">With the network down, does ovs-vsctl show that it is connected to the controller?</div><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795HOEnZb"><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 28, 2017 at 2:21 PM, Gustavo Randich <span dir="ltr"><<a href="mailto:gustavo.randich@gmail.com" target="_blank">gustavo.randich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Exactly, we access via a tagged interface, which is part of br-ex</div><div><br></div><div># ip a show vlan171</div><div>16: vlan171: <BROADCAST,MULTICAST,UP,LOWER_<wbr>UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1</div><div> link/ether 8e:14:8d:c1:1a:5f brd ff:ff:ff:ff:ff:ff</div><div> inet <a href="http://10.171.1.240/20" target="_blank">10.171.1.240/20</a> brd 10.171.15.255 scope global vlan171</div><div> valid_lft forever preferred_lft forever</div><div> inet6 fe80::8c14:8dff:fec1:1a5f/64 scope link</div><div> valid_lft forever preferred_lft forever</div><div><br></div><div># ovs-vsctl show<br></div><div> ...</div><div><div> Bridge br-ex</div><div> Controller "tcp:<a href="http://127.0.0.1:6633" target="_blank">127.0.0.1:6633</a>"</div><div> is_connected: true</div><div> Port "vlan171"<br></div><div> tag: 171</div><div> Interface "vlan171"</div><div> type: internal</div></div><div> ...<br></div><div><br></div></div><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795m_-783380953580201951HOEnZb"><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795m_-783380953580201951h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 28, 2017 at 3:03 PM, Kevin Benton <span dir="ltr"><<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Ok, that's likely not the issue then. I assume the way you access each host is via an IP assigned to an OVS bridge or an interface that somehow depends on OVS? </div><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795m_-783380953580201951m_-4280796470617272085HOEnZb"><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795m_-783380953580201951m_-4280796470617272085h5"><div class="gmail_extra"><br><div class="gmail_quote">On Apr 28, 2017 12:04, "Gustavo Randich" <<a href="mailto:gustavo.randich@gmail.com" target="_blank">gustavo.randich@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Kevin, we are using the default listen address of loopback interface:<div><br></div><div><div># grep -r of_listen_address /etc/neutron</div><div>/etc/neutron/plugins/ml2/openv<wbr>switch_agent.ini:#of_listen_ad<wbr>dress = 127.0.0.1</div></div><div><br></div><div><br></div><div> tcp/<a href="http://127.0.0.1:6640" target="_blank">127.0.0.1:6640</a> -> ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvs<wbr>witch/db.sock --private-key=db:Open_vSwitch,<wbr>SSL,private_key --certificate=db:Open_vSwitch,<wbr>SSL,certificate --bootstrap-ca-cert=db:Open_vS<wbr>witch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitc<wbr>h/ovsdb-server.log --pidfile=/var/run/openvswitch<wbr>/ovsdb-server.pid --detach --monitor<br></div><div><br></div><div>Thanks</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 28, 2017 at 5:00 AM, Kevin Benton <span dir="ltr"><<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Are you using an of_listen_address value of an interface being brought down? </div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795m_-783380953580201951m_-4280796470617272085m_7805883755788286471m_9151329541636933453h5">On Apr 25, 2017 17:34, "Gustavo Randich" <<a href="mailto:gustavo.randich@gmail.com" target="_blank">gustavo.randich@gmail.com</a>> wrote:<br type="attribution"></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-3681328053216849904m_-169860072123775951m_6044387042368577405m_-5407443101878878750m_-1896800966919572795m_-783380953580201951m_-4280796470617272085m_7805883755788286471m_9151329541636933453h5"><div dir="ltr"><div>(using Mitaka / Ubuntu 16 / Neutron DVR / OVS / VXLAN / l2_population)</div><div><br></div><div>This sounds very strange (to me): recently, after a switch outage, we lost connectivity to all our Mitaka hosts. We had to enter via iLO host by host and restart networking service to regain access. Then restart neutron-openvswitch-agent to regain access to VMs.</div><div><br></div><div>At first glance we thought it was a problem with the NIC linux driver of the hosts not detecting link state correctly.</div><div><br></div><div>Then we reproduced the issue simply bringing down physical interfaces for around 5 minutes, then up again. Same issue.</div><div><br></div><div>And then.... we found that if instead of using native (ryu) OpenFlow interface in Neutron Openvswitch we used ovs-ofctl, the problem disappears.</div><div><br></div><div>Any clue?</div><div><br></div><div>Thanks in advance.</div><div><br></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<br></blockquote></div></div>
</blockquote></div><br></div>
</blockquote></div></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div></div>