<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>Kevin,<br></div>
<div><br></div>
<div> I have filed a bug with neutron - <a href="https://bugs.launchpad.net/neutron/+bug/1708731">https://bugs.launchpad.net/neutron/+bug/1708731</a> with the below details.<br></div>
<div><br></div>
<div><br></div>
<div>Thanks.</div>
<div><br></div>
<div id="sig20346837"><div class="signature">Aju Francis<br></div>
</div>
<div><br></div>
<div><br></div>
<div>On Fri, Aug 4, 2017, at 06:38 PM, Kevin Benton wrote:<br></div>
<blockquote type="cite"><div dir="ltr"><div>Hey,<br></div>
<div><div><br></div>
<div>This is probably a bug. That exception of <span class="size" style="font-size:12.8px">OVSFWPortNOtFound shouldn't have been leaked that far up at a minimum. Can you file a bug against neutron with the contents of this email here: <a href="https://bugs.launchpad.net/neutron/+filebug">https://bugs.launchpad.net/neutron/+filebug</a></span><br></div>
</div>
<div><span class="size" style="font-size:12.8px"></span><br></div>
<div><span class="size" style="font-size:12.8px">Cheers</span><br></div>
</div>
<div><div><br></div>
<div><div>On Fri, Aug 4, 2017 at 1:52 PM, Aju L Francis <span dir="ltr"><<a href="mailto:aju@iraw.net">aju@iraw.net</a>></span> wrote:<br></div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;"><div>Hello All,<br></div>
<div><br></div>
<div><br></div>
<div>We have VMs running GRE tunnels between them with OVSFW and SG<br></div>
<div> implemented along with GRE conntrack helper loaded on the hypervisor.<br></div>
<div> GRE works as expected but the tunnel breaks whenever there is a<br></div>
<div> neutron ovs agent event causing some exception like the below AMQP<br></div>
<div> timeouts or OVSFW port not found :<br></div>
<div><br></div>
<div> AMQP Timeout :<br></div>
<div><br></div>
<div> 2017-04-07 19:07:03.001 5275 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> MessagingTimeout: Timed out waiting for a reply to message ID<br></div>
<div> 4035644808d24ce9aae65a6ee56702<wbr>1c<br></div>
<div> 2017-04-07 19:07:03.001 5275 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> 2017-04-07 19:07:03.003 5275 WARNING oslo.service.loopingcall [-]<br></div>
<div> Function<br></div>
<div> 'neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent.OVSNeutronAgent._report_<wbr>state'<br></div>
<div> run outlasted interval by 120.01 sec<br></div>
<div> 2017-04-07 19:07:03.041 5275 INFO<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent [-]<br></div>
<div> Agent has just been revived. Doing a full sync.<br></div>
<div> 2017-04-07 19:07:06.747 5275 INFO<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> [req-521c07b4-f53d-4665-b728-<wbr>fc5f00191294 - - - - -] rpc_loop doing a<br></div>
<div> full sync.<br></div>
<div> 2017-04-07 19:07:06.841 5275 INFO<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> [req-521c07b4-f53d-4665-b728-<wbr>fc5f00191294 - - - - -] Agent out of sync<br></div>
<div> with plugin!<br></div>
<div><br></div>
<div> OVSFWPortNOtFound:<br></div>
<div><br></div>
<div> 2017-03-30 18:31:05.048 5160 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> self.firewall.prepare_port_<wbr>filter(device)<br></div>
<div> 2017-03-30 18:31:05.048 5160 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent File<br></div>
<div> "/openstack/venvs/neutron-14.<wbr>0.5/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>openvswitch_firewall/firewall.<wbr>py",<br></div>
<div> line 272, in prepare_port_filter<br></div>
<div> 2017-03-30 18:31:05.048 5160 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> of_port = self.get_or_create_ofport(<wbr>port)<br></div>
<div> 2017-03-30 18:31:05.048 5160 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent File<br></div>
<div> "/openstack/venvs/neutron-14.<wbr>0.5/lib/python2.7/site-<wbr>packages/neutron/agent/linux/<wbr>openvswitch_firewall/firewall.<wbr>py",<br></div>
<div> line 246, in get_or_create_ofport<br></div>
<div> 2017-03-30 18:31:05.048 5160 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent raise<br></div>
<div> OVSFWPortNotFound(port_id=<wbr>port_id)<br></div>
<div> 2017-03-30 18:31:05.048 5160 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> OVSFWPortNotFound: Port 01f7c714-1828-4768-9810-<wbr>a0ec25dd2b92 is not<br></div>
<div> managed by this agent.<br></div>
<div> 2017-03-30 18:31:05.048 5160 ERROR<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> 2017-03-30 18:31:05.072 5160 INFO<br></div>
<div> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br></div>
<div> [req-db74f32b-5370-4a5f-86bf-<wbr>935eba1490d0 - - - - -] Agent out of sync<br></div>
<div> with plugin!<br></div>
<div><br></div>
<div> The agent throws out of sync messages and starts to initialize neutron<br></div>
<div> ports once again along with fresh SG rules.<br></div>
<div><br></div>
<div> 2017-04-07 19:07:07.110 5275 INFO neutron.agent.securitygroups_<wbr>rpc<br></div>
<div> [req-521c07b4-f53d-4665-b728-<wbr>fc5f00191294 - - - - -] Preparing filters<br></div>
<div> for devices set([u'4b14619f-3b9e-4103-<wbr>b9d7-9c7e52c797d8'])<br></div>
<div> 2017-04-07 19:07:07.215 5275 ERROR<br></div>
<div> neutron.agent.linux.<wbr>openvswitch_firewall.firewall<br></div>
<div> [req-521c07b4-f53d-4665-b728-<wbr>fc5f00191294 - - - - -] Initializing port<br></div>
<div> 4b14619f-3b9e-4103-b9d7-<wbr>9c7e52c797d8 that was already initialized.<br></div>
<div><br></div>
<div> During this process, when it prepares new filters for all ports, its<br></div>
<div> marking the conntrack entry for certain GRE connection(high traffic)<br></div>
<div> as invalid.<br></div>
<div><br></div>
<div> root@server:/var/log# conntrack -L -o extended -p gre -f ipv4<br></div>
<div> ipv4 2 gre 47 178 src=1.1.1.203 dst=2.2.2.66 srckey=0x0 dstkey=0x0<br></div>
<div> src=2.2.2.66 dst=1.1.1.203 srckey=0x0 dstkey=0x0 [ASSURED] mark=1<br></div>
<div> zone=5 use=1<br></div>
<div> ipv4 2 gre 47 179 src=5.5.5.104 dst=4.4.4.187 srckey=0x0 dstkey=0x0<br></div>
<div> src=4.4.4.187 dst=5.5.5.104 srckey=0x0 dstkey=0x0 [ASSURED] mark=0<br></div>
<div> zone=5 use=1<br></div>
<div><br></div>
<div> And that connection state remains invalid, unless someone reboots the<br></div>
<div> VM, or flushes the connection directly on the conntrack or through<br></div>
<div> OVS.<br></div>
<div><br></div>
<div> We have a blanket any protocol any port any IP SG rule during this<br></div>
<div> scenario, we even tried adding specific rules to allow IP 47 for GRE.<br></div>
<div> But nothing fixed this problem.<br></div>
<div><br></div>
<div> Was checking for ovs-conntrack helper specific bugs and came across<br></div>
<div> <a href="http://patchwork.ozlabs.org/patch/755615/">patchwork.ozlabs.org/patch/<wbr>755615/</a> - is this bug being<br></div>
<div> triggered in the above scenario ? Is this a bug in the ovs-fw code or<br></div>
<div> this something on the ovs-conntrack implementation.<br></div>
<div><br></div>
<div> OpenStack Version : Newton.<br></div>
<div> Hypervisor OS : Ubuntu 16.04.2<br></div>
<div> Kernel version : 4.4.0-70-generic<br></div>
<div> OVS version : 2.6.1<br></div>
<div><br></div>
<div><br></div>
<div>Thanks.<br></div>
<div><br></div>
<div>Aju Kalarickal<br></div>
<div><br></div>
<div>______________________________<wbr>_________________<br></div>
<div>Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a><br></div>
<div>Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br></div>
<div>Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack</a><br></div>
</blockquote></div>
</div>
</blockquote><div><br></div>
</body>
</html>