<div dir="ltr">I think the first workaround is the solution we're looking for as it better reflects the fact that opencontrail is a db-less plugin.<div>I hope it will be the easier too, but you can never be too sure with neutron unit tests.</div><div><br></div><div>Salvatore</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 4 May 2015 at 12:56, Pavel Bondar <span dir="ltr"><<a href="mailto:pbondar@infoblox.com" target="_blank">pbondar@infoblox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Kevin,<br>
<br>
Thanks for your answer, that is what I was looking for!<br>
I'll check with you in irc to decide which workaround is better:<br>
1. Mocking NeutronDbSubnet fetch_subnet for opencontrail tests.<br>
2. Using session.query() directly in NeutronDbSubnet fetch_subnet.<br>
<br>
- Pavel Bondar<br>
<div><div class="h5"><br>
On 30.04.2015 22:46, Kevin Benton wrote:<br>
> The OpenContrail plugin itself doesn't even use the Neutron DB. I<br>
> believe what you are observing is a side effect of the fake server they<br>
> have for their tests, which does inherit the neutron DB.<br>
><br>
> When you call a method on the core plugin in the contrail unit test<br>
> case, it will go through their request logic and will be piped into the<br>
> fake server. During this time, the db session that was associated with<br>
> the original context passed to the core plugin will be lost do to its<br>
> conversion to a dict.[1, 2]<br>
><br>
> So I believe what you're seeing is this.<br>
><br>
> 1. The FakeServer gets create_port called and starts its transactions.<br>
> 2. It now hits the ipam driver which calls out to the neutron manager to<br>
> get the core plugin handle, which is actually the contrail plugin and<br>
> not the FakeServer.<br>
> 3. IPAM calls _get_subnet on the contrail plugin, which serializes the<br>
> context[1] and sends it to the FakeServer.<br>
> 4. The FakeServer code receives the request and deserializes the<br>
> context[2], which no longer has the db session.<br>
> 5. The FakeServer then ends up starting a new session to read the<br>
> subnet, which will interfere with the transaction you created the port<br>
> under since they are from the same engine.<br>
><br>
> This is why you can query the DB directly rather than calling the core<br>
> plugin. The good news is that you don't have to worry because the actual<br>
> contrail plugin won't be using any of this logic so you're not actually<br>
> breaking anything.<br>
><br>
> I think what you'll want to do is add a mock.patch for the<br>
> NeutronDbSubnet fetch_subnet method to monkey patch in a reference to<br>
> their FakeServer's _get_subnet method. Ping me on IRC (kevinbenton) if<br>
> you need help.<br>
><br>
> 1.<br>
> <a href="https://github.com/openstack/neutron/blob/master/neutron/plugins/opencontrail/contrail_plugin.py#L111" target="_blank">https://github.com/openstack/neutron/blob/master/neutron/plugins/opencontrail/contrail_plugin.py#L111</a><br>
> 2.<br>
> <a href="https://github.com/openstack/neutron/blob/master/neutron/tests/unit/plugins/opencontrail/test_contrail_plugin.py#L121" target="_blank">https://github.com/openstack/neutron/blob/master/neutron/tests/unit/plugins/opencontrail/test_contrail_plugin.py#L121</a><br>
><br>
> On Thu, Apr 30, 2015 at 6:37 AM, Pavel Bondar <<a href="mailto:pbondar@infoblox.com">pbondar@infoblox.com</a><br>
</div></div><div><div class="h5">> <mailto:<a href="mailto:pbondar@infoblox.com">pbondar@infoblox.com</a>>> wrote:<br>
><br>
> Hi,<br>
><br>
> I am debugging issue observed in OpenContrail tests[1] and so far it<br>
> does not look obvious.<br>
><br>
> Issue:<br>
><br>
> In create_port[2] new transaction is started.<br>
> Port gets created, but disappears right after reading subnet from plugin<br>
> in reference ipam driver[3]:<br>
><br>
> > plugin = manager.NeutronManager.get_plugin()<br>
> > return plugin._get_subnet(context, id)<br>
><br>
> Port no longer seen in transaction, like it never existed before<br>
> (magic?). As a result inserting IPAllocation fails with foreing key<br>
> constraint error:<br>
><br>
> DBReferenceError: (IntegrityError) FOREIGN KEY constraint failed<br>
> u'INSERT INTO ipallocations (port_id, ip_address, subnet_id, network_id)<br>
> VALUES (?, ?, ?, ?)' ('aba6eaa2-2b2f-4ab9-97b0-4d8a36659363',<br>
> u'10.0.0.2', u'be7bb05b-d501-4cf3-a29a-3861b3b54950',<br>
> u'169f6a61-b5d0-493a-b7fa-74fd5b445c84')<br>
> }}}<br>
><br>
> Only OpenContrail tests fail with that error (116 failures[1]). Tests<br>
> for other plugin passes fine. As I see OpenContrail is different from<br>
> other plugins: each call to plugin is wrapped into http request, so<br>
> getting subnet happens in another transaction. In tests requests.post()<br>
> is mocked and http call gets translated into self.get_subnet(...).<br>
> Stack trace from plugin._get_subnet() to db_base get_subnet() in open<br>
> contrail tests looks next[4].<br>
><br>
> Also single test failure with full db debug was uploaded for<br>
> investigation[5]:<br>
> - Port is inserted at 362.<br>
> - Subnet is read by plugin at 384.<br>
> - IPAllocation was tried to be inserted at 407.<br>
> Between Port and IPAllocation insert no COMMIT/ROLLBACK or delete<br>
> statement were issued, so can't find explanation why port no longer<br>
> exists on IPAllocation insert step.<br>
> Am I missing something obvious?<br>
><br>
> For now I have several workarounds, which are basically do not use<br>
> plugin._get_subnet(). Direct session.query() works without such side<br>
> effects.<br>
> But this issue bothers me much since I can't explain why it even happens<br>
> in OpenContrail tests.<br>
> Any ideas are welcome!<br>
><br>
> My best theory for now: OpenContrail silently wipes currently running<br>
> transaction in tests (in this case it doesn't sound good).<br>
><br>
> Anyone can checkout and debug patch set 50 (where issue is observed)<br>
> from review page[6].<br>
><br>
> Thank you in advance.<br>
><br>
> - Pavel Bondar<br>
><br>
> [1]<br>
> <a href="http://logs.openstack.org/36/153236/50/check/gate-neutron-python27/dd83d43/testr_results.html.gz" target="_blank">http://logs.openstack.org/36/153236/50/check/gate-neutron-python27/dd83d43/testr_results.html.gz</a><br>
> [2]<br>
> <a href="https://review.openstack.org/#/c/153236/50/neutron/db/db_base_plugin_v2.py" target="_blank">https://review.openstack.org/#/c/153236/50/neutron/db/db_base_plugin_v2.py</a><br>
> line 1578 / line 1857<br>
> [3]<br>
> <a href="https://review.openstack.org/#/c/153236/50/neutron/ipam/drivers/neutrondb_ipam/driver.py" target="_blank">https://review.openstack.org/#/c/153236/50/neutron/ipam/drivers/neutrondb_ipam/driver.py</a><br>
> line 50<br>
> [4] <a href="http://pastebin.com/n0AqhC5x" target="_blank">http://pastebin.com/n0AqhC5x</a><br>
> [5] <a href="http://pastebin.com/BDBAXFy9" target="_blank">http://pastebin.com/BDBAXFy9</a><br>
> [6] <a href="http://logs.openstack.org/36/153236/" target="_blank">http://logs.openstack.org/36/153236/</a><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</div></div>> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<div class="HOEnZb"><div class="h5">> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Kevin Benton<br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>