Open Stack

Hi Kevin,

Thanks for your answer, that is what I was looking for!
I'll check with you in irc to decide which workaround is better:
1. Mocking NeutronDbSubnet fetch_subnet for opencontrail tests.
2. Using session.query() directly in NeutronDbSubnet fetch_subnet.

- Pavel Bondar

On 30.04.2015 22:46, Kevin Benton wrote:
> The OpenContrail plugin itself doesn't even use the Neutron DB. I
> believe what you are observing is a side effect of the fake server they
> have for their tests, which does inherit the neutron DB.
> 
> When you call a method on the core plugin in the contrail unit test
> case, it will go through their request logic and will be piped into the
> fake server. During this time, the db session that was associated with
> the original context passed to the core plugin will be lost do to its
> conversion to a dict.[1, 2]
> 
> So I believe what you're seeing is this. 
> 
> 1. The FakeServer gets create_port called and starts its transactions. 
> 2. It now hits the ipam driver which calls out to the neutron manager to
> get the core plugin handle, which is actually the contrail plugin and
> not the FakeServer.
> 3. IPAM calls _get_subnet on the contrail plugin, which serializes the
> context[1] and sends it to the FakeServer.
> 4. The FakeServer code receives the request and deserializes the
> context[2], which no longer has the db session.
> 5. The FakeServer then ends up starting a new session to read the
> subnet, which will interfere with the transaction you created the port
> under since they are from the same engine.
> 
> This is why you can query the DB directly rather than calling the core
> plugin. The good news is that you don't have to worry because the actual
> contrail plugin won't be using any of this logic so you're not actually
> breaking anything.
> 
> I think what you'll want to do is add a mock.patch for the
> NeutronDbSubnet fetch_subnet method to monkey patch in a reference to
> their FakeServer's _get_subnet method. Ping me on IRC (kevinbenton) if
> you need help.
> 
> 1.
> https://github.com/openstack/neutron/blob/master/neutron/plugins/opencontrail/contrail_plugin.py#L111
> 2.
> https://github.com/openstack/neutron/blob/master/neutron/tests/unit/plugins/opencontrail/test_contrail_plugin.py#L121
> 
> On Thu, Apr 30, 2015 at 6:37 AM, Pavel Bondar <pbondar at infoblox.com
> <mailto:pbondar at infoblox.com>> wrote:
> 
>     Hi,
> 
>     I am debugging issue observed in OpenContrail tests[1] and so far it
>     does not look obvious.
> 
>     Issue:
> 
>     In create_port[2] new transaction is started.
>     Port gets created, but disappears right after reading subnet from plugin
>     in reference ipam driver[3]:
> 
>     >    plugin = manager.NeutronManager.get_plugin()
>     >    return plugin._get_subnet(context, id)
> 
>     Port no longer seen in transaction, like it never existed before
>     (magic?). As a result inserting IPAllocation fails with foreing key
>     constraint error:
> 
>     DBReferenceError: (IntegrityError) FOREIGN KEY constraint failed
>     u'INSERT INTO ipallocations (port_id, ip_address, subnet_id, network_id)
>     VALUES (?, ?, ?, ?)' ('aba6eaa2-2b2f-4ab9-97b0-4d8a36659363',
>     u'10.0.0.2', u'be7bb05b-d501-4cf3-a29a-3861b3b54950',
>     u'169f6a61-b5d0-493a-b7fa-74fd5b445c84')
>     }}}
> 
>     Only OpenContrail tests fail with that error (116 failures[1]). Tests
>     for other plugin passes fine. As I see OpenContrail is different from
>     other plugins: each call to plugin is wrapped into http request, so
>     getting subnet happens in another transaction. In tests requests.post()
>     is mocked and http call gets translated into self.get_subnet(...).
>     Stack trace from plugin._get_subnet() to db_base get_subnet() in open
>     contrail tests looks next[4].
> 
>     Also single test failure with full db debug was uploaded for
>     investigation[5]:
>     - Port is inserted at 362.
>     - Subnet is read by plugin at 384.
>     - IPAllocation was tried to be inserted at 407.
>     Between Port and IPAllocation insert no COMMIT/ROLLBACK or delete
>     statement were issued, so can't find explanation why port no longer
>     exists on IPAllocation insert step.
>     Am I missing something obvious?
> 
>     For now I have several workarounds, which are basically do not use
>     plugin._get_subnet(). Direct session.query() works without such side
>     effects.
>     But this issue bothers me much since I can't explain why it even happens
>     in OpenContrail tests.
>     Any ideas are welcome!
> 
>     My best theory for now: OpenContrail silently wipes currently running
>     transaction in tests (in this case it doesn't sound good).
> 
>     Anyone can checkout and debug patch set 50 (where issue is observed)
>     from review page[6].
> 
>     Thank you in advance.
> 
>     - Pavel Bondar
> 
>     [1]
>     http://logs.openstack.org/36/153236/50/check/gate-neutron-python27/dd83d43/testr_results.html.gz
>     [2]
>     https://review.openstack.org/#/c/153236/50/neutron/db/db_base_plugin_v2.py
>     line 1578 / line 1857
>     [3]
>     https://review.openstack.org/#/c/153236/50/neutron/ipam/drivers/neutrondb_ipam/driver.py
>     line 50
>     [4] http://pastebin.com/n0AqhC5x
>     [5] http://pastebin.com/BDBAXFy9
>     [6] http://logs.openstack.org/36/153236/
> 
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> -- 
> Kevin Benton
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>