Open Stack

Hi,

I am debugging issue observed in OpenContrail tests[1] and so far it
does not look obvious.

Issue:

In create_port[2] new transaction is started.
Port gets created, but disappears right after reading subnet from plugin
in reference ipam driver[3]:

>    plugin = manager.NeutronManager.get_plugin()
>    return plugin._get_subnet(context, id)

Port no longer seen in transaction, like it never existed before
(magic?). As a result inserting IPAllocation fails with foreing key
constraint error:

DBReferenceError: (IntegrityError) FOREIGN KEY constraint failed
u'INSERT INTO ipallocations (port_id, ip_address, subnet_id, network_id)
VALUES (?, ?, ?, ?)' ('aba6eaa2-2b2f-4ab9-97b0-4d8a36659363',
u'10.0.0.2', u'be7bb05b-d501-4cf3-a29a-3861b3b54950',
u'169f6a61-b5d0-493a-b7fa-74fd5b445c84')
}}}

Only OpenContrail tests fail with that error (116 failures[1]). Tests
for other plugin passes fine. As I see OpenContrail is different from
other plugins: each call to plugin is wrapped into http request, so
getting subnet happens in another transaction. In tests requests.post()
is mocked and http call gets translated into self.get_subnet(...).
Stack trace from plugin._get_subnet() to db_base get_subnet() in open
contrail tests looks next[4].

Also single test failure with full db debug was uploaded for
investigation[5]:
- Port is inserted at 362.
- Subnet is read by plugin at 384.
- IPAllocation was tried to be inserted at 407.
Between Port and IPAllocation insert no COMMIT/ROLLBACK or delete
statement were issued, so can't find explanation why port no longer
exists on IPAllocation insert step.
Am I missing something obvious?

For now I have several workarounds, which are basically do not use
plugin._get_subnet(). Direct session.query() works without such side
effects.
But this issue bothers me much since I can't explain why it even happens
in OpenContrail tests.
Any ideas are welcome!

My best theory for now: OpenContrail silently wipes currently running
transaction in tests (in this case it doesn't sound good).

Anyone can checkout and debug patch set 50 (where issue is observed)
from review page[6].

Thank you in advance.

- Pavel Bondar

[1]
http://logs.openstack.org/36/153236/50/check/gate-neutron-python27/dd83d43/testr_results.html.gz
[2]
https://review.openstack.org/#/c/153236/50/neutron/db/db_base_plugin_v2.py
line 1578 / line 1857
[3]
https://review.openstack.org/#/c/153236/50/neutron/ipam/drivers/neutrondb_ipam/driver.py
line 50
[4] http://pastebin.com/n0AqhC5x
[5] http://pastebin.com/BDBAXFy9
[6] http://logs.openstack.org/36/153236/