[openstack-dev] [Neutron] _get_subnet() in OpenContrail tests results in port deletion
Pavel Bondar
pbondar at infoblox.com
Tue May 5 11:18:19 UTC 2015
Salvatore,
Agree, I checked with Kevin and hi also prefer the first workaround.
Patch set 55 was uploaded to [1] implementing this solution.
UT passed fine locally, waiting results from jenkins.
[1] https://review.openstack.org/#/c/153236/
Thanks,
Pavel
On 04.05.2015 23:58, Salvatore Orlando wrote:
> I think the first workaround is the solution we're looking for as it
> better reflects the fact that opencontrail is a db-less plugin.
> I hope it will be the easier too, but you can never be too sure with
> neutron unit tests.
>
> Salvatore
>
> On 4 May 2015 at 12:56, Pavel Bondar <pbondar at infoblox.com
> <mailto:pbondar at infoblox.com>> wrote:
>
> Hi Kevin,
>
> Thanks for your answer, that is what I was looking for!
> I'll check with you in irc to decide which workaround is better:
> 1. Mocking NeutronDbSubnet fetch_subnet for opencontrail tests.
> 2. Using session.query() directly in NeutronDbSubnet fetch_subnet.
>
> - Pavel Bondar
>
> On 30.04.2015 22:46, Kevin Benton wrote:
> > The OpenContrail plugin itself doesn't even use the Neutron DB. I
> > believe what you are observing is a side effect of the fake server
> they
> > have for their tests, which does inherit the neutron DB.
> >
> > When you call a method on the core plugin in the contrail unit test
> > case, it will go through their request logic and will be piped
> into the
> > fake server. During this time, the db session that was associated with
> > the original context passed to the core plugin will be lost do to its
> > conversion to a dict.[1, 2]
> >
> > So I believe what you're seeing is this.
> >
> > 1. The FakeServer gets create_port called and starts its transactions.
> > 2. It now hits the ipam driver which calls out to the neutron
> manager to
> > get the core plugin handle, which is actually the contrail plugin and
> > not the FakeServer.
> > 3. IPAM calls _get_subnet on the contrail plugin, which serializes the
> > context[1] and sends it to the FakeServer.
> > 4. The FakeServer code receives the request and deserializes the
> > context[2], which no longer has the db session.
> > 5. The FakeServer then ends up starting a new session to read the
> > subnet, which will interfere with the transaction you created the port
> > under since they are from the same engine.
> >
> > This is why you can query the DB directly rather than calling the core
> > plugin. The good news is that you don't have to worry because the
> actual
> > contrail plugin won't be using any of this logic so you're not
> actually
> > breaking anything.
> >
> > I think what you'll want to do is add a mock.patch for the
> > NeutronDbSubnet fetch_subnet method to monkey patch in a reference to
> > their FakeServer's _get_subnet method. Ping me on IRC (kevinbenton) if
> > you need help.
> >
> > 1.
> >
> https://github.com/openstack/neutron/blob/master/neutron/plugins/opencontrail/contrail_plugin.py#L111
> > 2.
> >
> https://github.com/openstack/neutron/blob/master/neutron/tests/unit/plugins/opencontrail/test_contrail_plugin.py#L121
> >
> > On Thu, Apr 30, 2015 at 6:37 AM, Pavel Bondar
> <pbondar at infoblox.com <mailto:pbondar at infoblox.com>
> > <mailto:pbondar at infoblox.com <mailto:pbondar at infoblox.com>>> wrote:
> >
> > Hi,
> >
> > I am debugging issue observed in OpenContrail tests[1] and so
> far it
> > does not look obvious.
> >
> > Issue:
> >
> > In create_port[2] new transaction is started.
> > Port gets created, but disappears right after reading subnet
> from plugin
> > in reference ipam driver[3]:
> >
> > > plugin = manager.NeutronManager.get_plugin()
> > > return plugin._get_subnet(context, id)
> >
> > Port no longer seen in transaction, like it never existed before
> > (magic?). As a result inserting IPAllocation fails with
> foreing key
> > constraint error:
> >
> > DBReferenceError: (IntegrityError) FOREIGN KEY constraint failed
> > u'INSERT INTO ipallocations (port_id, ip_address, subnet_id,
> network_id)
> > VALUES (?, ?, ?, ?)' ('aba6eaa2-2b2f-4ab9-97b0-4d8a36659363',
> > u'10.0.0.2', u'be7bb05b-d501-4cf3-a29a-3861b3b54950',
> > u'169f6a61-b5d0-493a-b7fa-74fd5b445c84')
> > }}}
> >
> > Only OpenContrail tests fail with that error (116
> failures[1]). Tests
> > for other plugin passes fine. As I see OpenContrail is
> different from
> > other plugins: each call to plugin is wrapped into http
> request, so
> > getting subnet happens in another transaction. In tests
> requests.post()
> > is mocked and http call gets translated into self.get_subnet(...).
> > Stack trace from plugin._get_subnet() to db_base get_subnet()
> in open
> > contrail tests looks next[4].
> >
> > Also single test failure with full db debug was uploaded for
> > investigation[5]:
> > - Port is inserted at 362.
> > - Subnet is read by plugin at 384.
> > - IPAllocation was tried to be inserted at 407.
> > Between Port and IPAllocation insert no COMMIT/ROLLBACK or delete
> > statement were issued, so can't find explanation why port no
> longer
> > exists on IPAllocation insert step.
> > Am I missing something obvious?
> >
> > For now I have several workarounds, which are basically do not use
> > plugin._get_subnet(). Direct session.query() works without
> such side
> > effects.
> > But this issue bothers me much since I can't explain why it
> even happens
> > in OpenContrail tests.
> > Any ideas are welcome!
> >
> > My best theory for now: OpenContrail silently wipes currently
> running
> > transaction in tests (in this case it doesn't sound good).
> >
> > Anyone can checkout and debug patch set 50 (where issue is
> observed)
> > from review page[6].
> >
> > Thank you in advance.
> >
> > - Pavel Bondar
> >
> > [1]
> >
> http://logs.openstack.org/36/153236/50/check/gate-neutron-python27/dd83d43/testr_results.html.gz
> > [2]
> >
> https://review.openstack.org/#/c/153236/50/neutron/db/db_base_plugin_v2.py
> > line 1578 / line 1857
> > [3]
> >
> https://review.openstack.org/#/c/153236/50/neutron/ipam/drivers/neutrondb_ipam/driver.py
> > line 50
> > [4] http://pastebin.com/n0AqhC5x
> > [5] http://pastebin.com/BDBAXFy9
> > [6] http://logs.openstack.org/36/153236/
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> > --
> > Kevin Benton
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list