<div dir="ltr"><div>This one was tricky. :-)</div><div><br></div><div>This failure can be produced consistently by running the following two tests:<br></div><div>neutron.tests.unit.brocade.test_brocade_plugin.TestBrocadePortsV2.test_delete_network_port_exists_owned_by_network<br></div><div><div>neutron.tests.unit.db.test_migration.TestModelsMigrationsSyncMl2Psql.test_models_sync</div></div><div><br></div><div>This failure behavior started after change I6f67bb430c50ddacb2d53398de75fb5f494964a0 to use oslo.db for all of neutron connection handling instead of SQLAlchemy.[1] The failure message returned from the DB layer is very misleading. If you remove the catch that converts to that generic error about username/pass and DB being wrong/missing, you will get the real following error:</div><div><br></div><div><div>OperationalError: (OperationalError) database "dzmhwmgrou" is being accessed by other users</div><div>DETAIL: There are 1 other session(s) using the database.</div><div> 'drop database dzmhwmgrou;' {}</div></div><div><br></div><div><br></div><div>What is happening is that the oslo.db test case cleanup code is trying to destroy the db while a separate sqlalchemy engine (created from the alembic migration code) still has a connection to the db. The first test is required to help trigger the failure either because of imports or the run setting up database connections causing things to be cached in a module flag somewhere. I haven't looked into the exact source.</div><div><br></div><div>Here is the diff to fix your patch to pass the same session into the alembic migration code that is setup and torn down by the test case. This should allow you to proceed forward with your work.</div><div><br></div><div><br></div><div><div>~/code/neutron$ git diff</div><div>diff --git a/neutron/tests/unit/db/test_migration.py b/neutron/tests/unit/db/test_migration.py</div><div>index 6db8ae0..c29ab67 100644</div><div>--- a/neutron/tests/unit/db/test_migration.py</div><div>+++ b/neutron/tests/unit/db/test_migration.py</div><div>@@ -136,9 +136,12 @@ class ModelsMigrationsSyncMixin(object):</div><div> self.alembic_config.neutron_config = cfg.CONF</div><div><br></div><div> def db_sync(self, engine):</div><div>- cfg.CONF.set_override('connection', engine.url, group='database')</div><div>- migration.do_alembic_command(self.alembic_config, 'upgrade', 'head')</div><div>- cfg.CONF.clear_override('connection', group='database')</div><div>+ with mock.patch(</div><div>+ 'oslo.db.sqlalchemy.session.create_engine',</div><div>+ return_value=self.get_engine()</div><div>+ ):</div><div>+ migration.do_alembic_command(self.alembic_config,</div><div>+ 'upgrade', 'head')</div><div><br></div><div> def get_engine(self):</div><div> return self.engine</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>1. <a href="https://review.openstack.org/#/c/110016/" target="_blank">https://review.openstack.org/#/c/110016/</a></div><div><br></div><div>--</div><div>Kevin Benton</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 12, 2014 at 2:15 AM, Anna Kamyshnikova <span dir="ltr"><<a href="mailto:akamyshnikova@mirantis.com" target="_blank">akamyshnikova@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This is implementing ModelsMigrationsSync test from oslo [1]. For running it locally on Postgres you have to do the following things (it is mentioned in comments to test): <div><div> </div><div>For the opportunistic testing you need to set up a db named</div><div> 'openstack_citest' with user 'openstack_citest' and password</div><div> 'openstack_citest' on localhost.</div><div> The test will then use that db and user/password combo to run the tests.</div><div><br></div><div>For PostgreSQL on Ubuntu this can be done with the following commands::</div><div><br></div><div> sudo -u postgres psql</div><div> postgres=# create user openstack_citest with createdb login password</div><div> 'openstack_citest';</div><div> postgres=# create database openstack_citest with owner</div><div> openstack_citest;</div><div><br></div><div>For MySQL on Ubuntu this can be done with the following commands::<br></div><div><div><br></div><div> mysql -u root</div><div> >create database openstack_citest;</div><div> >grant all privilleges on openstack_citest.* to</div><div> openstack_citest@localhost identified by 'opensatck_citest';</div></div><div><br></div><div>As I said this error appeared only three weeks ago, although I'm working on this test since 29 of April, it passed Jenkins in August without any problems. Postgres is available there. </div><div><br></div><div>[1] - <a href="https://github.com/openstack/oslo.db/blob/master/oslo/db/sqlalchemy/test_migrations.py#L277" target="_blank">https://github.com/openstack/oslo.db/blob/master/oslo/db/sqlalchemy/test_migrations.py#L277</a></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 12, 2014 at 12:28 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Can you explain a bit about that test? I'm having trouble reproducing it.<div>On the system (upstream Jenkins) that it's failing on, is postgres available with that database?</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Sep 11, 2014 at 7:07 AM, Anna Kamyshnikova <span dir="ltr"><<a href="mailto:akamyshnikova@mirantis.com" target="_blank">akamyshnikova@mirantis.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Hello everyone!</span></p><br><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">I'm working on implementing test in Neutron that checks that models are synchronized with database state [1] [2]. This is very important change as during Juno cycle big changes of database structure were done. </span></p><br><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">I was working on it for rather long time but about three weeks ago strange error appeared [3], using AssertionPool shows [4]. The problem is that somehow there are more than one connection to database from each test. I tried to use locks from lockutils, but it didn’t help. On db meeting we decided to add TestCase just for one Ml2 plugin for starters, and then continue working on this strange error, that is why there are two change requests [1] and [2]. But I found out that somehow even one testcase fails with the same error [5] from time to time.</span></p><br><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">I’m asking for any suggestions that could be done in this case. It is very important to get at least [1] merged in Juno.</span></p><br><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">[1] - <a href="https://review.openstack.org/76520" target="_blank">https://review.openstack.org/76520</a></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">[2] - <a href="https://review.openstack.org/120040" target="_blank">https://review.openstack.org/120040</a></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">[3] - <a href="http://paste.openstack.org/show/110158/" target="_blank">http://paste.openstack.org/show/110158/</a></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">[4] - <a href="http://paste.openstack.org/show/110159/" target="_blank">http://paste.openstack.org/show/110159/</a></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">[5] - <a href="http://logs.openstack.org/20/76520/68/check/gate-neutron-python27/63938f9/testr_results.html.gz" target="_blank">http://logs.openstack.org/20/76520/68/check/gate-neutron-python27/63938f9/testr_results.html.gz</a></span></p><br><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">Regards,</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap;background-color:transparent">Ann</span></p><br><br></span></div></div>
<br></div></div>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</font></span></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>