Open Stack

This one was tricky. :-)

This failure can be produced consistently by running the following two
tests:
neutron.tests.unit.brocade.test_brocade_plugin.TestBrocadePortsV2.test_delete_network_port_exists_owned_by_network
neutron.tests.unit.db.test_migration.TestModelsMigrationsSyncMl2Psql.test_models_sync

This failure behavior started after
change I6f67bb430c50ddacb2d53398de75fb5f494964a0 to use oslo.db for all of
neutron connection handling instead of SQLAlchemy.[1] The failure message
returned from the DB layer is very misleading. If you remove the catch that
converts to that generic error about username/pass and DB being
wrong/missing, you will get the real following error:

OperationalError: (OperationalError) database "dzmhwmgrou" is being
accessed by other users
DETAIL:  There are 1 other session(s) using the database.
 'drop database dzmhwmgrou;' {}


What is happening is that the oslo.db test case cleanup code is trying to
destroy the db while a separate sqlalchemy engine (created from the alembic
migration code) still has a connection to the db. The first test is
required to help trigger the failure either because of imports or the run
setting up database connections causing things to be cached in a module
flag somewhere. I haven't looked into the exact source.

Here is the diff to fix your patch to pass the same session into the
alembic migration code that is setup and torn down by the test case. This
should allow you to proceed forward with your work.


~/code/neutron$ git diff
diff --git a/neutron/tests/unit/db/test_migration.py
b/neutron/tests/unit/db/test_migration.py
index 6db8ae0..c29ab67 100644
--- a/neutron/tests/unit/db/test_migration.py
+++ b/neutron/tests/unit/db/test_migration.py
@@ -136,9 +136,12 @@ class ModelsMigrationsSyncMixin(object):
         self.alembic_config.neutron_config = cfg.CONF

     def db_sync(self, engine):
-        cfg.CONF.set_override('connection', engine.url, group='database')
-        migration.do_alembic_command(self.alembic_config, 'upgrade',
'head')
-        cfg.CONF.clear_override('connection', group='database')
+        with mock.patch(
+            'oslo.db.sqlalchemy.session.create_engine',
+            return_value=self.get_engine()
+        ):
+            migration.do_alembic_command(self.alembic_config,
+                                         'upgrade', 'head')

     def get_engine(self):
         return self.engine




1. https://review.openstack.org/#/c/110016/

--
Kevin Benton

On Fri, Sep 12, 2014 at 2:15 AM, Anna Kamyshnikova <
akamyshnikova at mirantis.com> wrote:

> This is implementing ModelsMigrationsSync test from oslo [1]. For running
> it locally on Postgres you have to do the following things (it is mentioned
> in comments to test):
>
> For the opportunistic testing you need to set up a db named
>     'openstack_citest' with user 'openstack_citest' and password
>     'openstack_citest' on localhost.
>     The test will then use that db and user/password combo to run the
> tests.
>
> For PostgreSQL on Ubuntu this can be done with the following commands::
>
>         sudo -u postgres psql
>         postgres=# create user openstack_citest with createdb login
> password
>                   'openstack_citest';
>         postgres=# create database openstack_citest with owner
>                    openstack_citest;
>
> For MySQL on Ubuntu this can be done with the following commands::
>
>         mysql -u root
>         >create database openstack_citest;
>         >grant all privilleges on openstack_citest.* to
>          openstack_citest at localhost identified by 'opensatck_citest';
>
> As I said this error appeared only three weeks ago, although I'm working
> on this test since 29 of April, it passed Jenkins in August without any
> problems. Postgres is available there.
>
> [1] -
> https://github.com/openstack/oslo.db/blob/master/oslo/db/sqlalchemy/test_migrations.py#L277
>
> On Fri, Sep 12, 2014 at 12:28 PM, Kevin Benton <blak111 at gmail.com> wrote:
>
>> Can you explain a bit about that test? I'm having trouble reproducing it.
>> On the system (upstream Jenkins) that it's failing on, is postgres
>> available with that database?
>>
>> On Thu, Sep 11, 2014 at 7:07 AM, Anna Kamyshnikova <
>> akamyshnikova at mirantis.com> wrote:
>>
>>> Hello everyone!
>>>
>>> I'm working on implementing test in Neutron that checks that models are
>>> synchronized with database state [1] [2]. This is very important change as
>>> during Juno cycle big changes of database structure were done.
>>>
>>> I was working on it for rather long time but about three weeks ago
>>> strange error appeared [3], using AssertionPool shows [4]. The problem is
>>> that somehow there are more than one connection to database from each test.
>>> I tried to use locks from lockutils, but it didn’t help. On db meeting we
>>> decided to add TestCase just for one Ml2 plugin for starters, and then
>>> continue working on this strange error, that is why there are two change
>>> requests [1] and [2]. But I found out that somehow even one testcase fails
>>> with the same error [5] from time to time.
>>>
>>> I’m asking for any suggestions that could be done in this case. It is
>>> very important to get at least [1] merged in Juno.
>>>
>>> [1] - https://review.openstack.org/76520
>>>
>>> [2] - https://review.openstack.org/120040
>>>
>>> [3] - http://paste.openstack.org/show/110158/
>>>
>>> [4] - http://paste.openstack.org/show/110159/
>>>
>>> [5] -
>>> http://logs.openstack.org/20/76520/68/check/gate-neutron-python27/63938f9/testr_results.html.gz
>>>
>>> Regards,
>>>
>>> Ann
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Kevin Benton
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140912/1efbb5fa/attachment.html>