Open Stack

Kevin Benton

Thank you very much!!! I lost hope to find a way to fix these tests!

Regards,
Ann Kamyshnikova

On Fri, Sep 12, 2014 at 5:17 PM, Kevin Benton <blak111 at gmail.com> wrote:

> This one was tricky. :-)
>
> This failure can be produced consistently by running the following two
> tests:
>
> neutron.tests.unit.brocade.test_brocade_plugin.TestBrocadePortsV2.test_delete_network_port_exists_owned_by_network
>
> neutron.tests.unit.db.test_migration.TestModelsMigrationsSyncMl2Psql.test_models_sync
>
> This failure behavior started after
> change I6f67bb430c50ddacb2d53398de75fb5f494964a0 to use oslo.db for all of
> neutron connection handling instead of SQLAlchemy.[1] The failure message
> returned from the DB layer is very misleading. If you remove the catch that
> converts to that generic error about username/pass and DB being
> wrong/missing, you will get the real following error:
>
> OperationalError: (OperationalError) database "dzmhwmgrou" is being
> accessed by other users
> DETAIL:  There are 1 other session(s) using the database.
>  'drop database dzmhwmgrou;' {}
>
>
> What is happening is that the oslo.db test case cleanup code is trying to
> destroy the db while a separate sqlalchemy engine (created from the alembic
> migration code) still has a connection to the db. The first test is
> required to help trigger the failure either because of imports or the run
> setting up database connections causing things to be cached in a module
> flag somewhere. I haven't looked into the exact source.
>
> Here is the diff to fix your patch to pass the same session into the
> alembic migration code that is setup and torn down by the test case. This
> should allow you to proceed forward with your work.
>
>
> ~/code/neutron$ git diff
> diff --git a/neutron/tests/unit/db/test_migration.py
> b/neutron/tests/unit/db/test_migration.py
> index 6db8ae0..c29ab67 100644
> --- a/neutron/tests/unit/db/test_migration.py
> +++ b/neutron/tests/unit/db/test_migration.py
> @@ -136,9 +136,12 @@ class ModelsMigrationsSyncMixin(object):
>          self.alembic_config.neutron_config = cfg.CONF
>
>      def db_sync(self, engine):
> -        cfg.CONF.set_override('connection', engine.url, group='database')
> -        migration.do_alembic_command(self.alembic_config, 'upgrade',
> 'head')
> -        cfg.CONF.clear_override('connection', group='database')
> +        with mock.patch(
> +            'oslo.db.sqlalchemy.session.create_engine',
> +            return_value=self.get_engine()
> +        ):
> +            migration.do_alembic_command(self.alembic_config,
> +                                         'upgrade', 'head')
>
>      def get_engine(self):
>          return self.engine
>
>
>
>
> 1. https://review.openstack.org/#/c/110016/
>
> --
> Kevin Benton
>
> On Fri, Sep 12, 2014 at 2:15 AM, Anna Kamyshnikova <
> akamyshnikova at mirantis.com> wrote:
>
>> This is implementing ModelsMigrationsSync test from oslo [1]. For running
>> it locally on Postgres you have to do the following things (it is mentioned
>> in comments to test):
>>
>> For the opportunistic testing you need to set up a db named
>>     'openstack_citest' with user 'openstack_citest' and password
>>     'openstack_citest' on localhost.
>>     The test will then use that db and user/password combo to run the
>> tests.
>>
>> For PostgreSQL on Ubuntu this can be done with the following commands::
>>
>>         sudo -u postgres psql
>>         postgres=# create user openstack_citest with createdb login
>> password
>>                   'openstack_citest';
>>         postgres=# create database openstack_citest with owner
>>                    openstack_citest;
>>
>> For MySQL on Ubuntu this can be done with the following commands::
>>
>>         mysql -u root
>>         >create database openstack_citest;
>>         >grant all privilleges on openstack_citest.* to
>>          openstack_citest at localhost identified by 'opensatck_citest';
>>
>> As I said this error appeared only three weeks ago, although I'm working
>> on this test since 29 of April, it passed Jenkins in August without any
>> problems. Postgres is available there.
>>
>> [1] -
>> https://github.com/openstack/oslo.db/blob/master/oslo/db/sqlalchemy/test_migrations.py#L277
>>
>> On Fri, Sep 12, 2014 at 12:28 PM, Kevin Benton <blak111 at gmail.com> wrote:
>>
>>> Can you explain a bit about that test? I'm having trouble reproducing it.
>>> On the system (upstream Jenkins) that it's failing on, is postgres
>>> available with that database?
>>>
>>> On Thu, Sep 11, 2014 at 7:07 AM, Anna Kamyshnikova <
>>> akamyshnikova at mirantis.com> wrote:
>>>
>>>> Hello everyone!
>>>>
>>>> I'm working on implementing test in Neutron that checks that models are
>>>> synchronized with database state [1] [2]. This is very important change as
>>>> during Juno cycle big changes of database structure were done.
>>>>
>>>> I was working on it for rather long time but about three weeks ago
>>>> strange error appeared [3], using AssertionPool shows [4]. The problem is
>>>> that somehow there are more than one connection to database from each test.
>>>> I tried to use locks from lockutils, but it didn’t help. On db meeting we
>>>> decided to add TestCase just for one Ml2 plugin for starters, and then
>>>> continue working on this strange error, that is why there are two change
>>>> requests [1] and [2]. But I found out that somehow even one testcase fails
>>>> with the same error [5] from time to time.
>>>>
>>>> I’m asking for any suggestions that could be done in this case. It is
>>>> very important to get at least [1] merged in Juno.
>>>>
>>>> [1] - https://review.openstack.org/76520
>>>>
>>>> [2] - https://review.openstack.org/120040
>>>>
>>>> [3] - http://paste.openstack.org/show/110158/
>>>>
>>>> [4] - http://paste.openstack.org/show/110159/
>>>>
>>>> [5] -
>>>> http://logs.openstack.org/20/76520/68/check/gate-neutron-python27/63938f9/testr_results.html.gz
>>>>
>>>> Regards,
>>>>
>>>> Ann
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Kevin Benton
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Kevin Benton
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140912/723f2e94/attachment.html>