Open Stack

I think some things are broken in the oslo-incubator db migration code.

Ironic moved to this when Juno opened and things seemed fine, until
recently when Lucas tried to add a DB migration and noticed that it didn't
run... So I looked into it a bit today. Below are my findings.

Firstly, I filed this bug and proposed a fix, because I think that tests
that don't run any code should not report that they passed -- they should
report that they were skipped.
  https://bugs.launchpad.net/oslo/+bug/1327397
  "No notice given when db migrations are not run due to missing engine"

Then, I edited the test_migrations.conf file appropriately for my local
mysql service, ran the tests again, and verified that migration tests ran
-- and they passed. Great!

Now, a little background... Ironic's TestMigrations class inherits from
oslo's BaseMigrationTestCase, then "opportunistically" checks each
back-end, if it's available. This opportunistic checking was inherited from
Nova so that tests could pass on developer workstations where not all
backends are present (eg, I have mysql installed, but not postgres), and
still transparently run on all backends in the gate. I couldn't find such
opportunistic testing in the oslo db migration test code, unfortunately -
but maybe it's well hidden.

Anyhow. When I stopped the local mysql service (leaving the configuration
unchanged), I expected the tests to be skipped, but instead I got two
surprise failures:
- test_mysql_opportunistically() failed because setUp() raises an exception
before the test code could call calling _have_mysql()
- test_mysql_connect_fail() actually failed! Again, because setUp() raises
an exception before running the test itself

Unfortunately, there's one more problem... when I run the tests in
parallel, they fail randomly because sometimes two test threads run
different migration tests, and the setUp() for one thread (remember, it
calls _reset_databases) blows up the other test.

Out of 10 runs, it failed three times, each with different errors:
  NoSuchTableError: `chassis`
  ERROR 1007 (HY000) at line 1: Can't create database 'test_migrations';
database exists
  ProgrammingError: (ProgrammingError) (1146, "Table
'test_migrations.alembic_version' doesn't exist")

As far as I can tell, this is all coming from:

https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/test_migrations.py#L86;L111


So, Ironic devs -- if you see a DB migration proposed, pay extra attention
to it. We aren't running migration tests in our check or gate queues right
now, and we shouldn't enable them until this fixed.

Regards,
Devananda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140606/0ff8c397/attachment.html>