Open Stack

Hi,

I recently started looking into running the unit tests with MySQL.  I
pushed the start of a change here to do that:

https://review.openstack.org/#/c/82872/

In particular, that change implements something we've wanted to do for a
long time elsewhere in OpenStack -- create a dedicated MySQL database
for each test so that tests can safely run in parallel.  However, that
has uncovered the following issues:

 * The tests do not use alembic, except for the migration test. This
   could lead to variations in what the unit tests are testing against
   as compared to production.

 * The model does not specify an engine for MySQL (this is a difference
   between the model and the alembic migrations). So when the unit tests
   are run on MySQL, they may end up on MyISAM (they do in the gate).

 * The two errors that show up in the test results for my patch relate
   to how timestamps are stored differently between MySQL and SQLite.

 * The errors that you can not see in the test report from Jenkins but
   would if you ran in an environment with InnoDB as the default engine
   for MySQL are several instances of foreign key constraint errors.

I think it is desirable to run the unit tests with MySQL largely because
I think we should be testing it in the way we intend to use it in
production.  Note especially the last point above -- our unit tests have
significant problems with MySQL.  That suggests one of two things to me:
either the tests aren't testing real behavior, or our system is actually
broken in production.  Neither one seems like a good situation.

My inclination would be to use MySQL for unit tests, use alembic to
create the schema, and fix the errors currently uncovered by this.

It's not going to be a trivial change, but I think it's worth doing.
Before embarking on it, I'd like to see if we can get consensus.  We can
talk about it here or possibly at the meeting tomorrow.

Ruslan has commented in that review with the following:

> But, I think that we should consider work done in major OpenStack
> projects. They don't use real MySQL to run unit-tests. Here is what
> the do (or plan to do):
>
> * Run unit-tests on SQLite. DB schema is created from SQLA metadata
> * Run DB migration tests on MySQL and Postgres
> * Run tests to compare SQLA metadata and DB migrations to make sure
>   that they're in sync. Here is WIP patch
>   https://review.openstack.org/#/c/
>
> I'm not opposed to this direction, I just want to make sure we're
> aligned with the common practices in OpenStack

(That last link doesn't seem to be correct.)

That's a good point, but I personally find testing as close to
production as possible desirable -- MySQL and SQLite are not compatible
so we shouldn't pretend that they are.  If we prove this is a viable
strategy, then I think we should recommend OpenStack projects adopt it
as well.

-Jim