Open Stack

On Wed, Mar 26, 2014 at 4:30 PM, James E. Blair <jeblair at openstack.org>wrote:

> Hi,
>
> I recently started looking into running the unit tests with MySQL.  I
> pushed the start of a change here to do that:
>
> https://review.openstack.org/#/c/82872/
>
> In particular, that change implements something we've wanted to do for a
> long time elsewhere in OpenStack -- create a dedicated MySQL database
> for each test so that tests can safely run in parallel.  However, that
> has uncovered the following issues:
>
>  * The tests do not use alembic, except for the migration test. This
>    could lead to variations in what the unit tests are testing against
>    as compared to production.
>
>  * The model does not specify an engine for MySQL (this is a difference
>    between the model and the alembic migrations). So when the unit tests
>    are run on MySQL, they may end up on MyISAM (they do in the gate).
>
>  * The two errors that show up in the test results for my patch relate
>    to how timestamps are stored differently between MySQL and SQLite.
>
>  * The errors that you can not see in the test report from Jenkins but
>    would if you ran in an environment with InnoDB as the default engine
>    for MySQL are several instances of foreign key constraint errors.
>
> I think it is desirable to run the unit tests with MySQL largely because
> I think we should be testing it in the way we intend to use it in
> production.  Note especially the last point above -- our unit tests have
> significant problems with MySQL.  That suggests one of two things to me:
> either the tests aren't testing real behavior, or our system is actually
> broken in production.  Neither one seems like a good situation.
>
> My inclination would be to use MySQL for unit tests, use alembic to
> create the schema, and fix the errors currently uncovered by this.
>
> It's not going to be a trivial change, but I think it's worth doing.
> Before embarking on it, I'd like to see if we can get consensus.  We can
> talk about it here or possibly at the meeting tomorrow.
>
> Ruslan has commented in that review with the following:
>
> > But, I think that we should consider work done in major OpenStack
> > projects. They don't use real MySQL to run unit-tests. Here is what
> > the do (or plan to do):
> >
> > * Run unit-tests on SQLite. DB schema is created from SQLA metadata
> > * Run DB migration tests on MySQL and Postgres
> > * Run tests to compare SQLA metadata and DB migrations to make sure
> >   that they're in sync. Here is WIP patch
> >   https://review.openstack.org/#/c/
> >
> > I'm not opposed to this direction, I just want to make sure we're
> > aligned with the common practices in OpenStack
>
> (That last link doesn't seem to be correct.)
>
> That's a good point, but I personally find testing as close to
> production as possible desirable -- MySQL and SQLite are not compatible
> so we shouldn't pretend that they are.  If we prove this is a viable
> strategy, then I think we should recommend OpenStack projects adopt it
> as well.
>

Some of the folks from Mirantis who are working on oslo.db are working on
making it so we can use the models to create a database in our unit tests,
and then compare the resulting database with what the migrations do. That
*should* speed up tests, since we don't have to apply a set of migrations,
while still ensuring that we are testing using the same schema. It also has
the benefit of ensuring that our models represent what the migrations
create. I've copied Victor and Roman on this so they can add more detail.

Doug



>
> -Jim
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20140326/b5177d9a/attachment-0001.html>