Open Stack

On Wed, Mar 26, 2014 at 6:24 PM, Doug Hellmann
<doug.hellmann at dreamhost.com>wrote:

>
>
>
> On Wed, Mar 26, 2014 at 4:30 PM, James E. Blair <jeblair at openstack.org>wrote:
>
>> Hi,
>>
>> I recently started looking into running the unit tests with MySQL.  I
>> pushed the start of a change here to do that:
>>
>> https://review.openstack.org/#/c/82872/
>>
>> In particular, that change implements something we've wanted to do for a
>> long time elsewhere in OpenStack -- create a dedicated MySQL database
>> for each test so that tests can safely run in parallel.  However, that
>> has uncovered the following issues:
>>
>>  * The tests do not use alembic, except for the migration test. This
>>    could lead to variations in what the unit tests are testing against
>>    as compared to production.
>>
>>  * The model does not specify an engine for MySQL (this is a difference
>>    between the model and the alembic migrations). So when the unit tests
>>    are run on MySQL, they may end up on MyISAM (they do in the gate).
>>
>>  * The two errors that show up in the test results for my patch relate
>>    to how timestamps are stored differently between MySQL and SQLite.
>>
>>  * The errors that you can not see in the test report from Jenkins but
>>    would if you ran in an environment with InnoDB as the default engine
>>    for MySQL are several instances of foreign key constraint errors.
>>
>> I think it is desirable to run the unit tests with MySQL largely because
>> I think we should be testing it in the way we intend to use it in
>> production.  Note especially the last point above -- our unit tests have
>> significant problems with MySQL.  That suggests one of two things to me:
>> either the tests aren't testing real behavior, or our system is actually
>> broken in production.  Neither one seems like a good situation.
>>
>> My inclination would be to use MySQL for unit tests, use alembic to
>> create the schema, and fix the errors currently uncovered by this.
>>
>> It's not going to be a trivial change, but I think it's worth doing.
>> Before embarking on it, I'd like to see if we can get consensus.  We can
>> talk about it here or possibly at the meeting tomorrow.
>>
>> Ruslan has commented in that review with the following:
>>
>> > But, I think that we should consider work done in major OpenStack
>> > projects. They don't use real MySQL to run unit-tests. Here is what
>> > the do (or plan to do):
>> >
>> > * Run unit-tests on SQLite. DB schema is created from SQLA metadata
>> > * Run DB migration tests on MySQL and Postgres
>> > * Run tests to compare SQLA metadata and DB migrations to make sure
>> >   that they're in sync. Here is WIP patch
>> >   https://review.openstack.org/#/c/
>> >
>> > I'm not opposed to this direction, I just want to make sure we're
>> > aligned with the common practices in OpenStack
>>
>> (That last link doesn't seem to be correct.)
>>
>> That's a good point, but I personally find testing as close to
>> production as possible desirable -- MySQL and SQLite are not compatible
>> so we shouldn't pretend that they are.  If we prove this is a viable
>> strategy, then I think we should recommend OpenStack projects adopt it
>> as well.
>>
>
> Some of the folks from Mirantis who are working on oslo.db are working on
> making it so we can use the models to create a database in our unit tests,
> and then compare the resulting database with what the migrations do. That
> *should* speed up tests, since we don't have to apply a set of migrations,
> while still ensuring that we are testing using the same schema. It also has
> the benefit of ensuring that our models represent what the migrations
> create. I've copied Victor and Roman on this so they can add more detail.
>

And, that's the same work Ruslan was linking to based on his follow-up
message.

Doug



>
> Doug
>
>
>
>>
>> -Jim
>>
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20140326/43c554be/attachment.html>