Open Stack

Hi all -

Thanks for all the responses I got on my "Make EngineFacade a Facade” spec - plenty of people have commented, pretty much all positively so I’m pretty confident we can start building the basic idea of that out into a new review. 

I want to point out that there is another, closely related spec that has been around several weeks longer, which is to overhaul the capabilities of our test runner system: "Add long-lived-transactionalized-db-fixtures” - https://review.openstack.org/#/c/117335/.    I’ve talked about this spec several times before on this list, and it is still out there, and I additionally have most of the implementation working for several weeks now.    The spec and implementation has been mostly twisting in the wind, partially due to a little bit of waiting for some possible changes to namespace packages and test invocation, however I’d like to reiterate that A. the whole series works right now independently of those changes (https://review.openstack.org/#/c/117335/, https://review.openstack.org/#/c/120870/), and B. the spec describing the system has hardly been +1/-1’ed by anyone in any case (comments positive or negative are appreciated!   I redid the whole thing in response to previous comments many weeks ago).

Just to try to pitch this series, yet again, here’s what we get:

1. a solid and extensible system of building up and tearing down databases (provisioning), as well as a DROP ALL of database objects, in a database-specific way (e.g. drops special objects like PG ENUM objects and such).   This is completed, works right now.

2. the ability to produce “transactionalized” test fixtures, where you can run as many tests as you want against a single schema that remains in place, each test instead has all of its changes rolled back inside of a transaction.  In particular this will make it lots easier for large test suites like Nova’s DB api suite to run against many databases efficiently, as it won’t have to drop and rebuild the whole schema for each test.    This mechanism is completed, works right now, as soon as it’s merged I can start doing a proof of concept for Nova’s test_db_api.py.

3. the ability to run non-transactionalized tests like we do now, which going forward would remain appropriate at least for migration tests, on a fixed database-per-subprocess, emitting an unconditional DROP of all objects remaining in the schema at the end of each test without actually dropping the whole database.   Completed and works right now, I’ve done a test against Neutron’s migration tests and the optimising suite system works.

4. An overhaul to how connectivity for multiple databases is set up, e.g. Postgresql, MySQL, others.   The usual system of “opportunistic” looking around for backends remains unchanged, but you can affect the specific URLs that will be queried, as well as limit the test run to any particular database URL using an environment variable. The system also supports other databases besides the three of SQLite, PG, and MySQL now whereas it had some issues before which would prevent that.   Completed and works right now!

5. The ability to have a single test suite run automatically for any number of backends, including future backends that might not be added to oslo.db yet, replacing the current system of subclassing MySQLOpportunisticTest and PostgresqlOpportunisticTest.   Not completed!   But is fairly trivial.

6. Once the EngineFacade overhaul is in place, the two systems will integrate together so that it will be very simple for projects whose test suite currently runs off of CONF + sqlite will be able to use the new system just by dropping in a mixin class.

The four “pillars” I’m trying to get through, hopefully by the end of Kilo are:   1. application connectivity and transaction control 2. test connectivity and transaction 3. query modernization and 4. getting ready for Alembic (where I add SQLite support and multiple branch support).     We definitely need #1 and #2, so I’d like to get continued feedback #2 so I can point that in the correct direction.

thanks all for your support!