<div dir="ltr">Hello Sean, all,<div><br></div><div>Currently there are ~30 test classes in DB API tests, containing ~370 test cases. setUpClass()/tearDownClass() would be definitely an improvement, but applying of all DB schema migrations for MySQL 30 times is going to take a long time...</div>
<div><br><div>Thanks,</div><div>Roman</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 21, 2013 at 3:02 PM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 06/21/2013 07:40 AM, Roman Podolyaka wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, all!<br>
<br>
In Nova we've got a DB access layer known as "DB API" and tests for it.<br>
Currently, those tests are run only for SQLite in-memory DB, which is<br>
great for speed, but doesn't allow us to spot backend-specific errors.<br>
<br>
There is a blueprint<br>
(<a href="https://blueprints.launchpad.net/nova/+spec/db-api-tests-on-all-backends" target="_blank">https://blueprints.launchpad.<u></u>net/nova/+spec/db-api-tests-<u></u>on-all-backends</a>)<br>
by Boris Pavlovic, which goal is to run the DB API tests on all DB<br>
backends (e. g. SQLite, MySQL and PosgreSQL). Recently, I've been<br>
working on implementation of this BP<br>
(<a href="https://review.openstack.org/#/c/33236/" target="_blank">https://review.openstack.org/<u></u>#/c/33236/</a>).<br>
<br>
The chosen approach for implementing this is best explained by going<br>
through a list of problems which must be solved:<br>
<br>
1. Tests should be executed concurrently by testr.<br>
<br>
testr creates a few worker processes each running a portion of test<br>
cases. When SQLite in-memory DB is used for testing, each of those<br>
processes has it's own DB in its address space, so no race conditions<br>
are possible. If we used a shared MySQL/PostgreSQL DB, the test suite<br>
would fail due to various race conditions. Thus, we must create a<br>
separate DB for each of test running processes and drop those, when all<br>
tests end.<br>
<br>
The question is, where we should create/drop those DBs? There are a few<br>
possible places in our code:<br>
1) setUp()/tearDown() methods of test cases. These are executed for<br>
each test case (there are ~370 tests in test_db_api). So it must be a<br>
bad idea to create/apply migrations/drop DB 370 times, if MySQL or<br>
PostgreSQL are used instead of SQLite in-memory DB<br>
2) testr supports creation of isolated test environments<br>
(<a href="https://testrepository.readthedocs.org/en/latest/MANUAL.html#remote-or-isolated-test-environments" target="_blank">https://testrepository.<u></u>readthedocs.org/en/latest/<u></u>MANUAL.html#remote-or-<u></u>isolated-test-environments</a>).<br>
Long story short: we can specify commands to execute before tests are<br>
run, after test have ended and how to run tests<br>
3) module/package level setUp()/tearDown(), but these are probably<br>
supported only in nosetest<br>
</blockquote>
<br></div></div>
How many Classes are we talking about? We're actually going after a similar problem in Tempest that setUp isn't cheap, so Matt Treinish has an experimental patch to testr which allows class level partitioning instead. Then you can use setupClass / teardownClass for expensive resource setup.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
So:<br>
1) before tests are run, a few test DBs are created (the number of<br>
created DBs is equal to the used concurrency level value)<br>
2) for each test running process an env variable, containing the<br>
connection string to the created DB, is set;<br>
3) after all test running processes have ended, the created DBs are<br>
dropped.<br>
<br>
2. Tests cleanup should be fast.<br>
<br>
For SQLite in-memory DB we use "create DB/apply migrations/run test/drop<br>
DB" pattern, but that would be too slow for running tests on MySQL or<br>
PostgreSQL.<br>
<br>
Another option would be to create DB only once for each of test running<br>
processes, apply DB migrations and then run each test case within a DB<br>
transaction which is rolled back after a test ends. Combining with<br>
something like "fsync = off" option of PostgreSQL this approach works<br>
really fast (on my PC it takes ~5 s to run DB API tests on SQLite and<br>
~10 s on PostgreSQL).<br>
</blockquote>
<br></div>
I like the idea of creating a transaction in setup, and triggering rollback in teardown, that's pretty clever.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3. Tests should be easy to run for developers as well as for Jenkins.<br>
<br>
DB API tests are the only tests which should be run on different<br>
backends. All other test cases can be run on SQLite. The convenient way<br>
to do this is to create a separate tox env, running only DB API tests.<br>
Developers specify the DB connection string which effectively defines<br>
the backend that should be used for running tests.<br>
<br>
I'd rather not run those tests 'opportunistically' in py26 and py27 as<br>
we do for migrations, because they are going to be broken for some time<br>
(most problems are described here<br>
<a href="https://docs.google.com/a/mirantis.com/document/d/1H82lIxd54CRmy-22oPRUS1sBkEtiguMU8N0whtye-BE/edit" target="_blank">https://docs.google.com/a/<u></u>mirantis.com/document/d/<u></u>1H82lIxd54CRmy-<u></u>22oPRUS1sBkEtiguMU8N0whtye-BE/<u></u>edit</a>).<br>
So it would be really nice to have a separate non-voting gate test.<br>
</blockquote>
<br></div>
Seperate tox env is the right approach IMHO, that would let it run issolated non-voting until we get to the bottom of the issues. For simplicity I'd still use the opportunistic db user / pass, as that will mean it could run upstream today.<span class="HOEnZb"><font color="#888888"><br>
<br>
-Sean<br>
<br>
-- <br>
Sean Dague<br>
<a href="http://dague.net" target="_blank">http://dague.net</a><br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</font></span></blockquote></div><br></div></div>