Open Stack

On 07/03/2014 03:49 AM, Luke Gorrie wrote:
> On 3 July 2014 02:44, Michael Still <mikal at stillhq.com
> <mailto:mikal at stillhq.com>> wrote:
>
>     I have seen both. Normally there's a failure, reviewers notice, and
>     then the developer spins trying out fixes by uploading new patch sets.
>
>
> Interesting. Yes, I can see that you need fast response from CIs to
> support that scenario. 12-hour edit-compile-run loop will ruin anybody's
> day/week/month.
>
>     My rule of thumb is three hours by the way. I'd like to say something
>     like "not significantly slower than jenkins", but that's hard to
>     quantify.
>
> How do people normally throughput-optimize their CIs?

By using single-use slaves and things like nodepool to manage a "ready 
set" of VMs that are used for these single-use slaves.

> I suppose that parallelism is the basic trick, but how do people achieve
> it? Concurrent tempest runs on the same host? or on a pool of (virtual)
> hosts? or in one-shot disposable VMs?

One-shot disposable VMs, managed by nodepool or some other custom script.

> The CI that I operate is currently running tests serially on a "bare
> metal" server. This is okay for now (no backlog) but I would like to be
> able to ramp up the number of tests performed and then performance could
> become an issue.
>
> The idea I'm playing with now is to support N servers each running M
> parallel virtual machines for tempest. I'm tempted to use a disposable
> Vagrant VM for each tempest run both in order to isolate tests from each
> other (those running in parallel and also those that have run before)
> and perhaps even make it possible for others (4th parties?) to grab my
> Vagrantfile and replicate my test environment (if they want a faster
> turn-around than via Gerrit).

I don't think Vagrant will give you anything other than a slow launch 
time, frankly. You will want to have a system that keeps a pool of 
available single-use slave VMs ready to run a test job, report the 
results to Gearman (or Gerrit if you prefer to skip the entire queue 
mechanism), and terminate the VM.

> I'd be very curious to know what is working well/badly for others at the
> moment so that I can avoid stepping on land mines :-).

devstack-gate works very well for what it is supposed to do:

  * Checkout git repos of all projects related to OpenStack test runs
  * Checkout the SHA1 of the code in Gerrit repo for the project under 
test (Neutron in your case)
  * Configure all of the OpenStack services and start all the services 
in separate processes (screen sessions)
  * Direct logs to a standardized location
  * Run Tempest or another test suite based on environment variables

For your 100 line shell script project (which I can almost guarantee 
will end up being more than 100 lines ;), I would recommend using 
devstack-gate as much as you can, as it's taken dozens of engineers many 
man-months to get it right, and you can be sure that you will run the 
same tests that are executed in the gate, set up in an environment that 
is exactly like the gate, therefore achieving some level of consistency 
with the gate.

Best,
-jay