Open Stack

On 01/09/2014 04:16 PM, Russell Bryant wrote:
> On 01/08/2014 05:53 PM, Joe Gordon wrote:
>> Hi All, 
>>
>> As you know the gate has been in particularly bad shape (gate queue over
>> 100!) this week due to a number of factors. One factor is how many major
>> outstanding bugs we have in the gate.  Below is a list of the top 4 open
>> gate bugs.
>>
>> Here are some fun facts about this list:
>> * All bugs have been open for over a month
>> * All are nova bugs
>> * These 4 bugs alone were hit 588 times which averages to 42 hits per
>> day (data is over two weeks)!
>>
>> If we want the gate queue to drop and not have to continuously run
>> 'recheck bug x' we need to fix these bugs.  So I'm looking for
>> volunteers to help debug and fix these bugs.
> 
> I created the following etherpad to help track the most important Nova
> gate bugs. who is actively working on them, and any patches that we have
> in flight to help address them:
> 
>   https://etherpad.openstack.org/p/nova-gate-issue-tracking
> 
> Please jump in if you can.  We shouldn't wait for the gate bug day to
> move on these.  Even if others are already looking at a bug, feel free
> to do the same.  We need multiple sets of eyes on each of these issues.
> 

Some good progress from the last few days:

After looking at a lot of failures, we determined that the vast majority
of failures are performance related.  The load being put on the
OpenStack deployment is just too high.  We're working to address this to
make the gate more reliable in a number of ways.

1) (merged) https://review.openstack.org/#/c/65760/

The large-ops test was cut back from spawning 100 instances to 50.  From
the commit message:

  It turns out the variance in cloud instances is very high, especially
  when comparing different cloud providers and regions. This test was
  originally added as a regression test for the nova-network issues with
  rootwrap. At which time this test wouldn't pass for 30 instances.  So
  50 is still a valid regression test.

2) (merged) https://review.openstack.org/#/c/45766/

nova-compute is able to do work in parallel very well.  nova-conductor
can not by default due to the details of our use of eventlet + how we
talk to MySQL.  The way you allow nova-conductor to do its work in
parallel is by running multiple conductor workers.  We had not enabled
this by default in devstack, so our 4 vCPU test nodes were only using a
single conductor worker.  They now use 4 conductor workers.

3) (still testing) https://review.openstack.org/#/c/65805/

Right now when tempest runs in the devstack-gate jobs, it runs with
concurrency=4 (run 4 tests at once).  Unfortunately, it appears that
this maxes out the deployment and results in timeouts (usually network
related).

This patch changes tempest concurrency to 2 instead of 4.  The initial
results are quite promising.  The tests have been passing reliably so
far, but we're going to continue to recheck this for a while longer for
more data.

One very interesting observation on this came from Jim where he said "A
quick glance suggests 1.2x -- 1.4x change in runtime."  If the
deployment were *not* being maxed out, we would expect this change to
result in much closer to a 2x runtime increase.

4) (approved, not yet merged) https://review.openstack.org/#/c/65784/

nova-network seems to be the largest bottleneck in terms of performance
problems when nova is maxed out on these test nodes.  This patch is one
quick speedup we can make by not using rootwrap in a few cases where it
wasn't necessary.  These really add up.

5) https://review.openstack.org/#/c/65989/

This patch isn't a candidate for merging, but was written to test the
theory that by updating nova-network to use conductor instead of direct
database access, nova-network will be able to do work in parallel better
than it does today, just as we have observed with nova-compute.

Dan's initial test results from this are **very** promising.  Initial
testing showed a 20% speedup in runtime and a 33% decrease in CPU
consumption by nova-network.

Doing this properly will not be quick, but I'm hopeful that we can
complete it by the Icehouse release.  We will need to convert
nova-network to use Nova's object model.  Much of this work is starting
to catch nova-network up on work that we've been doing in the rest of
the tree but have passed on doing for nova-network due to nova-network
being in a freeze.

6) (no patch yet)

We haven't had time to dive too deep into this yet, but we would also
like to revisit our locking usage and how it is affecting nova-network
performance.  There may be some more significant improvements we can
make there.


Final notes:

I am hopeful that by addressing these performance issues both in Nova's
code, as well as by turning down the test load, that we will see a
significant increase in gate reliability in the near future.  I
apologize on behalf of the Nova team for Nova's contribution to gate
instability.

*Thank you* to everyone who has been helping out!

-- 
Russell Bryant