Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/12/2013 08:20 AM, Sean Dague wrote:
> Current Gate Length: 12hrs*, 41 deep
> 
> (top of gate entered 12hrs ago)
> 
> It's been an *exciting* week this week. For people not paying
> attention we had 2 external events which made things terrible
> earlier in the week.
> 
> ========================== Event 1: sphinx 1.2 complete breakage -
> MOSTLY RESOLVED ==========================
> 
> It turns out sphinx 1.2 + distutils (which pbr magic call through)
> means total sadness. The fix for this was a requirements pin to
> sphinx < 1.2, and until a project has taken that they will fail in
> the gate.
> 
> It also turns out that tox installs pre-released software by
> default (a terrible default behavior), so you also need a tox.ini
> change like this -
> https://github.com/openstack/nova/blob/master/tox.ini#L9 otherwise 
> local users will install things like sphinx 1.2b3. They will also
> break in other ways.
> 
> Not all projects have merged this. If you are a project that
> hasn't, please don't send any other jobs to the gate until you do.
> A lot of delay was added to the gate yesterday by Glance patches
> being pushed to the gate before their doc jobs were done.
> 
> ========================== Event 2: apt.puppetlabs.com outage -
> RESOLVED ==========================
> 
> We use that apt repository to setup the devstack nodes in nodepool
> with puppet. We were triggering an issue with grenade where it's
> apt-get calls were failing, because it does apt-get update once to
> make sure life is good. This only triggered in grenade (noth other
> devstack runs) because we do set -o errexit aggressively.
> 
> A fix in grenade to ignore these errors was merged yesterday
> afternoon (the purple line -
> http://status.openstack.org/elastic-recheck/ you can see where it
> showed up).
> 
> ========================== Top Gate Bugs 
> ==========================
> 
> We normally do this as a list, and you can see the whole list here
> - http://status.openstack.org/elastic-recheck/ (now sorted by
> number of FAILURES in the last 2 weeks)
> 
> That being said, our bigs race bug is currently this one bug - 
> https://bugs.launchpad.net/tempest/+bug/1253896 - and if you want
> to merge patches, fixing that one bug will be huge.
> 
> Basically, you can't ssh into guests that get created. That's sort
> of a fundamental property of a cloud. It shows up more frequently
> on neutron jobs, possibly due to actually testing the metadata
> server path. There have been many attempts on retry logic on this,
> we actually retry for 196 seconds to get in and only fail once we
> can't get in, so waiting isn't helping. It doesn't seem like the
> env is under that much load.
> 
> Until we resolve this, life will not be good in landing patches.
> 
> -Sean
> 
> 
> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
Thanks Sean:

This is a terrific summary which really makes my task of confirming
and following up much more manageable.

Just by way of preempting the "its neutron's fault" pile-on, just in
case anyone is tempted, a few facts:

We were paying attention, as it happens to the sphinx pin. Patches to
neutron and neutronclient have merged:
http://git.openstack.org/cgit/openstack/neutron/tree/test-requirements.txt#n9
http://git.openstack.org/cgit/openstack/python-neutronclient/tree/test-requirements.txt#n9

The addition of the -U flag for pip install in tox.ini for
neutronclient https://review.openstack.org/#/c/60825/4 is in the check
queue, it tripped on the sphinx pin for neutronclient. Here is the one
for neutron: https://review.openstack.org/#/c/60825/4 I've just
alerted all neutron-core to refrain from +A'ing until these are merged.

We had been tracking 1253896 quite closely and I at least was working
wtih the belief we had done the work we needed to do for that bug.
Since it now comes to light that I am in error with regards to
neutron's responsibility to 1253896, I welcome all interested parties
to #openstack-neutron so that we can again work together to submit a
patch that addresses this issue.

Thanks Sean,
Anita.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSqcpnAAoJELmKyZugNFU0X40H/0GX6zpTEdyQJVgDLDmOlo7v
X5W3ioRfjQL28Rm+ISAoq/CTHuekggimhIsTz/TIJpYeS+657Bg+rPE2BE4S6Aag
IdyMOQcMVfjUxcit9UTssMueSsw7VcnLJSbi7hEcBAtIRkf6IA2gxZ/lvrx7VmD5
Odfg3mdUrnckVdv7Y6/7tWCMY+3sXuqLQwat3a83mP13jRjgbQw9QnVhib9yoHg3
+qJnDoH9BT3+PWig42u7893qaasCzqFiiyjlGnjg9YznrRRZvq0Szwqux/JgzWy4
ypmX5Xo4ueZGuLMpmb2Sb8RbE83q3u9nx15nTWFdC+IUxa12DnX1sid27YCDva4=
=oVW8
-----END PGP SIGNATURE-----