Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/12/2013 09:39 AM, Anita Kuno wrote:
> On 12/12/2013 08:20 AM, Sean Dague wrote:
>> Current Gate Length: 12hrs*, 41 deep
> 
>> (top of gate entered 12hrs ago)
> 
>> It's been an *exciting* week this week. For people not paying 
>> attention we had 2 external events which made things terrible 
>> earlier in the week.
> 
>> ========================== Event 1: sphinx 1.2 complete breakage
>> - MOSTLY RESOLVED ==========================
> 
>> It turns out sphinx 1.2 + distutils (which pbr magic call
>> through) means total sadness. The fix for this was a requirements
>> pin to sphinx < 1.2, and until a project has taken that they will
>> fail in the gate.
> 
>> It also turns out that tox installs pre-released software by 
>> default (a terrible default behavior), so you also need a
>> tox.ini change like this - 
>> https://github.com/openstack/nova/blob/master/tox.ini#L9
>> otherwise local users will install things like sphinx 1.2b3. They
>> will also break in other ways.
> 
>> Not all projects have merged this. If you are a project that 
>> hasn't, please don't send any other jobs to the gate until you
>> do. A lot of delay was added to the gate yesterday by Glance
>> patches being pushed to the gate before their doc jobs were
>> done.
> 
>> ========================== Event 2: apt.puppetlabs.com outage - 
>> RESOLVED ==========================
> 
>> We use that apt repository to setup the devstack nodes in
>> nodepool with puppet. We were triggering an issue with grenade
>> where it's apt-get calls were failing, because it does apt-get
>> update once to make sure life is good. This only triggered in
>> grenade (noth other devstack runs) because we do set -o errexit
>> aggressively.
> 
>> A fix in grenade to ignore these errors was merged yesterday 
>> afternoon (the purple line - 
>> http://status.openstack.org/elastic-recheck/ you can see where
>> it showed up).
> 
>> ========================== Top Gate Bugs 
>> ==========================
> 
>> We normally do this as a list, and you can see the whole list
>> here - http://status.openstack.org/elastic-recheck/ (now sorted
>> by number of FAILURES in the last 2 weeks)
> 
>> That being said, our bigs race bug is currently this one bug - 
>> https://bugs.launchpad.net/tempest/+bug/1253896 - and if you
>> want to merge patches, fixing that one bug will be huge.
> 
>> Basically, you can't ssh into guests that get created. That's
>> sort of a fundamental property of a cloud. It shows up more
>> frequently on neutron jobs, possibly due to actually testing the
>> metadata server path. There have been many attempts on retry
>> logic on this, we actually retry for 196 seconds to get in and
>> only fail once we can't get in, so waiting isn't helping. It
>> doesn't seem like the env is under that much load.
> 
>> Until we resolve this, life will not be good in landing patches.
> 
>> -Sean
> 
> 
> 
>> _______________________________________________ OpenStack-dev 
>> mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>> 
> Thanks Sean:
> 
> This is a terrific summary which really makes my task of
> confirming and following up much more manageable.
> 
> Just by way of preempting the "its neutron's fault" pile-on, just
> in case anyone is tempted, a few facts:
> 
> We were paying attention, as it happens to the sphinx pin. Patches
> to neutron and neutronclient have merged: 
> http://git.openstack.org/cgit/openstack/neutron/tree/test-requirements.txt#n9
>
> 
http://git.openstack.org/cgit/openstack/python-neutronclient/tree/test-requirements.txt#n9
> 
A post to the mailing list isn't complete it appears unless I make a
mistake in it.
Here is our patch for neutronclient:
https://review.openstack.org/#/c/61508/

> The addition of the -U flag for pip install in tox.ini for 
> neutronclient https://review.openstack.org/#/c/60825/4 is in the
> check queue, it tripped on the sphinx pin for neutronclient. Here
> is the one for neutron: https://review.openstack.org/#/c/60825/4
> I've just alerted all neutron-core to refrain from +A'ing until
> these are merged.
> 
> We had been tracking 1253896 quite closely and I at least was
> working wtih the belief we had done the work we needed to do for
> that bug. Since it now comes to light that I am in error with
> regards to neutron's responsibility to 1253896, I welcome all
> interested parties to #openstack-neutron so that we can again work
> together to submit a patch that addresses this issue.
> 
> Thanks Sean, Anita.
> 
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSqdRnAAoJELmKyZugNFU0H80IAMnjI4r3ulePeZ3eenAwOYlY
TEC6hAClr374oq1B7Tk3JYP0dl+qMed7TaCSYI32wB9sVbYKsjKrjsNXARDpqqUe
r1Mb05jknDZtwfwBYNsoPtsn/sf9mtXm8T+Czk7ojMVogwjng8ps4juyG4ZveUNl
+zDkPtFgoWrzKA/pmUIfue5RgaH0MfX14ftojUmSroYfTuMK5fxSPc9mhV//2ZBw
JyNowczOdhOVf4pBZ7JFDDd1potoi+CtrOHxW13aUC5XyPLN/HU3ZoZRoWVeLzDT
FWgHgWo1DoK49+R12KsjzX0xhUVs2YV3oVrraPnXsMSqiWzWOU1xNglGwa6qWdg=
=PIFo
-----END PGP SIGNATURE-----