Open Stack

On 07/28/2013 08:02 PM, Monty Taylor wrote:
> Hey all!
>
> There is currently an issue with which is causing a very high failure
> rate in the gate. From IRC:
>
> 18:32:19          clarkb | the grenade failures seem to get very
> consistent in the gate at 2013-0-27 1552UTC
> 18:32:27          clarkb | before that the success rate is much higher
> 18:34:53          clarkb | *2013-07-27
> 18:40:01          clarkb | https://review.openstack.org/#/c/38810/ was
> the last change to pass grenade when it was semi consistently passing
> 18:41:31          clarkb | 38587 and 28082 seem like strong candidates
> for the breakage
>
> The working hypothesis is that since the grenade gate is assymetrical
> (it consumes grizzly and trunk but only gates trunk) that a change to
> grizzly went in that broke something for trunk. Obviously this is
> something we want to avoid - but since this is our first time gating on
> upgrade patterns in this way, it's also probably a good chance for us to
> learn about the process of doing that.
>
> In any case, although I'm sure dtroyer and sdague will take a look as
> soon as they are online, it's unlikely that anything is going to land
> until this is sorted- so I'm sure they'd appreciate any help from anyone
> who can look in to the actual issue.

I won't be able to get to trying code until tomorrow morning, however 
the mostly likely culprit line I'm seeing in the logs is this - 
http://logs-dev.openstack.org/51/38951/2/check/gate-grenade-devstack-vm/22154/logs/new/screen-c-sch.txt.gz#2013-07-28%2006%3A27%3A34.974

Cinder isn't able to schedule volumes, which is bad. How we got to this 
stage of bad post upgrade is unknown to me.

The other thing that is suspicious is this - 
http://logs-dev.openstack.org/51/38951/2/check/gate-grenade-devstack-vm/22154/logs/new/screen-c-vol.txt.gz#2013-07-28%2006%3A26%3A36.466

Especially given that the last change that passed the gate was adding 
jsonschema to the tempest requirements list. Maybe this is all just a 
crazy requirements unwind?

Anyway, help appreciated on debugging. This is actually catching a real 
problem with cinder, which is what it was designed to do. How we got to 
the real problem is however kind of up in the air.

	-Sean

-- 
Sean Dague
http://dague.net