Open Stack

On Thu, Mar 06, 2014 at 07:53:03AM -0500, Sean Dague wrote:
> We're at Freeze, so I want to pick up and understand where we currently
> stand with both Neutron and Heat actually getting tested fully in the gate.
> 
> First Neutron -
> https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full
> 
> 
> We know that this is *close* as the full job is running non voting
> everywhere, and typically passing. How close are we? Or should we be
> defering this until Juno (which would be unfortunate).

Can you please clarify - does FF apply to tempest *tests*?  My assumption
was that we could move from feature development to testing during the FF/RC
phase of the cycle, the natural by-product of which will be additional
tempest testcases..

> Second Heat -
> https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration
> 
> The Heat tests that are in a normal Tempest job are relatively trivial
> surface verification, and in no way actually make sure that Heat is
> operating at a real level. This fact is a contributing factor to why
> Heat was broken in i2.

Yeah, we have a list of wishlist bugs so we can track the additional
less-trivial tests we'd like to implement:

https://bugs.launchpad.net/heat/+bugs?field.tag=tempest

I was hoping we'd get patches at least proposed for most of these before
Icehouse ships, but we'll have to see how it goes.. :)

> The first real verification for Heat is in the Heat slow job (which we
> created to give Heat a separate time budget, because doing work that
> requires real guests takes time).
> 
> The heat slow job looks like it is finally passing much of the time -
> http://logstash.openstack.org/#eyJzZWFyY2giOiIobWVzc2FnZTpcIkZpbmlzaGVkOiBTVUNDRVNTXCIgT1IgbWVzc2FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIpIEFORCBidWlsZF9uYW1lOmNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvdyIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDEwOTg3NDQ4OH0=
> 
> It's seeing a 78% pass rate in check. Can anyone in the Heat team
> confirm that the Failures in this job are actually real failures on
> patches that should have been blocked?
> 
> I'd like to get that turned on (and on all the projects) as soon as the
> Heat team is confident on it so that Heat actually participates in the
> tempest/devstack gate in a material way and we can prevent future issues
> where a keystone, nova, neutron or whatever change would break Heat in git.

Steve Baker can make the final decision, but I am +2 on turning it on -
during the process of getting my instance-users patches through the gate,
this test consistently found real issues with no false positives (other
than those caused by other known bugs).

Steve