Open Stack

On 03/06/2014 08:11 AM, Steven Hardy wrote:
> On Thu, Mar 06, 2014 at 07:53:03AM -0500, Sean Dague wrote:
>> We're at Freeze, so I want to pick up and understand where we currently
>> stand with both Neutron and Heat actually getting tested fully in the gate.
>>
>> First Neutron -
>> https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full
>>
>>
>> We know that this is *close* as the full job is running non voting
>> everywhere, and typically passing. How close are we? Or should we be
>> defering this until Juno (which would be unfortunate).
> 
> Can you please clarify - does FF apply to tempest *tests*?  My assumption
> was that we could move from feature development to testing during the FF/RC
> phase of the cycle, the natural by-product of which will be additional
> tempest testcases..

It doesn't. Typically we're adding additional validation up to the
release to catch things we believe are still exposed (especially in
conjunction with bugs that are found). That being said, if the core
teams here are punting, it would be good to know, so we don't waste time
asking the questions over and over again. :)

>> Second Heat -
>> https://blueprints.launchpad.net/tempest/+spec/tempest-heat-integration
>>
>> The Heat tests that are in a normal Tempest job are relatively trivial
>> surface verification, and in no way actually make sure that Heat is
>> operating at a real level. This fact is a contributing factor to why
>> Heat was broken in i2.
> 
> Yeah, we have a list of wishlist bugs so we can track the additional
> less-trivial tests we'd like to implement:
> 
> https://bugs.launchpad.net/heat/+bugs?field.tag=tempest
> 
> I was hoping we'd get patches at least proposed for most of these before
> Icehouse ships, but we'll have to see how it goes.. :)

+1. The more the better.

>> The first real verification for Heat is in the Heat slow job (which we
>> created to give Heat a separate time budget, because doing work that
>> requires real guests takes time).
>>
>> The heat slow job looks like it is finally passing much of the time -
>> http://logstash.openstack.org/#eyJzZWFyY2giOiIobWVzc2FnZTpcIkZpbmlzaGVkOiBTVUNDRVNTXCIgT1IgbWVzc2FnZTpcIkZpbmlzaGVkOiBGQUlMVVJFXCIpIEFORCBidWlsZF9uYW1lOmNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWhlYXQtc2xvdyIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDEwOTg3NDQ4OH0=
>>
>> It's seeing a 78% pass rate in check. Can anyone in the Heat team
>> confirm that the Failures in this job are actually real failures on
>> patches that should have been blocked?
>>
>> I'd like to get that turned on (and on all the projects) as soon as the
>> Heat team is confident on it so that Heat actually participates in the
>> tempest/devstack gate in a material way and we can prevent future issues
>> where a keystone, nova, neutron or whatever change would break Heat in git.
> 
> Steve Baker can make the final decision, but I am +2 on turning it on -
> during the process of getting my instance-users patches through the gate,
> this test consistently found real issues with no false positives (other
> than those caused by other known bugs).

Great. Looking forward to turning that on.

	-Sean

-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140306/768e7792/attachment.pgp>