Open Stack

On 4/28/2015 4:19 PM, David Kranz wrote:
> On 04/28/2015 06:38 AM, Sean Dague wrote:
>> The Tempest Smoke tag was originally introduced to provide a quick view
>> of your OpenStack environment to ensure that a few basic things were
>> working. It was intended to be fast.
>>
>> However, during Icehouse the smoke tag was repurposed as a way to let
>> neutron not backslide (so it's massively overloaded with network tests).
>> It current runs at about 15 minutes on neutron jobs. This is why grenade
>> neutron takes *so* long, because we run tempest smoke twice.
>>
>> The smoke tag needs a diet. I believe our working definition should be
>> something as follows:
>>
>>   - Total run time should be fast (<= 5 minutes)
>>   - No negative tests
>>   - No admin tests
>>   - No tests that test optional extensions
>>   - No tests that test advanced services (like lbaas, vpnaas)
>>   - No proxy service tests
>>
>> The criteria for a good set of tests is CRUD operations on basic
>> services. For instance, with compute we should have built a few servers,
>> ensure we can shut them down. For neutron we should have done some basic
>> network / port plugging.
> That makes sense. On IRC, Sean and I agreed that this would include
> creation of users, projects, etc. So some of the keystone smoke tests
> will be left in even though admin. IMO, it is debatable whether admin is
> relevant as part of the criteria for smoke.
>>
>> We also previously had the 'smoke' tag include all of the scenario
>> tests, which was fine when we had 6 scenario tests. However as those
>> have grown I think that should be trimmed back to a few basic through
>> scenarios.
>>
>> The results of this are -
>> https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:smoke,n,z
>>
>>
>> The impacts on our upstream gate will mean that grenade jobs will speed
>> up dramatically (20 minutes faster on grenade neutron).
>>
>> There is one edge condition which exists, which is the
>> check-tempest-dsvm-neutron-icehouse job. Neutron couldn't pass either a
>> full or parallel tempest run in icehouse (it's far too racy). So that's
>> current running the smoke-serial tag. This would end up reducing the
>> number of tests run on that job. However, based on the number of
>> rechecks I've had to run in this series, that job is currently at about
>> a 30% fail rate - http://goo.gl/N2w7qc - which means some test reduction
>> is probably in order anyway, as it's mostly just preventing other people
>> from landing unrelated patches.
>>
>> This was something we were originally planning on doing during the QA
>> Sprint but ran out of time. It looks like we'll plan to land this right
>> after Tempest 4 is cut this week, so that people that really want the
>> old behavior can stay on the Tempest 4 release, but master is moving
>> forward.
>>
>> I think that once we trim down we can decide to point add specific tests
>> later. I expect smoke to be a bit more fluid over time, so it's not a
>> tag that anyone should count on a test going into that tag and staying
>> forever.
> Agreed. The criteria and purpose should stay the same but individual
> tests may be added or removed from smoke.
> Thanks for doing this.
>
>   -David
>>
>>     -Sean
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Yeah why would you not include admin tests?  Like listing services and 
hosts in nova?

-- 

Thanks,

Matt Riedemann