Open Stack

On 04/28/2015 06:38 AM, Sean Dague wrote:
> The Tempest Smoke tag was originally introduced to provide a quick view
> of your OpenStack environment to ensure that a few basic things were
> working. It was intended to be fast.
>
> However, during Icehouse the smoke tag was repurposed as a way to let
> neutron not backslide (so it's massively overloaded with network tests).
> It current runs at about 15 minutes on neutron jobs. This is why grenade
> neutron takes *so* long, because we run tempest smoke twice.
>
> The smoke tag needs a diet. I believe our working definition should be
> something as follows:
>
>   - Total run time should be fast (<= 5 minutes)
>   - No negative tests
>   - No admin tests
>   - No tests that test optional extensions
>   - No tests that test advanced services (like lbaas, vpnaas)
>   - No proxy service tests
>
> The criteria for a good set of tests is CRUD operations on basic
> services. For instance, with compute we should have built a few servers,
> ensure we can shut them down. For neutron we should have done some basic
> network / port plugging.
That makes sense. On IRC, Sean and I agreed that this would include 
creation of users, projects, etc. So some of the keystone smoke tests
will be left in even though admin. IMO, it is debatable whether admin is 
relevant as part of the criteria for smoke.
>
> We also previously had the 'smoke' tag include all of the scenario
> tests, which was fine when we had 6 scenario tests. However as those
> have grown I think that should be trimmed back to a few basic through
> scenarios.
>
> The results of this are -
> https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:smoke,n,z
>
> The impacts on our upstream gate will mean that grenade jobs will speed
> up dramatically (20 minutes faster on grenade neutron).
>
> There is one edge condition which exists, which is the
> check-tempest-dsvm-neutron-icehouse job. Neutron couldn't pass either a
> full or parallel tempest run in icehouse (it's far too racy). So that's
> current running the smoke-serial tag. This would end up reducing the
> number of tests run on that job. However, based on the number of
> rechecks I've had to run in this series, that job is currently at about
> a 30% fail rate - http://goo.gl/N2w7qc - which means some test reduction
> is probably in order anyway, as it's mostly just preventing other people
> from landing unrelated patches.
>
> This was something we were originally planning on doing during the QA
> Sprint but ran out of time. It looks like we'll plan to land this right
> after Tempest 4 is cut this week, so that people that really want the
> old behavior can stay on the Tempest 4 release, but master is moving
> forward.
>
> I think that once we trim down we can decide to point add specific tests
> later. I expect smoke to be a bit more fluid over time, so it's not a
> tag that anyone should count on a test going into that tag and staying
> forever.
Agreed. The criteria and purpose should stay the same but individual 
tests may be added or removed from smoke.
Thanks for doing this.

  -David
>
> 	-Sean
>