On 12/13/2018 7:27 AM, Chris Dent wrote:
I guess in this case these tests were exposed by their failing, and it was only once investigating that you realized they weren't truly integration tests? Have you, Matt, got any ideas on how to find other non-integration tests that are being treated as integration which we could move to their own things? De-tangling the spaghetti is likely to reveal plenty of improvements but also plenty of areas that need more attention.
I haven't done a full audit anytime recent, no. I'm sure there are lots of tempest tests that are single-service, e.g. doing things in just cinder, glance, nova or neutron, which don't require any other services (nova might be the exception in some cases since we need at least glance and neutron for building a server). There was a time a few years ago where QA folk were working on pulling stuff like that out of tempest and moving it into the project trees if their functional testing would cover it, e.g. anything that just required the compute API and DB was a candidate to move into nova functional tests (like flavor and aggregates tests). However, interop tests are based on tempest so there are some tests you just can't remove because they are used by refstack.