Open Stack

On Thu, Oct 30, 2014 at 6:30 AM, Eoghan Glynn <eglynn at redhat.com> wrote:

>
>
> > Hi everyone,
> >
> > Before we start the larger discussion at summit next week about the
> future of
> > testing in OpenStack - specifically about spinning up functional testing
> and
> > how
> > it relates to tempest - I would like to share some of my thoughts on how
> we
> > can
> > get things started and how I think they'll eventually come together.
> >
> > Currently in tempest we have a large number of tests (mostly api-focused)
> > which are probably a better fit for a project's functional test suite.
> The
> > best
> > example I can think of is the nova flavors tests. Validation of flavor
> > manipulation doesn't need to run in the integrated test suite on every
> commit
> > to
> > every project because it only requires Nova. A simple win for initiating
> > in-tree
> > functional testing would be to move these kinds of tests into the
> projects
> > and
> > run the tests from the project repos instead of from tempest.
> >
> > This would have the advantage of making tempest slimmer for every project
> > and begin the process of getting projects to take responsibility for
> their
> > functional testing rather than relying on tempest. As tests are moved
> tempest
> > can start to become the integration test suite it was meant to be. It
> would
> > retain only tests that involve multiple projects and stop being the
> OpenStack
> > black box testing suite. I think that this is the right direction for
> tempest
> > moving forward, especially as we move to having project-specific
> functional
> > testing.
> >
> > Doing this migration is dependent on some refactors in tempest and moving
> > the required bits to tempest-lib so they can be easily consumed by the
> > other projects. This will be discussed at summit, is being planned
> > for implementation this cycle, and is similar to what is currently in
> > progress
> > for the cli tests.
> >
> > The only reason this testing existed in tempest in the first place was as
> > mechanism to block and then add friction against breaking api changes.
> > Tempest's
> > api testing has been been pretty successful at achieving these goals.
> We'll
> > want
> > to ensure that migrated tests retain these characteristics. If we are
> using
> > clients from tempest-lib we should get this automatically since to break
> > the api you'd have to change the api client. Another option proposed was
> to
> > introduce a hacking rule that would block changes to api tests at the
> same
> > time
> > other code was being changed.
> >
> > There is also a concern for external consumers of tempest if we move the
> > tests
> > out of the tempest tree (I'm thinking refstack). I think the solution is
> > to maintain a load_tests discovery method inside of tempest or elsewhere
> that
> > will run the appropriate tests from the other repos for something like
> > refstack.
> > Assuming that things are built in a compatible way using the same
> framework
> > then
> > running the tests from separate repos should be a simple matter of
> pointing
> > the
> > test runner in the right direction.
> >
> > I also want to comment on the role of functional testing. What I've
> proposed
> > here is only one piece of what project specific functional testing
> should be
> > and just what I feel is a good/easy start. I don't feel that this should
> be
> > the only testing done in the projects.  I'm suggesting this as a first
> > step because the tests already exist and it should be a relatively simple
> > task.
> > I also feel that using tempest-lib like this shouldn't be a hard
> requirement.
> > Ideally the client definitions shouldn't have to live externally, or if
> they
> > did
> > they would be the official clients, but I am suggesting this as a first
> step
> > to
> > start a migration out of tempest.
> >
> > I don't want anyone to feel that they need block their functional testing
> > efforts until tempest-lib becomes more useable. The larger value from
> > functional
> > testing is actually in enabling testing more tightly coupled to the
> projects
> > (e.g. whitebox testing). I feel that any work necessary to enable
> functional
> > testing should be prioritized.
>
> Thanks Matt for getting the ball rolling on this conversation in advance
> of summit.
>
> Probably stating the obvious here, but I feel we should make a concious
> effort to keep the approaches to in-tree functional testing as consistent
> as possible across projects.
>
> Towards that end, it would be good for folks with an interest in this area
> to attend each other's sessions where possible:
>
>
+1


>  Cross-project: Tue, 12:05 [1]
>  Heat:          Wed, 13:50 [2]
>  Nova:          Wed, 16:30 [3]
>  Ceilometer:    Wed, 17:20 [4]
>  QA:            Wed, 17:20 [5]


I think Keystone was planning on dedicating some time to this on Friday, so
our dev/hackathon day. I'm
interested in the process that will be in place for tracking status of the
"functional test migration" if there will be one.


> Unfortunately there's a clash there between the QA tempest-lib session
> and the ceilo session. I'm not sure how reasonable it would be to make
> a last-minute schedule change to avoid that clash.
>
> Cheers,
> Eoghan
>
> [1]
> http://kilodesignsummit.sched.org/event/575938e4837e8293615845582d7e3e7f
> [2]
> http://kilodesignsummit.sched.org/event/eb261fb08b18ec1eaa2c67492e7fc385
> [3]
> http://kilodesignsummit.sched.org/event/271a9075c1ced6c1269100ff4b8efc37
> [4]
> http://kilodesignsummit.sched.org/event/daf63526a1883e84cec107c70cc6cad3
> [5]
> http://kilodesignsummit.sched.org/event/a45e9af23c2c3fc17c12cc22e8bb3704
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141030/9fc62cab/attachment.html>