Open Stack

On 10/30/2014 09:52 AM, Sean Dague wrote:
> On 10/30/2014 09:33 AM, David Kranz wrote:
>> On 10/30/2014 07:49 AM, Sean Dague wrote:
>>> On 10/29/2014 12:30 PM, Matthew Treinish wrote:
>>>> Hi everyone,
>>>>
>>>> Before we start the larger discussion at summit next week about the future of
>>>> testing in OpenStack - specifically about spinning up functional testing and how
>>>> it relates to tempest - I would like to share some of my thoughts on how we can
>>>> get things started and how I think they'll eventually come together.
>>>>
>>>> Currently in tempest we have a large number of tests (mostly api-focused)
>>>> which are probably a better fit for a project's functional test suite. The best
>>>> example I can think of is the nova flavors tests. Validation of flavor
>>>> manipulation doesn't need to run in the integrated test suite on every commit to
>>>> every project because it only requires Nova. A simple win for initiating in-tree
>>>> functional testing would be to move these kinds of tests into the projects and
>>>> run the tests from the project repos instead of from tempest.
>>> I think a lot of the negative API testing is also a great thing to 
>>> be done back at the project level. All of that testing should be 
>>> able to work without a full OpenStack, as it should be caught and 
>>> managed by the API service and never get any further than that.
>>>
>>>> This would have the advantage of making tempest slimmer for every project
>>>> and begin the process of getting projects to take responsibility for their
>>>> functional testing rather than relying on tempest. As tests are moved tempest
>>>> can start to become the integration test suite it was meant to be. It would
>>>> retain only tests that involve multiple projects and stop being the OpenStack
>>>> black box testing suite. I think that this is the right direction for tempest
>>>> moving forward, especially as we move to having project-specific functional
>>>> testing.
>>>>
>>>> Doing this migration is dependent on some refactors in tempest and moving
>>>> the required bits to tempest-lib so they can be easily consumed by the
>>>> other projects. This will be discussed at summit, is being planned
>>>> for implementation this cycle, and is similar to what is currently in progress
>>>> for the cli tests.
>>>>
>>>> The only reason this testing existed in tempest in the first place was as
>>>> mechanism to block and then add friction against breaking api changes. Tempest's
>>>> api testing has been been pretty successful at achieving these goals. We'll want
>>>> to ensure that migrated tests retain these characteristics. If we are using
>>>> clients from tempest-lib we should get this automatically since to break
>>>> the api you'd have to change the api client. Another option proposed was to
>>>> introduce a hacking rule that would block changes to api tests at the same time
>>>> other code was being changed.
>>>>
>>>> There is also a concern for external consumers of tempest if we move the tests
>>>> out of the tempest tree (I'm thinking refstack). I think the solution is
>>>> to maintain a load_tests discovery method inside of tempest or elsewhere that
>>>> will run the appropriate tests from the other repos for something like refstack.
>>>> Assuming that things are built in a compatible way using the same framework then
>>>> running the tests from separate repos should be a simple matter of pointing the
>>>> test runner in the right direction.
>>> I think we can see where this takes us. I'm still skeptical of cross 
>>> project loading of tests because it's often quite fragile. However, 
>>> if you look at what refstack did they had a giant evaluation of all 
>>> of tempest and pruned a bunch of stuff out. I would imagine maybe 
>>> there is a conversation there about tests that refstack feels are 
>>> important to stay in Tempest for their validation reasons. I think 
>>> having a few paths that are tested both in Tempest and in project 
>>> functional tests is not a bad thing.
>> Refstack is not the only thing that cares about validation of real 
>> clouds. As we move forward with this, it would be good to separate 
>> the issues of "in which repo does a functional test live" and "can a 
>> functional test be run against a real cloud". IMO, over use of 
>> mocking (broadly defined) in functional tests should be avoided 
>> unless it is configurable to also work in an unmocked fashion. 
>> Whether the way to combine all of the functional tests is by cross 
>> project loading of tests or by some other means is more of an 
>> implementation detail.
> Part of the perspective I'm bringing in is actually knowing what to do 
> when your tests fail. Using Tempest against real clouds is great, 
> people should keep doing that. But if you are rolling out a real cloud 
> yourself, in the future you should be running the functional tests in 
> staging to ensure you are functioning. Those will also provide you, 
> hopefully, with a better path to understand what's wrong.
Sean, sorry if I was unclear. By "real clouds", I just meant the tests 
should be able to use  OpenStack apis with no mocking.

  -David



>
> This will mean that as an arbitrary 3rd party accessing a public 
> cloud, you don't have a test suite that pushes every button of the 
> cloud. But I think that's an acceptable trade off. Because pushing on 
> odd edge cases might produce a fail, but if there is no clear path 
> from the fail to a fix, it doesn't help anyone.
>
> Finding failures in OpenStack is not hard. Finding failures in 
> OpenStack in a way that points towards resolution is what we need to 
> be striving for.
>
>>>
>>> But I think that's an end of cycle at best discussion.
>>>
>>> Also, there probably need to be a few discussions anyway of 
>>> refstack/tempest/defcore. The fact that Keystone was dropped from 
>>> defcore because there were no non admin Keystone tests explicitly in 
>>> Tempest (even though we make over 5000 keystone non admin API calls 
>>> over a tempest run) was very odd. That is something that could have 
>>> been fixed in a day.
>>>
>>>> I also want to comment on the role of functional testing. What I've proposed
>>>> here is only one piece of what project specific functional testing should be
>>>> and just what I feel is a good/easy start. I don't feel that this should be
>>>> the only testing done in the projects.  I'm suggesting this as a first
>>>> step because the tests already exist and it should be a relatively simple task.
>>>> I also feel that using tempest-lib like this shouldn't be a hard requirement.
>>>> Ideally the client definitions shouldn't have to live externally, or if they did
>>>> they would be the official clients, but I am suggesting this as a first step to
>>>> start a migration out of tempest.
>>>>
>>>> I don't want anyone to feel that they need block their functional testing
>>>> efforts until tempest-lib becomes more useable. The larger value from functional
>>>> testing is actually in enabling testing more tightly coupled to the projects
>>>> (e.g. whitebox testing). I feel that any work necessary to enable functional
>>>> testing should be prioritized.
>>> Agreed. From a Nova perspective I'm going to be layering up some of 
>>> this functionality with existing in tree tools. We also have a bunch 
>>> of potentially needed functional tests based on what I found in bug 
>>> triage this fall. So the priority is going to be getting something 
>>> working right to nail the bugs in Nova first, then align where it 
>>> makes sense.
>> I'm not sure enabling white-box testing is the whole story. Going 
>> forward, with project gates spending more time on relevant tests and 
>> more tests running post-merge, there are plenty of valuable black-box 
>> tests that we could have. Whitebox testing would be a valuable 
>> addition of course.
>>>
>>> I actually think the negative test generator code in tempest would 
>>> be a good candidate for tempest lib, and would actually work better 
>>> with those negative definitions in the project as functional tests.
>> ++
>>
>> I think a main reason more negative tests have not been created using 
>> the generator is that it requires deep knowledge of the schemas. 
>> Ideally, these schemas would be the output of some generator in each 
>> project, or provided by the project manually.
>>
>>  -David
>>>
>>>     -Sean
>>>
>>>> -Matt Treinish
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> -- 
>>> Sean Dague
>>> http://dague.net
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> -- 
> Sean Dague
> http://dague.net
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141030/45afe52e/attachment.html>