Open Stack

On 3/27/2013 7:37 AM, Sean Dague wrote:
> I think there is value here, though tests like this are long enough 
> that we need to think really hard about if we'd run it in the gate. 
> And if not, what the value of having them in tempest is.
The main question I have is: what kinds of bugs will such tests likely 
find that the current tempest tests will not?
If these tests are making API calls that tempest does not, then tempest 
should be fixed to include those calls.
If the issue is that these calls in some particular order might cause 
bugs while the way tempest makes the calls does not show the bugs, we 
should have tests that stress that ordering.
But there is no reason to think that such stress will be related to some 
call order that makes sense to a user. From the system view, there are 
lots of users and their
requests are handled in an arbitrary interleaved way.

I think what we really need are better stress tests, and to run them 
regularly. The existing stress tests found a lot of bugs at the 
beginning but they are overly complex in part because they duplicate current
tempest abstractions that were not present at the time the stress tests 
were created. They are also lamed by the spurious ERRORs in log files 
even when nothing is wrong, a problem which is being worked on. The 
kinds of scenarios suggested here could be good building blocks for 
stress tests.

  -David

>
> As we've seen with the stress tests, they bit rot really quickly. So 
> if there aren't regular runs of these tests being reported back to the 
> community, their value is quite minimal (if not completely counter 
> productive, because they don't work).
>
> It might also be worth putting these in a fully separate project tree, 
> because I can see the value in running these tests with the official 
> clients.
>
>     -Sean
>
> On 03/26/2013 08:33 PM, Masayuki Igawa wrote:
>> Hi,
>>
>> Here is a blueprint I've registered.
>> https://blueprints.launchpad.net/tempest/+spec/add-scenario-tests
>>
>> This blueprint proposes to add a new category "scenario tests" in 
>> Tempest.
>> Scenario tests are for testing across the components such as Nova, 
>> Keystone, Glance and so on.
>> By scenario tests, developers will be able to check whether a new 
>> code (bug-fixes or features) will cause a side effect to the other 
>> components.
>> We will be able to check OpenStack basic functions by common use case 
>> scenarios.
>> I'd like to propose a minimum basic scenario test that must work at 
>> any OpenStack environments.
>>
>> Implementation
>> ===============
>> * add tempest/tests/scenario directory
>> * add scenario tests in the directory like following
>>
>> """
>> Scenario-1: very small cloud, basic life-cycle scenario, 
>> single-flavor, single-image, single-tenant...
>> - for admin user
>> 1. Create a flavor
>> 2. Create a image
>> 3. Configure network settings
>> 4. Create & configure a member role, a project, a quota, a user
>> - for regular user
>> 5. Create a key-pair
>> 6. Boot a instance. List & show the instance
>> 7. Create a volume. List & show the volume
>> 8. Attach the volume
>> 9. Detach the volume
>> 10. Delete the instance
>> - for admin user
>> 11. Delete the user, the role, the network, the image, the tenant, 
>> and the flavor
>> """
>>
>> We think that the scenario tests can raise the use case coverage that 
>> is important for user's viewpoint.
>>
>> We'd like to discuss the following points, so any comments are welcome.
>>   * Points of the implementation
>>     * How do we implement the scenario tests?
>>       * via CLI/RestClient/client library...
>>       * Where do we implement in the directory?
>>     * Points of Scenarios
>>       * scenario categories: Private cloud, Public cloud, VPC
>>       * the scale: # of tenants, users, networks, and so on
>>       * the way of verifying: RESTful APIs, ping, ssh, and so on
>>
>>
>> Thanks,
>> Masayuki Igawa
>>
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>
>