Open Stack

On 5/2/2013 9:59 AM, Sean Dague wrote:
> On 05/01/2013 04:08 PM, David Kranz wrote:
>> The stress tests in Tempest are never run. They are also incomplete and
>> too complicated, though
>> they did find many bugs when they were first put in use.
>> The history of the complexity is that they tried to hit the system as
>> hard as possible within a single thread and
>> were created before there were stable Tempest rest clients. I would
>> really like to see these tests
>> run so I uploaded https://review.openstack.org/#/c/27950/ which is a
>> drastically simplified version.
>>
>> This version spawns multiple processes to do the bashing and each action
>> script is a simple "scenario".
>> I wanted to get some opinions about this as a direction for stress
>> tests. There is currently just a simple
>> create/destroy test there but it does sometimes fail with an error in
>> the log for which there is a bug
>> which was closed as fixed.
>>
>> After getting any feedback, I would like to add some more scenarios and
>> put this in a periodic job.
>> There is still a big problem in that ERRORs are still written into nova
>> logs when there s actually nothing
>> wrong. There are bugs filed about that but I believe the problem is
>> still there. The stress tests cannot be
>> really useful unless clean runs produce no log ERRORS.
>
> I like the idea (just reading through the code now), but in my ideal 
> world it would be nice to have the error log handling don't through a 
> nova extension, so we could fully orchestrate this via APIs instead of 
> requiring setting up ssh credentials to go find logs (it feels like 
> that could break down a little too easily).
>
> If adding a nova extension for the log fetch is time prohibitive at 
> this point, this would still be a big win over what we've got.
>
>     -Sean
>
What you see is basically the same mechanism as was used by the original 
stress tests. I tend to agree with your ideal but:

1. Many real systems (and devstack does as well) will have central 
logging of errors so it may not break down for the cases we care about
2. We have not been able to get the false errors out of the nova logs. 
We should do that before requesting a new API :-) .

  -David