Open Stack

Hi, Sean.

Thank you very much for your comments.

I submitted a prototype but I don't stick to that.
If there are better ideas, I'd like to adopt them.
Here are my candidates. I think first one is good because it is simple and insusceptible from other factors.
1) Use middleware to catch APIs.
   This is my prototype. This supplies necessary functions almost all in one.
2) Use nova-api log.
   I've implemented this before but I found this was not helpful because the log format changes frequently.
3) Use nova coverage report.
   We can use existing code, but source code analysis is needed I think and I don't know how to realize it.
4) Use tempest log.
   I worry about the same problem as 2).
5) Implement a new function to record tested APIs on tempest instead of using logs.
   This needs a function on Nova to get API lists and check.

> For instance, the 132 tests on /project_id/servers tell you very little on whether we are passing good or bad attributes to it.

That's right.
We should implement request body analysis in the next step.

Any suggestions?

Thanks.
Mitsuhiko Yamazaki

-----Original Message-----
From: Sean Dague [mailto:sdague at gmail.com] On Behalf Of Sean Dague
Sent: Tuesday, April 02, 2013 7:52 PM
To: All Things QA.
Cc: Mitsuhiko Yamazaki; openstack-dev at lists.openstack.org
Subject: Re: [openstack-qa] Blueprint: list-tested-apis

Is there a reason you aren't using the nova coverage extension instead? 
I think if the eventlet issue was addressed we'd be able to get a much 
more consistent report out of that.

For instance, the 132 tests on /project_id/servers tell you very little 
on whether we are passing good or bad attributes to it.

I like the eventual format, but it feel like we should try to leverage 
the existing code instead of building yet another coverage counter.

	-Sean

On 04/02/2013 01:49 AM, Mitsuhiko Yamazaki wrote:
> Hi,
>
> Here is a blueprint I've registered.
> https://blueprints.launchpad.net/nova/+spec/list-tested-apis
>
> Now Tempest team adds test cases aggressively for quality assurance.
> This blueprint intends to find effectively which test case should be added.
>
> As far as I investigated, Nova has 240 RESTful APIs, but only 30%(75/240) APIs are tested in Tempest.
> The method of this blueprint enables developers to find which API is tested using new APIs(start/stop recording, create a report).
> This is useful for developers, and it would promote the test development.
>
> At first, my target is to detect which API is called in Tempest.
> The prototype code is registered at the following URL and we can get a report like below.
> https://review.openstack.org/#/c/25882/
> ---------report output------------
> Tested API: 75/240
> Tested No  Method Path                                                         Action
>       0 000 GET    /
>       0 001 POST   /{project_id}/servers/:server_id/consoles
>       0 002 GET    /{project_id}/servers/:server_id/consoles
>       0 003 PUT    /{project_id}/servers/:server_id/consoles/:(id)
>       0 004 DELETE /{project_id}/servers/:server_id/consoles/:(id)
>       0 005 GET    /{project_id}/servers/:server_id/consoles/:(id)
>     132 006 POST   /{project_id}/servers
>      13 007 GET    /{project_id}/servers/detail
>      41 008 GET    /{project_id}/servers
>      14 009 PUT    /{project_id}/servers/:(id)
>       0 010 POST   /{project_id}/servers/:(id)/action                           restore
>       1 011 POST   /{project_id}/servers/:(id)/action                           suspend
>       0 012 POST   /{project_id}/servers/:(id)/action                           migrate
>     [...]
>       0 239 GET    /{project_id}/os-snapshots/:(id)
> --------------------------------
>
> We'd like to discuss the design. Any comments are welcome.
>
> Thanks,
> Mitsuhiko Yamazaki
>
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>


-- 
Sean Dague
http://dague.net