Open Stack

On Fri, 2013-05-17 at 07:52 -0400, Sean Dague wrote:
> I like your idea of adding metadata to test cases that we could generate 
> this later.
> 
> One of the reasons I was writing all the "Tempest Guide to .... " files 
> as README.rst was I'm planning on creating a docs job once we are done 
> that will rebuild qa.openstack.org to be a collection of those READMEs. 
> Going from rst -> html should be pretty easy, and will mean that changes 
> in the code will be the website.
> 
> I think something equivalent like this could really be useful as well 
> and part of that publishing stream.
> 
> Now it just needs a champion to dive in and make it happen! :)
> 

I really like idea about this overview table, and I would like to do it.
But to get real data to do it, it would require all those tags/attrs in
place.

So for start I would like to start maybe from existing state:
* create tool which generates table:
  * for each tempest/tests subfolder
  * mark columns for each where we have gate/smoke/positive/negative
labels
* show how colors would like there and also the links to the source etc
* in the meantime add those service-object/action labels
  * probably in external github repo, just as example how it could look
like and see how many of those tags it would really require
  * switch the table generation to those real tags
* later this would have to be automated somehow (=>update repo on commit
to tempest or to the descriptions if they would be kept separately)
  * can we use service hook on github repo for this?

Also I'm not sure about the possibility of maintaining those 'manually'
assigned states(colors).
* it could be maintained somewhere in db etc, so it would require access
auth etc etc
* better would be to keep rst somewhere with service-object/action +
state + description ... that could be merged together with tags from
tempest code and put together in that table
My main issue with this is, that it would require manual upkeep and that
those manually assigned states could divert from real state in tempest
code (like test cases rewritten/removed/whatever).
But whatever, this looks like thing that can be done as future
improvement.


> 	-Sean
> 
> On 05/17/2013 07:01 AM, Attila Fazekas wrote:
> > I like the ACC model, but it looks like too high level overview for starting,
> >   but it can be a good continuation.
> >
> > A simple model for mainly API testing can be good for now.
> >
> > Initial considerations:
> > * We have services or components and and sub-part of this at API level.
> >    (3D matrix mapped to 2D)
> > * We have operations to test
> > * We have things to verify
> > * I would like to know is something missing or not covered properly
> >
> > I have constructed a matrix, than I simplified it.
> >
> > Something like below as a colored HTML table seams OK.
> >
> > ------------------------------------------------------------------------
> > | component    | CRUD | list | associations | quota | security |  misc |
> > ------------------------------------------------------------------------
> > | nova-server  |      |      |              |       |          |       |
> > ------------------------------------------------------------------------
> > | swift-object |      |      |              |       |          |       |
> > ------------------------------------------------------------------------
> > | keystone-user|      |      |              |       |          |       |
> > ------------------------------------------------------------------------
> > | misc         |      |      |              |       |          |       |
> > ------------------------------------------------------------------------
> >
> > CRUD: create, read, update, get
> > list: list, list detailed, filtering, pagination
> > misc: action , bulk-action
> > security: tenant-isolates, admin-denied
> > association: member, attach, assign
> >
> >
> > * In every intersection we should have multiple links to the source
> >    (github, automatically generated link).
> > * One test case can be in multiple intersection
> > * every line and row should have a short description
> > * every intersection should have a short description
> > * based on the description anyone can color the intersections
> >    1. Black: Does not makes sense in this context
> >    2. Red: completely missing
> >    3. yellow: partially completed
> >    4. light green: good progress
> >    5. green: OK
> >    6. white: unknown
> >
> > We could have tables for different API versions or CLI ..
> >
> > In theory if we add enough meta-data (attribute) on test cases to be map able
> >   into this kind of table. The only paper work will be:
> >
> > * create short description about what we need to verify
> >   (likely the columns will tell it alone)
> > * Color the table
> >
> >
> > Possible addition:
> > * Links to additional resources like blueprints, bugs..
> 


-- 
Pavel Sedlák <psedlak at redhat.com>
Red Hat