Open Stack

On Wed, 26 Mar 2014 21:02:58 -0700
Maru Newby <marun at redhat.com> wrote:
> > 
> > If an API is made and documented properly what domain expertise
> > would be needed to use it? The opposite is true for tempest and the
> > tests themselves. The tempest team focuses on just tests so they
> > know how to write good tests and are able to leverage common
> > underlying framework code.
> 
> Given that documentation is typically finalized only late in the
> cycle, are you suggesting that we forego api testing until possibly
> well after the code has been written?  Plus, it is a rare api that
> doesn't have to evolve in response to real-world experience with
> early implementations.  The sooner we can write functional api tests,
> the sooner we can identify shortcomings that need to be addressed -
> and the less costly they will be to fix.

So although "proper" documentation may only be finalized late in the
cycle, there really should be a specification of the API and how
it behaves done before the implementation is finished. 

At least in Nova land I think the lack of this has been a major cause
of features having trouble merging and also us having flaws in both our
implementation (semantic behaviour which was accidental rather than
designed) and testing (incomplete coverage).

Also we have API unit testing in Nova but the tempest API testing still
ends up picking up more issues (perhaps because its generally not the
person writing the code who ends up writing the tests). I think it also
increases the chance that a backwards incompatible
API changes will get picked up. 

Chris