Open Stack

On 8/10/2012 11:16 AM, Kevin L. Mitchell wrote:
> On Fri, 2012-08-10 at 08:01 -0500, Christopher B Ferris wrote:
>> The next sentence in HACKING.rst goes a step further and requires that
>> bug fixes for features that lack unit tests add them. Given that the
>> XML API code has been around for a while, it begs the question: why is
>> the coverage so poor? Collectively, we need to fix both the symptom
>> and the underlying cause.
> It's important to note that the unit test coverage is not poor; for
> almost everything for which there's an XML template, there is also unit
> tests covering that template serialization.  (The weasel word is only
> because I haven't done a thorough review to verify my assertion for 100%
> of cases.)  For that matter, when I contributed the XML template code, I
> went through all existing extensions and created templates, even for
> those that did not have XML support at the time, and when I do reviews
> for new extensions, I try to ensure that there is a template for those
> extensions.
>
> The problem comes from the fact that there are several extensions that
> have been added which I personally did not review, and for which there
> exists no XML template.  The unit tests are perfectly adequate to test
> those new extensions, but there is obviously no new serializer, and thus
> no serializer unit tests.  It isn't that our test coverage is poor, it
> is that our code is actually deficient, and we reviewers are directly to
> blame for not enforcing that all API extensions must have working and
> tested XML handling.
>
> Note also that the unit tests are orthogonal to the tests being
> discussed in this thread, which are _integration_ tests; those test
> suites are apparently, to date, not testing the XML side of the API.
> Remember also that these test suites are *outside* the nova code base,
> and would also have to be updated to cover any newly-added extension;
> I'm not currently aware of any process in place which ensures that they
> get updated in that situation.
There has been a lot of discussion about this issue recently among the 
Tempest team. At a minimum, a message should be posted to the openstack 
qa list every
time a new api (or any feature) is checked in to master. Please do it!  
Ideally, QA would be involved at an earlier stage and integration test 
plans would be represented in the blueprints. We are working on this.

As to why Tempest currently has no XML client, there are several 
reasons. One is that Daryl has been working on it. Another is that 
Tempest is quite young and the first priority was having a suite to 
detect regressions in or across projects and unauthorized API changes. 
The lack of specs for many features was a huge obstacle. The number of 
active contributors to Tempest is small given the ever-growing volume of 
new features and projects. I also have to admit, at least for myself, 
that the presence of so many new apis in nova and entire projects that 
do not have xml specs or support biased me towards thinking it was less 
important than perhaps it should have been. Mea culpa.

Insufficient product and engineering management and the associated lack 
of accountability is at the root of most of these issues. This is really 
hard in a truly global open-source project. But we can and need to do 
better.

  -David