Open Stack

On Fri, Jun 17, 2016, at 04:16 PM, Matt Riedemann wrote:
> I was reviewing this today:
> 
> https://review.openstack.org/#/c/326940/
> 
> And I said to myself, 'self, do we really need to subclass the API 
> samples functional tests for this microversion given this change doesn't 
> modify the request/response body, it's only adding paging support?'.
> 
> https://review.openstack.org/#/c/326940/6/nova/tests/functional/api_sample_tests/test_hypervisors.py
> 
> The only change here is listing hypervisors, and being able to page on 
> those if the microversion is high enough. So the API samples don't 
> change at all, they are just running against a different microversion.
> 
> The same goes for the REST API unit tests really:
> 
> https://review.openstack.org/#/c/326940/6/nova/tests/unit/api/openstack/compute/test_hypervisors.py
> 
> I'm not sure if the test subclassing is just done like this for new 
> microversions because it's convenient or if it's because of regression 
> testing - knowing that we aren't changing a bunch of other REST methods 
> in the process, so the subclassed tests aren't testing anything 
> different from the microversion that came before them.
> 
> The thing I don't like about the test subclassing is all of the 
> redundant testing that goes on, and people might add tests to the parent 
> class not realizing it's subclassed and thus duplicating test cases with 
> no functional change.

I agree that the naive subclassing is wasteful. I would rather see tests
that purposely check the changes not just duplicate things at a
different microversion. If there's a concern about regressions I think a
better approach would be to have a check against 'latest' which would
catch accidental changes, not check unchanged request/reponses against
microversions where they aren't intended to change. For my own curiosity
I checked the timings and the duplicated tests took about 20 seconds to
run. That could quickly add up to the point where a significant amount
of time is spent on unnecessary testing.

> 
> Am I just having some Friday crazies? Ultimately this doesn't hurt 
> anything really but thought I'd ask.
> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev