Open Stack

On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote:
> On 7/21/2016 2:03 AM, Bhor, Dinesh wrote:
> > Hi Nova Devs,
> > 
> > 
> > 
> > Many times, there are a number of data sets that we have to run the same
> > tests on.
> > 
> > And, to create a different test for each data set values is
> > time-consuming and inefficient.
> > 
> > 
> > 
> > Data Driven Testing [1] overcomes this issue. Data-driven testing (DDT)
> > is taking a test,
> > 
> > parameterizing it and then running that test with varying data. This
> > allows you to run the
> > 
> > same test case with many varying inputs, therefore increasing coverage
> > from a single test,
> > 
> > reduces code duplication and can ease up error tracing as well.
> > 
> > 
> > 
> > DDT is a third party library needs to be installed separately and invoke the
> > 
> > module when writing the tests. At present DDT is used in cinder and rally.
> 
> There are several projects using it:
> 
> http://codesearch.openstack.org/?q=ddt%3E%3D1.0.1&i=nope&files=&repos=
> 
> I first came across it when working a little in manila.
> 
> > 
> > 
> > 
> > To start with, I have reported this as a bug [2] and added initial patch
> > [3] for the same,
> > 
> > but couple of reviewers has suggested to discuss about this on ML as
> > this is not a real bug.
> > 
> > IMO this is not a feature implementation and it’s just a effort for
> > simplifying our tests,
> > 
> > so a blueprint will be sufficient to track its progress.
> > 
> > 
> > 
> > So please let me know whether I can file a new blueprint or nova-specs
> > to proceed with this.
> > 
> > 
> > 
> > [1] http://ddt.readthedocs.io/en/latest/index.html
> > 
> > [2] https://bugs.launchpad.net/nova/+bug/1604798
> > 
> > [3] https://review.openstack.org/#/c/344820/
> > 
> > 
> > 
> > Thank you,
> > 
> > Dinesh Bhor
> > 
> > 
> > ______________________________________________________________________
> > Disclaimer: This email and any attachments are sent in strictest confidence
> > for the sole use of the addressee and may contain legally privileged,
> > confidential, and proprietary data. If you are not the intended recipient,
> > please advise the sender by replying promptly to this email and then delete
> > and destroy this email and any attachments without any further use, copying
> > or forwarding.
> > 
> > 
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> I agree that it's not a bug. I also agree that it helps in some specific
> types of tests which are doing some kind of input validation (like the patch
> you've proposed) or are simply iterating over some list of values (status
> values on a server instance for example).
> 
> Using DDT in Nova has come up before and one of the concerns was hiding
> details in how the tests are run with a library, and if there would be a
> learning curve. Depending on the usage, I personally don't have a problem
> with it. When I used it in manila it took a little getting used to but I was
> basically just looking at existing tests and figuring out what they were
> doing when adding new ones.

I don't think there's significant learning curve there - the way it
lets you annotate the test methods is pretty easy to understand and
the ddt docs spell it out clearly for newbies. We've far worse things
in our code that create a hard learning curve which people will hit
first :-)

People have essentially been re-inventing ddt in nova tests already
by defining one helper method and them having multiple tests methods
all calling the same helper with a different dataset. So ddt is just
formalizing what we're already doing in many places, with less code
and greater clarity.

> I definitely think DDT is easier to use/understand than something like
> testscenarios, which we're already using in Nova.

Yeah, testscenarios feels little over-engineered for what we want most
of the time.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|