Open Stack

Maru,

I've seen your changes and I'm totally fine with them.
I think it makes sense for you to lead this part of the discussion, since
you've studied the subject much more than me!

One caveat: during the discussion let's remember to keep focused on how we
should rework the unit tests, avoiding the risk of falling into a
discussion on unit vs functional vs integration tests, which, even if
interesting, is probably not what the audience might expect from this
session.

See you tomorrow,
Salvatore


On 14 April 2013 19:22, Maru Newby <marun at redhat.com> wrote:

> Hi Salvatore,
>
> I hope you don't mind, but I've fleshed out the 'types of tests' section
> to provide more background.  It was only afterwards that I thought of
> checkpointing before I started to allow you to revert if my changes don't
> suit your intensions.
>
> My goal was to achieve some clarity on different types of testing so that
> reviewers will be in a better position to evaluate test quality.
>
> Examples for refactoring to follow.
>
> Cheers,
>
>
> Maru
>
> On Apr 12, 2013, at 9:33 AM, Salvatore Orlando <sorlando at nicira.com>
> wrote:
>
> > I am preparing an etherpad for this session -
> https://etherpad.openstack.org/havana-quantum-unittests
> > There is a session for restructuring unit tests in which you can put
> links to code samples we can discuss in the summit session.
> > I am unable to give an exact timing for walking through the examples, as
> it depends on how many samples we have. Looking at the other topics in the
> etherpad, and the time it might take to discuss them, I guess we can spend
> about half of the session discussing code.
> >
> > Salvatore
> >
> >
> > On 12 April 2013 09:12, Maru Newby <marun at redhat.com> wrote:
> > Hi Salvatore,
> >
> > I will be happy to provide some samples and walk through refactoring
> them.  How much time would intend for such an exercise so that I could
> target my effort?
> >
> > Cheers,
> >
> >
> > Maru
> >
> > On Apr 11, 2013, at 2:04 PM, Salvatore Orlando <sorlando at nicira.com>
> wrote:
> >
> > > [REPOSTING THIS REPLY]
> > >
> > > Please let me know if others want to jump in the discussion on Quantum
> unit tests
> > >
> > > Hi Maru,
> > >
> > > thanks for contributing to this topic.
> > > More comments inline.
> > >
> > >
> > > I was thinking of providing code samples during the summit session
> explaining how existing tests could be refactored. If you can provide some
> samples it would be great; also if you're ok with walking the audience
> through these samples, that would be awesome!
> > >
> > > Salvatore
> > >
> > >
> > > On 4 April 2013 03:34, Maru Newby <marun at redhat.com> wrote:
> > > Hi Salvatore,
> > >
> > > I'm working on a patch to swap pep8 for flake8, which will increase
> the automation of mechanical review.  In the process of doing the cleanup
> required for the patch, I have seen some ways that quantum's test suite
> could be improved:
> > >
> > >  -  An api method and its serialization format (xml, json) are
> orthogonal (or should be) and could be tested as such.  For plugins that
> test both serialization formats for each api method, test execution time is
> effectively doubled with no benefit to coverage.
> > >
> > > I totally agree. Once we get past the WSGI layer, there is not
> difference between the JSON or the XML use case. The Serialization tests
> should just validate data serialization and deserialization.
> > >
> > >
> > >  - Many plugins contain duplicate code and tests (e.g.
> get_port_from_device and its associated tests appear in each of the nec,
> openvswitch, ryu, and linuxbridge plugins).  Common code could be factored
> out and tested only once.
> > >
> > > That's another good point that I wanted to address. It seems these
> tests have the same coverage, so, as you say, they should be factored out
> in a test module aimed at covering that specific functionality.
> > >
> > >
> > > - Many quantum tests evaluate low-level functionality through a web
> api rather than isolating the functionality under test.  Writing more
> focused unit tests (and fewer functional and integration tests ala
> http://www.martinfowler.com/bliki/TestPyramid.html) could allow for
> better coverage at a lower cost in both maintenance and execution.
> > >
> > > This is the first goal of reworking our unit tests. We have always
> been aware that those were not unit tests. However, those were the closest
> think we could provide to a gatetests. Now that we have proper gate tests -
> we should make sure our unit tests are proper unit tests! Also, it would be
> good to discuss what to make of the current unit tests, which are probably
> suitable for some sort of automated functional testing.
> > >
> > >
> > > The technical aspects of these potential improvements are
> straightforward, but will require a degree of developer and reviewer
> education to ensure that they are properly implemented and maintained going
> forward.  Do you think the summit session you've proposed would be an
> appropriate venue, or should it be done separately?
> > >
> > > I think the summit session is just a starter. 'Education' should be
> enforced via code reviews. The main goal of the summit session is achieve
> consensus on the general direction, identify priorities, and possibly
> resources.
> > >
> > >
> > >
> > > Cheers,
> > >
> > >
> > > Maru
> > >
> > >
> > > On 3 April 2013 18:34, Maru Newby <marun at redhat.com> wrote:
> > > Hi Salvatore,
> > >
> > > I'm working on a patch to swap pep8 for flake8, which will increase
> the automation of mechanical review.  In the process of doing the cleanup
> required for the patch, I have seen some ways that quantum's test suite
> could be improved:
> > >
> > >  -  An api method and its serialization format (xml, json) are
> orthogonal (or should be) and could be tested as such.  For plugins that
> test both serialization formats for each api method, test execution time is
> effectively doubled with no benefit to coverage.
> > >
> > >  - Many plugins contain duplicate code and tests (e.g.
> get_port_from_device and its associated tests appear in each of the nec,
> openvswitch, ryu, and linuxbridge plugins).  Common code could be factored
> out and tested only once.
> > >
> > > - Many quantum tests evaluate low-level functionality through a web
> api rather than isolating the functionality under test.  Writing more
> focused unit tests (and fewer functional and integration tests ala
> http://www.martinfowler.com/bliki/TestPyramid.html) could allow for
> better coverage at a lower cost in both maintenance and execution.
> > >
> > > The technical aspects of these potential improvements are
> straightforward, but will require a degree of developer and reviewer
> education to ensure that they are properly implemented and maintained going
> forward.  Do you think the summit session you've proposed would be an
> appropriate venue, or should it be done separately?
> > >
> > > Cheers,
> > >
> > >
> > > Maru
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130414/8244eca6/attachment.html>