Open Stack

Hi Nachi,

While it is certainly useful to be able to test against nova-provisioned resources, we also need to be able to test in complete isolation - without nova.  For example, if I configure this security group rule, is it applied correctly as an iptables rule on the system and does the filtering work as expected?  Ideally we would exhaustively test this kind of use case without a VM and then have an integration test that only has to ensure that filtering can be applied successfully when a VM is involved - all the corner cases would already have been tested.

As per the testing pyramid, testing should be done at as low a level as possible, and that implies that we not test everything at the integration level and only vary the execution targets (fakevm, real vm).  We would ideally have unit tests that evaluate every single code path, functional tests that perform a representative evaluation of system interaction, and integration tests that ensure that all the pieces (i.e. Quantum and Nova) work together.


m.

On May 23, 2013, at 5:48 PM, Nachi Ueno <nachi at ntti3.com> wrote:

> Hi Maru
> 
> +1
> My intention for writing quantum-debug command is for that purpose.
> 
> Yamahata has proposed similar framework which is called fakevm (this
> one supports emulation of VM migrations)
> https://review.openstack.org/#/c/20617/ (currently it looks stopped though)
> 
> One possible another idea is let nova support this fakevm mode for
> quick integration tests.
> On this plan, we can use same integration test code for quick test and
> real test.
> 
> Best
> Nachi
> 
> 
> 
> 2013/5/23 Maru Newby <marun at redhat.com>:
>> Recently, I've been working with netns quite a bit.  This exposure has convinced me that we should be doing functional testing for OpenStack Networking without hypervisor VM's.  In many use cases, the VM is just a container for a network device, and a namespace could easily substitute for a VM.  Much of the project's functionality - security groups, bridge configuration (port, vlans and tunneling), physical network integration, software filtering and routing, load balancing - could be (mostly) tested without a VM in the mix.  Even better, most of the infrastructure for such testing could be isolated in network namespaces and cleanly built up and torn down.  This would allow tests to run with minimal risk of test artifacts building up - ideal for enabling a developer to run the tests early and often in their dev environment.
>> 
>> Currently, if testing exists for the functionality I'm talking about, it is either at the unit level (with system interaction mocked out), or at the integration/scenario level (with VM's) via Tempest.  Neither approach is appropriate for the kind of functional testing we need.  To that end, I'd like to propose that we add a functional testing path to Quantum - quantum.tests.functional - and that tests under that path use neither mocks nor VM's.
>> 
>> To be clear, this effort would in no way replace scenario and api testing in Tempest.  We need that too.  But if we are to continue to iterate rapidly while limiting regressions, we need to minimize the overhead of creating, running, and maintaining our tests.  With this in mind, I'd like to see us write functional tests at as low a level as possible.
>> 
>> Thoughts?
>> 
>> 
>> m.
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev