Open Stack

On Mon, May 27, 2013 at 11:50:02AM -0700, Maru Newby wrote:
> 
> On May 24, 2013, at 3:54 AM, Isaku Yamahata <yamahata at valinux.co.jp> wrote:
> 
> > Hi.
> > 
> > 
> > On Thu, May 23, 2013 at 07:08:26PM -0700, Maru Newby wrote:
> >> Hi Nachi,
> >> 
> >> While it is certainly useful to be able to test against nova-provisioned resources, we also need to be able to test in complete isolation - without nova.  For example, if I configure this security group rule, is it applied correctly as an iptables rule on the system and does the filtering work as expected?  Ideally we would exhaustively test this kind of use case without a VM and then have an integration test that only has to ensure that filtering can be applied successfully when a VM is involved - all the corner cases would already have been tested.
> >> 
> > 
> > Great. You've also reached to same conclusion. When I proposed it,
> > no one seemed interested in it unfortunately. So I've suspended the effort.
> > If there are interests, I'm willing to resume it.
> > Right now, I have semi-working fakevm on the master branch.
> 
> I agree, we have reached a similar conclusion.  However, my immediate proposal is intended to test system interaction without a VM - even a fake one.  I want to see functional testing for all the primitives that make up system interaction (create bridge, create veth, apply firewall rule, etc) before we consider testing how those primitives will combine to support VM connectivity.  I would characterize this as needing to walk before running, or building bottom-up vs top-down.  I would advocate for this approach because, as per the testing pyramid, it is hard to know what still needs testing at a higher level if the lower level of testing isn't at least minimally fleshed out.

Hmm the name, fakevm, seems somewhat misleading you.
It can exercise plug/unplug port with vif driver or interface driver.
i.e. creating veth and plug into ovs port.
It is exactly what you described above, I think.

firewall stuff needs more enhancement, though.

thanks,

> m.
> 
> > 
> > 
> >> As per the testing pyramid, testing should be done at as low a level as possible, and that implies that we not test everything at the integration level and only vary the execution targets (fakevm, real vm).  We would ideally have unit tests that evaluate every single code path, functional tests that perform a representative evaluation of system interaction, and integration tests that ensure that all the pieces (i.e. Quantum and Nova) work together.
> > 
> > Yes, that is exactly my vision with fakevm.
> > And emulation of multi cpu-node within single physical machine is greatly
> > helpful for developers of networking. And it will help for exercising
> > various plugins. It is quite difficult/time-consuming to test gre-tunneling
> > or similar features that requires multi cpu node.
> > 
> > My plan goal is 
> > - make fakevm work again and fill the missing peace for the later step
> > - create test with only quantum without nova (except nova vif driver)
> > - push those tests to somewhere for test gating
> >  under quantum test or devstack with special setting in localrc or
> >  tempest with special setting.
> >  I'm not sure where this kind of tests fit in.
> > 
> > thanks,
> > 
> >> m.
> >> 
> >> On May 23, 2013, at 5:48 PM, Nachi Ueno <nachi at ntti3.com> wrote:
> >> 
> >>> Hi Maru
> >>> 
> >>> +1
> >>> My intention for writing quantum-debug command is for that purpose.
> >>> 
> >>> Yamahata has proposed similar framework which is called fakevm (this
> >>> one supports emulation of VM migrations)
> >>> https://review.openstack.org/#/c/20617/ (currently it looks stopped though)
> >>> 
> >>> One possible another idea is let nova support this fakevm mode for
> >>> quick integration tests.
> >>> On this plan, we can use same integration test code for quick test and
> >>> real test.
> >>> 
> >>> Best
> >>> Nachi
> >>> 
> >>> 
> >>> 
> >>> 2013/5/23 Maru Newby <marun at redhat.com>:
> >>>> Recently, I've been working with netns quite a bit.  This exposure has convinced me that we should be doing functional testing for OpenStack Networking without hypervisor VM's.  In many use cases, the VM is just a container for a network device, and a namespace could easily substitute for a VM.  Much of the project's functionality - security groups, bridge configuration (port, vlans and tunneling), physical network integration, software filtering and routing, load balancing - could be (mostly) tested without a VM in the mix.  Even better, most of the infrastructure for such testing could be isolated in network namespaces and cleanly built up and torn down.  This would allow tests to run with minimal risk of test artifacts building up - ideal for enabling a developer to run the tests early and often in their dev environment.
> >>>> 
> >>>> Currently, if testing exists for the functionality I'm talking about, it is either at the unit level (with system interaction mocked out), or at the integration/scenario level (with VM's) via Tempest.  Neither approach is appropriate for the kind of functional testing we need.  To that end, I'd like to propose that we add a functional testing path to Quantum - quantum.tests.functional - and that tests under that path use neither mocks nor VM's.
> >>>> 
> >>>> To be clear, this effort would in no way replace scenario and api testing in Tempest.  We need that too.  But if we are to continue to iterate rapidly while limiting regressions, we need to minimize the overhead of creating, running, and maintaining our tests.  With this in mind, I'd like to see us write functional tests at as low a level as possible.
> >>>> 
> >>>> Thoughts?
> >>>> 
> >>>> 
> >>>> m.
> >>>> 
> >>>> 
> >>>> 
> >>>> _______________________________________________
> >>>> OpenStack-dev mailing list
> >>>> OpenStack-dev at lists.openstack.org
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>> 
> >>> _______________________________________________
> >>> OpenStack-dev mailing list
> >>> OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> 
> >> 
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> 
> > 
> > -- 
> > yamahata
> > 
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
yamahata