Open Stack

On 01/19/2014 02:06 AM, Yair Fried wrote:
> MT:"Is your issue here that it's just called basic ops and you don't think that's
> reflective of what is being tested in that file anymore"
> 
> No.
> My issue is, that the current scenario is, in fact, at least 2 separate scenarios:
> 1. original basic_ops
> 2. reassociate_floating_ip
> to which I would like to add ( https://review.openstack.org/#/c/55146/ ):
> 4. check external/internal connectivity
> 3. update dns
> 
> While #2 includes #1 as part of its setup, its failing shouldn't prevent #1 from passing. the obvious solution would be to create separate modules for each test case, but since they all share the same setup sequence, IMO, they should at least share code.
> Notice that in my patch, #2 still includes #1.
> 
> Actually, the more network scenario we get, the more we will need to do something in that direction, since most of the scenarios will require the setup of a VM with a floating-ip to ssh into.
> 
> So either we do this, or we move all of this code to scenario.manager which is also becoming very complicated

If #2 is always supposed to work, then I don't actually understand why
#1 being part of the test or not part of the test is really relevant.
And being part of the same test saves substantial time.

If you have tests that do:
 * A -> B -> C
 * A -> B -> D -> F

There really isn't value in a test for A -> B *as long* as you have
sufficient sign posting to know in the fail logs that A -> B worked fine.

And there are sufficient detriments in making it a separate test,
because it's just adding time to the runs without actually testing
anything different.

	-Sean

> 
> Yair
> 
> ----- Original Message -----
> From: "Matthew Treinish" <mtreinish at kortar.org>
> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
> Sent: Friday, January 17, 2014 6:17:55 AM
> Subject: Re: [openstack-dev] [qa][Neutron][Tempest][Network] Break down NetworkBasicOps to smaller test cases
> 
> On Wed, Jan 15, 2014 at 11:20:22AM -0500, Yair Fried wrote:
>> Hi Guys
>> As Maru pointed out - NetworkBasicOps scenario has grown out of proportion and is no longer "basic" ops.
> 
> Is your issue here that it's just called basic ops and you don't think that's
> reflective of what is being tested in that file anymore. If that's the case
> then just change the name.
> 
>>
>> So, I started breaking it down to smaller test cases that can fail independently.
> 
> I'm not convinced this is needed. Some scenarios are going to be very involved
> and complex. Each scenario tests is designed to simulate real use cases in the
> cloud, so obviously some of them will be fairly large. The solution for making
> debugging easier in these cases is to make sure that any failures have clear
> messages. Also make sure there is plenty of signposting debug log messages so
> when something goes wrong you know what state the test was in. 
> 
> If you split things up into smaller individual tests you'll most likely end up
> making tests that are really aren't scenario tests. They'd be closer to API
> tests, just using the official clients, which really shouldn't be in the
> scenario tests.
> 
>>
>> Those test cases share the same setup and tear-down code:
>> 1. create network resources (and verify them)
>> 2. create VM with floating IP.
>>
>> I see 2 options to manage these resources:
>> 1. Completely isolated - resources are created and cleaned using setUp() and tearDown() methods [1]. Moving cleanup to tearDown revealed this bug [2]. Apparently the previous way (with tearDownClass) wasn't as fast). This has the side effect of having expensive resources (ie VMs and floating IPs) created and discarded  multiple times though they are unchanged.
>>
>> 2. Shared Resources - Using the idea of (or actually using) Fixtures - use the same resources unless a test case fails, in which case resources are deleted and created by the next test case [3].
> 
> If you're doing this and splitting things into smaller tests then it has to be
> option 1. Scenarios have to be isolated if there are resources shared between
> scenario tests that really is only one scenario and shouldn't be split. In fact
> I've been working on a change that fixes the scenario test tearDowns that has the
> side effect of enforcing this policy.
> 
> Also just for the record we've tried doing option 2 in the past, for example
> there used to be a tenant-reuse config option. The problem with doing that was
> actually tends to cause more non-deterministic failures or adding a not
> insignificant wait time to ensure the state is clean when you start the next
> test. Which is why we ended up pulling this out of tree. What ends up happening
> is that you get leftover state from previous tests and the second test ends up
> failing because things aren't in the clean state that the test case assumes. If
> you look at some of the oneserver files in the API that is the only place we
> still do this in the tempest, and we've had many issues on making that work
> reliably. Those tests are in a relatively good place now but those are much
> simpler tests. Also between each test setUp has to check and ensure that the
> shared server is in the proper state. If it's not then the shared server has to
> be rebuilt. This methodology would become far more involved for the scenario
> tests where you have to manage more than one shared resource.
> 
>>
>> I would like to hear your opinions, and know if anyone has any thoughts or ideas on which direction is best, and why.
>>
>> Once this is completed, we can move on to other scenarios as well
>>
>> Regards
>> Yair
>>
>> [1] fully isolated - https://review.openstack.org/#/c/66879/
>> [2] https://bugs.launchpad.net/nova/+bug/1269407/+choose-affected-product
>> [3] shared resources - https://review.openstack.org/#/c/64622/
> 
> -Matt Treinish
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140119/6977a815/attachment.pgp>