Open Stack

On 12/1/2015 3:02 PM, Matt Riedemann wrote:
>
>
> On 12/1/2015 2:07 PM, Joshua Harlow wrote:
>> So whats needed to build out that 'framework'?
>>
>> Is it just dispatching provision requests to nova, and then seeing when
>> the quota becomes out of sync, and then backtracking through logs and
>> notifications (+- other) to then figure out the root cause?
>>
>> Or is that framework some kind of local functional testing built-in to
>> nova itself, something that can work without probing via the REST
>> endpoints...?
>>
>> IMHO it'd be nice to have something like rally scenarios that probe the
>> quota (via REST or not) but with post-validation that can be done on the
>> post scenario run and/or post request run. The post-validation ensures
>> the quota is as expected, using some locally computed 'expected' quota
>> that nova should also have locally computed, if those two computed
>> 'expected' quotas are different, then backtracking/log analysis... needs
>> to occur to figure out when the computed values started to diverge (and
>> to figure out why they diverged); rinse and repeat that and u probably
>> have a pretty powerful validation framework...
>>
>> Vilobh Meshram wrote:
>>> I am highly supportive for the idea of Nova Quota sub-team, for
>>> something as complex as Quota, as it helps to move quickly on reviews
>>> and changes.
>>>
>>> Agree with John, a test framework to test quotas will be helpful and can
>>> be one of the first task the Nova Quota sub team can focus on as that
>>> will lay the foundation for whether the bugs mentioned here
>>> http://bit.ly/1Pbr8YL are valid or not.
>>>
>>> Having worked in the area of Quotas for a while now by introducing
>>> features like Cinder Nested Quota Driver [1] [2] I strongly feel that
>>> something like a Nova Quota sub-team will definitely help. Mentioning
>>> about Cinder Quota driver since it was accepted in Mitaka design summit
>>> that Nova Nested Quota Driver[3] would like to pursue the route taken by
>>> Cinder.  Since Nested quota is a one part of Quota subsystem and working
>>> in small team helped to iterate quickly for Nested Quota
>>> patches[4][5][6][7] so IMHO forming a Nova quota subteam will help.
>>>
>>> Melanie,
>>>
>>> If you can share the details of the bug that Joe mentioned, to reproduce
>>> quota bugs locally, it would be helpful.
>>>
>>> -Vilobh (irc: vilobhmm)
>>>
>>> [1] Code : https://review.openstack.org/#/c/205369/
>>> [2] Blueprint :
>>> https://blueprints.launchpad.net/cinder/+spec/cinder-nested-quota-driver
>>> [3] Nova Nested Quota Spec : https://review.openstack.org/#/c/209969/
>>> [4] https://review.openstack.org/#/c/242568/
>>> [5] https://review.openstack.org/#/c/242500/
>>> [6] https://review.openstack.org/#/c/242514/
>>> [7] https://review.openstack.org/#/c/242626/
>>>
>>>
>>> On Mon, Nov 30, 2015 at 10:59 AM, melanie witt <melwittt at gmail.com
>>> <mailto:melwittt at gmail.com>> wrote:
>>>
>>>     On Nov 26, 2015, at 9:36, John Garbutt <john at johngarbutt.com
>>>     <mailto:john at johngarbutt.com>> wrote:
>>>
>>>     >  A suggestion in the past, that I like, is creating a nova
>>> functional
>>>     >  test that stress tests the quota code.
>>>     >
>>>     >  Hopefully that will be able to help reproduce the error.
>>>     >  That should help prove if any proposed fix actually works.
>>>
>>>     +1, I think it's wise to get some data on the current state of
>>>     quotas before choosing a redesign. IIRC, Joe Gordon described a test
>>>     scenario he used to use to reproduce quota bugs locally, in one of
>>>     the launchpad bugs. If we could automate something like that, we
>>>     could use it to demonstrate how quotas currently behave during
>>>     parallel requests and try things like disabling reservations. I also
>>>     like the idea of being able to verify the effects of proposed fixes.
>>>
>>>     -melanie (irc: melwitt)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>>
>>>
>>>     OpenStack Development Mailing List (not for usage questions)
>>>     Unsubscribe:
>>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> __________________________________________________________________________
>>>
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
> I seem to remember jogo teasing out quota bugs by randomly restarting
> the nova-compute service, I'm not sure how easy that is to recreate in a
> functional test setup though. We do have functional tests in tree that
> run a nova-compute service though with a backing database (sqlite), so
> it seems we could kick off some API request, restart the compute service
> immediately, and then check quotas once the compute service is back up.
> It's using the fake virt driver though so everything is going to be very
> fast and that might make finding races difficult.
>

This is the bug that had Joe's recreate scenario and notes:

https://bugs.launchpad.net/nova/+bug/1284424

-- 

Thanks,

Matt Riedemann