Open Stack

On 16 June 2015 at 18:49, Carl Baldwin <carl at ecbaldwin.net> wrote:

> On Thu, Jun 11, 2015 at 2:45 PM, Salvatore Orlando <sorlando at nicira.com>
> wrote:
> > I have been then following a different approach. And a set of patches,
> > including a devref one [2], if up for review [3]. This hardly completes
> the
> > job: more work is required on the testing side, both as unit and
> functional
> > tests.
> >
> > As for the spec, since I honestly would like to spare myself the hassle
> of
> > rewriting it, I would kindly ask our glorious drivers team if they're ok
> > with me submitting a spec in the shorter format approved for Liberty
> without
> > going through the RFE process, as the spec is however in the Kilo
> backlog.
>
> It took me a second read through to realize that you're talking to me
> among the drivers team.  Personally, I'm okay with this and our
> currently documented policy seems to allow for this until Liberty-1.
>

Great!


>
> I just hope that this isn't an indication that we're requiring too
> much in this new RFE process and scaring potential filers away.  I'm
> trying to learn how to write good RFEs, so let me give it a shot:
>
>   Summary:  "Need robust quota enforcement in Neutron."
>
>   Further Information:  "Neutron can allow exceeding the quota in
> certain cases.  Some investigation revealed that quotas in Neutron are
> subject to a race where parallel requests can each check quota and
> find there is just enough left to fulfill its individual request.
> Each request proceeds to fulfillment with no more regard to the quota.
> When all of the requests are eventually fulfilled, we find that they
> have exceeded the quota."
>
> Given my current knowledge of the RFE process, that is what I would
> file as a bug in launchpad and tag it with 'rfe.'
>

The RFE process is fine and relatively simple. I was just luring somebody
into giving me the exact text to put in it!
Jokes apart, I was suggesting this because since it was a "backlog" spec,
it was already assumed that it was something we wanted to have for Neutron
and thus skip the RFE approval step.


> > For testing I wonder what strategy do you advice for implementing
> functional
> > tests. I could do some black-box testing and verifying quota limits are
> > correctly enforced. However, I would also like to go a bit white-box and
> > also verify that reservation entries are created and removed as
> appropriate
> > when a reservation is committed or cancelled.
> > Finally it would be awesome if I was able to run in the gate functional
> > tests on multi-worker servers, and inject delays or faults to verify the
> > systems behaves correctly when it comes to quota enforcement.
>
> Full black box testing would be impossible to achieve without multiple
> workers, right?  We've proposed adding multiple worker processes to
> the gate a couple of times if I recall including a recent one to .
>

Yeah but Neutron was not as stable with multiple workers, and we had to
revert it (I think I did the revert)


> Fixing the failures has not yet been seen as a priority.
>

I wonder if this is because developers are too busy bikeshedding or chasing
unicorns,  or because the issues we saw are mostly due to the way we run
tests in the gate and are not found by operators in real deployments
(another option if that operators are too afraid of neutron's
unpredictability and they do not even try turning on multiple workers)


> I agree that some whitebox testing should be added.  It may sound a
> bit double-entry to some but I don't mind, especially given the
> challenges around block box testing.  Maybe Assaf can chime in here
> and set us straight.
>

I want white-box testing. I think it's important. Unit tests to an extent
do this, but they don't test the whole functionality. On the other hand
black-bot testing tests the functionality, but it does not tell you whether
the system is actually behaving as you expect. If it's not, it means you
have a fault. And that fault will eventually emerge as a failure. So we
need this kind of testing. However, I need hooks in Neutron in order to
achieve this. Like a sqlalchemy event listener that informs me of completed
transactions, for instance. Or hooks to perform fault injection - like
adding a delay, or altering the return value of a function. It would be
good for me to know whether this is in the testing roadmap for Liberty.


>
> > Do these kinds of test even make sense? And are they feasible at all? I
> > doubt we have any framework for injecting anything in neutron code under
> > test.
>
> Dunno.


> > Finally, please note I am using DB-level locks rather than non-locking
> > algorithms for making reservations. I can move to a non-locking
> algorithm,
> > Jay proposed one for nova for Kilo, and I can just implement that one,
> but
> > first I would like to be convinced with a decent proof (or sort of) that
> the
> > extra cost deriving from collision among workers is overshadowed by the
> cost
> > for having to handle a write-set certification failure and retry the
> > operation.
>
> Do you have a reference describing the algorithm Jay proposed?
>

Sure: https://review.openstack.org/#/c/135296/

>
> > Please advice.
> >
> > Regards,
> > Salvatore
> >
> > [1]
> >
> http://specs.openstack.org/openstack/neutron-specs/specs/kilo-backlog/better-quotas.html
> > [2] https://review.openstack.org/#/c/190798/
> > [3]
> >
> https://review.openstack.org/#/q/project:openstack/neutron+branch:master+topic:bp/better-quotas,n,z
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150616/eee17e12/attachment.html>