Open Stack

On Tue, Apr 23, 2013 at 4:46 PM, Doug Hellmann
<doug.hellmann at dreamhost.com>wrote:

>
>
>
> On Tue, Apr 23, 2013 at 1:09 PM, John Hopper <john.hopper at jpserver.net>wrote:
>
>> Hey Doug, we've been using falcon for a while now so I'll do my best to
>> answer your questions.
>>
>> On 04/23/2013 10:47 AM, Doug Hellmann wrote:
>> > On Tue, Apr 23, 2013 at 7:11 AM, Jay Pipes <jaypipes at gmail.com> wrote:
>> >
>> >> This project (Falcon) is only about 4 months old. While it looks to be
>> >> good solid code, it would be great if OpenStack projects could settle
>> on
>> >> a WSGI framework and everyone work towards the same end goal. By my
>> >> count, to date OpenStack projects have used Bottle, Flask, Pecan, WSME,
>> >> and custom-built frameworks.
>> >>
>> >
>> > FTR, WSME is just being used for (de)serialization, so it could
>> > theoretically be used with any of those other frameworks that handle the
>> > request dispatching.
>> >
>> >
>> >>
>> >> Why don't we get this into Oslo and at least make an attempt at driving
>> >> the car in the same direction? If the performance benefits of Falcon
>> are
>> >> as advertised, I would think the OpenStack dev community would be eager
>> >> to create a simple Oslo wrapper around the Falcon library and start to
>> >> standardize on a single framework for WSGI handling.
>> >>
>> >
>> > Those benchmarks look a little shallow. How does performance compare
>> when
>> > dispatching to the number of routes in the example app? Does searching a
>> > long list of regexes scale in the same way as object-dispatch models?
>>
>> Good question! I did a little bit of testing and it turns out that this
>> is indeed a problem with the current routing dispatch mechanism. Adding
>> 100k routes did degrade performance considerably at 82 req/sec. This
>> means that there's a scaling issue in the selection and dispatch
>> algorithm, however, I don't see this being difficult to rectify. Also,
>> being more reasonable with the number of routes added at 1K yielded
>> acceptable performance results of 5700 req/sec. For reference, my GET
>> request baseline pushed almost 17k req/sec.
>>
>> > What about development effort? How easy is it to add a new endpoint, or
>> set
>> > of endpoints? It looks like Falcon is using regex based dispatching,
>> > similar to Flask. The problem we found with that in ceilometer is you
>> can
>> > have "dead spots" in  your URL structure, where a URL is implied by a
>> path
>> > but doesn't actually have a handler registered. For example, the
>> benchmark
>> > app in the Falcon repo handles /hello/{account_id}/test, but does not
>> say
>> > what to do for /hello or /hello/{account_id}. That mistake would be more
>> > obvious in an object-dispatch system.
>>
>> These cases simply result in 404 returns - so yes, you can have dead
>> spots but they're well handled. I would argue it's up to the API writer
>> to make sure their resource structure is well written and correct.
>>
>
> The other frameworks also return 404. The difference is it's more obvious
> that there may be a spot in the URL namespace that doesn't point to
> something while the handlers are being developed, so a reviewer can see it
> if the patch author doesn't.
>
>
>>
>> > The other aspect to consider is whether the framework includes the
>> features
>> > we need. Does it do content-type handling? Does it handle unicode in
>> URLs
>> > correctly? What about alternate message body encodings? How well does it
>> > handle mal-formed requests? Can it handle streaming serialized data in a
>> > response?
>>
>> There are a lot of questions in there so I'll try to answer them in
>> series.
>>
>> Content type handling is left to the API writer but it would be trivial
>> to add a filter (see:
>> https://github.com/racker/falcon/blob/master/tests/test_example.py#L37-L42
>> )
>> that's smart enough to introspect this header and provide the
>> functionality desired.
>>
>> Unicode in the URL doesn't seem to route. I wrote a test containing しご
>> と in the URI template and was unable to address it via CURL.
>>
>> Alternate message body encodings appear to be left up to the API writer
>> as well but is not difficult to manage. This is probably best
>> implemented in a manner similar to the way I described handing content
>> types above.
>>
>> Malformed requests seem to be handled well in some cases but have me
>> worried in other cases. It looks like falcon formats the response with
>> what it extracted from the request sent in from the client. For example,
>> sending in a request with an incorrect HTTP version resulted in the same
>> incorrect version specified in the response. Obviously I would have to
>> spend more time with the other obvious cases and maybe design a test
>> suite for validation purposes.
>>
>
>> Streaming is supported for responses. If you're serializing data you
>> will have to manage that concern yourself by presenting the
>> serialization as a stream object. See
>> https://github.com/racker/falcon/blob/master/falcon/response.py#L24-L36
>>
>>
> So it sounds like Falcon is relatively fast at dispatching requests, but
> doesn't do nearly as much of the work for the developer as the other
> frameworks listed in the benchmarks.
>

Ryan Petrello, one of our devs, spent a couple of hours Tuesday working on
Pecan performance. He was able to find at least 30% improvement using the
Falcon benchmarks, just with low-hanging-fruit type changes. That code is
now in the latest Pecan release (0.2.3).

Doug


>
> Developer effort needs to be considered, too, IMHO.
>
>
>>  > Correctly handling all of the WSGI spec while remaining interoperable
>> with
>> > other WSGI tools is fairly complex. WebOb has a *lot* of existing use
>> and
>> > testing. Are the request objects in Falcon as robust as WebOb?
>>
>> If there's a test suite WebOb uses to validate how well it handled
>> certain use cases valid or otherwise, I'd be happy to try and run
>> similar tests against falcon. Granted, WebOb may have much of this
>> already done in unit test, making comparing the two frameworks
>> difficult. Put simply, given the maturity of WebOb in terms of projects
>> using it, I would imagine that it is certainly more robust in that its
>> had most of the edge cases vetted out by now.
>>
>
> Unfortunately, I'm not aware of a compatibility test suite that can be run
> like that. OTOH, there's a huge community around WebOb and they have all
> implicitly agreed to contribute back solutions to issues that come up. I
> see no benefit to OpenStack for us to go off in a different direction and
> build something completely different. If WebOb performance needs to be
> improved, we should start by trying to contribute upstream.
>
> Doug
>
>
>>
>> > The fact that we have several production-ready options for a framework,
>> > Pecan or otherwise, make me wonder why we are spending time building
>> > another one.
>> >
>> > Doug
>> >
>> >
>> >>
>> >> Best,
>> >> -jay
>> >>
>> >> On 04/23/2013 07:02 AM, Sandy Walsh wrote:
>> >>> haha ... classic case of left hand not knowing what the right hand is
>> >> doing.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> -S
>> >>> ________________________________________
>> >>> From: Thierry Carrez [thierry at openstack.org]
>> >>> Sent: Tuesday, April 23, 2013 6:20 AM
>> >>> To: openstack-dev at lists.openstack.org
>> >>> Subject: Re: [openstack-dev] Pecan vs Falcon?
>> >>>
>> >>> Sandy Walsh wrote:
>> >>>> Anyone had any experience with this? Specs are impressive.
>> >>>>
>> >>>> http://falconframework.org/
>> >>>
>> >>> Given that it is developed by Kurt Griffiths @ Rackspace, and used in
>> >>> Marconi, I'm pretty sure he has a strong opinion on it :)
>> >>>
>> >>> --
>> >>> Thierry Carrez (ttx)
>> >>> Release Manager, OpenStack
>> >>>
>> >>> _______________________________________________
>> >>> OpenStack-dev mailing list
>> >>> OpenStack-dev at lists.openstack.org
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>> _______________________________________________
>> >>> OpenStack-dev mailing list
>> >>> OpenStack-dev at lists.openstack.org
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >
>> >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130425/1aea1fb1/attachment.html>