Open Stack

On Tue, Apr 23, 2013 at 1:09 PM, John Hopper <john.hopper at jpserver.net>wrote:

> Hey Doug, we've been using falcon for a while now so I'll do my best to
> answer your questions.
>
> On 04/23/2013 10:47 AM, Doug Hellmann wrote:
> > On Tue, Apr 23, 2013 at 7:11 AM, Jay Pipes <jaypipes at gmail.com> wrote:
> >
> >> This project (Falcon) is only about 4 months old. While it looks to be
> >> good solid code, it would be great if OpenStack projects could settle on
> >> a WSGI framework and everyone work towards the same end goal. By my
> >> count, to date OpenStack projects have used Bottle, Flask, Pecan, WSME,
> >> and custom-built frameworks.
> >>
> >
> > FTR, WSME is just being used for (de)serialization, so it could
> > theoretically be used with any of those other frameworks that handle the
> > request dispatching.
> >
> >
> >>
> >> Why don't we get this into Oslo and at least make an attempt at driving
> >> the car in the same direction? If the performance benefits of Falcon are
> >> as advertised, I would think the OpenStack dev community would be eager
> >> to create a simple Oslo wrapper around the Falcon library and start to
> >> standardize on a single framework for WSGI handling.
> >>
> >
> > Those benchmarks look a little shallow. How does performance compare when
> > dispatching to the number of routes in the example app? Does searching a
> > long list of regexes scale in the same way as object-dispatch models?
>
> Good question! I did a little bit of testing and it turns out that this
> is indeed a problem with the current routing dispatch mechanism. Adding
> 100k routes did degrade performance considerably at 82 req/sec. This
> means that there's a scaling issue in the selection and dispatch
> algorithm, however, I don't see this being difficult to rectify. Also,
> being more reasonable with the number of routes added at 1K yielded
> acceptable performance results of 5700 req/sec. For reference, my GET
> request baseline pushed almost 17k req/sec.
>
> > What about development effort? How easy is it to add a new endpoint, or
> set
> > of endpoints? It looks like Falcon is using regex based dispatching,
> > similar to Flask. The problem we found with that in ceilometer is you can
> > have "dead spots" in  your URL structure, where a URL is implied by a
> path
> > but doesn't actually have a handler registered. For example, the
> benchmark
> > app in the Falcon repo handles /hello/{account_id}/test, but does not say
> > what to do for /hello or /hello/{account_id}. That mistake would be more
> > obvious in an object-dispatch system.
>
> These cases simply result in 404 returns - so yes, you can have dead
> spots but they're well handled. I would argue it's up to the API writer
> to make sure their resource structure is well written and correct.
>

The other frameworks also return 404. The difference is it's more obvious
that there may be a spot in the URL namespace that doesn't point to
something while the handlers are being developed, so a reviewer can see it
if the patch author doesn't.


>
> > The other aspect to consider is whether the framework includes the
> features
> > we need. Does it do content-type handling? Does it handle unicode in URLs
> > correctly? What about alternate message body encodings? How well does it
> > handle mal-formed requests? Can it handle streaming serialized data in a
> > response?
>
> There are a lot of questions in there so I'll try to answer them in series.
>
> Content type handling is left to the API writer but it would be trivial
> to add a filter (see:
> https://github.com/racker/falcon/blob/master/tests/test_example.py#L37-L42
> )
> that's smart enough to introspect this header and provide the
> functionality desired.
>
> Unicode in the URL doesn't seem to route. I wrote a test containing しご
> と in the URI template and was unable to address it via CURL.
>
> Alternate message body encodings appear to be left up to the API writer
> as well but is not difficult to manage. This is probably best
> implemented in a manner similar to the way I described handing content
> types above.
>
> Malformed requests seem to be handled well in some cases but have me
> worried in other cases. It looks like falcon formats the response with
> what it extracted from the request sent in from the client. For example,
> sending in a request with an incorrect HTTP version resulted in the same
> incorrect version specified in the response. Obviously I would have to
> spend more time with the other obvious cases and maybe design a test
> suite for validation purposes.
>

> Streaming is supported for responses. If you're serializing data you
> will have to manage that concern yourself by presenting the
> serialization as a stream object. See
> https://github.com/racker/falcon/blob/master/falcon/response.py#L24-L36
>
>
So it sounds like Falcon is relatively fast at dispatching requests, but
doesn't do nearly as much of the work for the developer as the other
frameworks listed in the benchmarks.

Developer effort needs to be considered, too, IMHO.


> > Correctly handling all of the WSGI spec while remaining interoperable
> with
> > other WSGI tools is fairly complex. WebOb has a *lot* of existing use and
> > testing. Are the request objects in Falcon as robust as WebOb?
>
> If there's a test suite WebOb uses to validate how well it handled
> certain use cases valid or otherwise, I'd be happy to try and run
> similar tests against falcon. Granted, WebOb may have much of this
> already done in unit test, making comparing the two frameworks
> difficult. Put simply, given the maturity of WebOb in terms of projects
> using it, I would imagine that it is certainly more robust in that its
> had most of the edge cases vetted out by now.
>

Unfortunately, I'm not aware of a compatibility test suite that can be run
like that. OTOH, there's a huge community around WebOb and they have all
implicitly agreed to contribute back solutions to issues that come up. I
see no benefit to OpenStack for us to go off in a different direction and
build something completely different. If WebOb performance needs to be
improved, we should start by trying to contribute upstream.

Doug


>
> > The fact that we have several production-ready options for a framework,
> > Pecan or otherwise, make me wonder why we are spending time building
> > another one.
> >
> > Doug
> >
> >
> >>
> >> Best,
> >> -jay
> >>
> >> On 04/23/2013 07:02 AM, Sandy Walsh wrote:
> >>> haha ... classic case of left hand not knowing what the right hand is
> >> doing.
> >>>
> >>> Thanks!
> >>>
> >>> -S
> >>> ________________________________________
> >>> From: Thierry Carrez [thierry at openstack.org]
> >>> Sent: Tuesday, April 23, 2013 6:20 AM
> >>> To: openstack-dev at lists.openstack.org
> >>> Subject: Re: [openstack-dev] Pecan vs Falcon?
> >>>
> >>> Sandy Walsh wrote:
> >>>> Anyone had any experience with this? Specs are impressive.
> >>>>
> >>>> http://falconframework.org/
> >>>
> >>> Given that it is developed by Kurt Griffiths @ Rackspace, and used in
> >>> Marconi, I'm pretty sure he has a strong opinion on it :)
> >>>
> >>> --
> >>> Thierry Carrez (ttx)
> >>> Release Manager, OpenStack
> >>>
> >>> _______________________________________________
> >>> OpenStack-dev mailing list
> >>> OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>> _______________________________________________
> >>> OpenStack-dev mailing list
> >>> OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130423/3384eb6e/attachment.html>