[openstack-dev] [Solum] Choice of API infrastructure

Doug Hellmann doug.hellmann at dreamhost.com
Tue Nov 5 03:20:22 UTC 2013

On Tue, Nov 5, 2013 at 10:37 AM, Jay Pipes <jaypipes at gmail.com> wrote:

> Doug, I respect you very much as an engineer and as a community member, so
> I want to preface this with a very important dislaimer: don't take these
> opinions as anything more than what they are...a discussion of differing
> viewpoints.

Ditto. :-)

> Comments inline.
> On 11/04/2013 06:58 PM, Doug Hellmann wrote:
>> On Sun, Nov 3, 2013 at 10:54 PM, Jay Pipes <jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>> wrote:
>>     On 11/02/2013 11:26 PM, Russell Bryant wrote:
>>         On 11/02/2013 11:54 AM, Adrian Otto wrote:
>>             Noorul,
>>             I agree that key decisions should be tracked in blueprints.
>>             This is the
>>             one for this decision which was made in our 2013-10-18
>>             public meeting.
>>             Jay's submission is consistent with the direction indicated
>>             by the team.
>>             https://blueprints.launchpad.__net/solum/+spec/rest-api-base
>>             <https://blueprints.launchpad.net/solum/+spec/rest-api-base>
>>             Transcript log:
>>             http://irclogs.solum.io/2013/__solum.2013-10-08-16.01.log.__
>> html
>>             <http://irclogs.solum.io/2013/solum.2013-10-08-16.01.log.html
>> >
>>             <http://irclogs.solum.io/2013/__solum.2013-10-08-16.01.log._
>> _html
>>             <http://irclogs.solum.io/2013/solum.2013-10-08-16.01.log.html
>> >>
>>         Heh, not much discussion there.  :-)
>>     Agreed. I actually didn't know anything about the discussion -- I
>>     wasn't at the meeting. I just figured I would throw some example
>>     code up to Gerrit that shows how Falcon can be used for the API
>>     plumbing. Like I mentioned in a previous email, I believe it's much
>>     easier to discuss things when there is sample code...
>>         Here's my take ... Pecan+WSME has been pushed as the thing to
>>         standardize on across most OpenStack APIs.  Ceilometer (and maybe
>>         others?) are already using it.  Others, such as Nova, are
>>         planning to
>>         use it this cycle. [1][2]
>>     I've used both actually, and I've come to prefer Falcon because of
>>     its simplicity and specifically because of the following things:
>>     * It's lack of integration with WSME, which I don't care for. I
>>     don't care for WSME because I believe it tries to put the model at
>>     the view layer, instead of where it belongs, at the model layer.
>> The "models" defined in WSME are completely different from the database
>> models, and should not be used outside of the API code. Think of them as
>> declaring the models for the consumer of the API, rather than the
>> implementer of the API.
> I don't really see the point of the distinction. At the end of the day,
> the consumer of the API is using the API to manipulate and retrieve data.
> That data is really the model, with some syntactic sugar like Atom links,
> etc. Even in a control API -- as opposed to a data API like Glance,
> Ceilometer or Swift -- the benefits of a heavy API layer like Pecan and
> WSME are pretty small, IMO.

That approach binds your API tightly to the database representation, which
we were trying to avoid.

> I would much rather see the Ceilometer models [1] actually be models that
> can validate the data that is used to construct the model object instead of
> having duplicated WSME "models" repeated in the WSGI controller code [2].
> The reason is because if/when I decide to make a Ceilometer API that uses a
> different protocol, say AMQP, instead of HTTP, now I need to duplicate all
> of the validation code that WSME is providing on the data model layer...
> however if the validation was in the models themselves, I could easily
> create an API on a different protocol using just the models for validation.

We do that in some cases. However, there is also a difference in some cases
between the validation at the API layer (a value must be a number, or a
UUID, etc.) and the validation in the database layer (a number must fall
within a range or a UUID must refer to an existing object). So there is a
place for both, and the validation done in the WSME classes is not meant to
be the only validation performed.

>  The benefits of declaring WSME classes include automatic serialization
>> in JSON and XML, generating WADL files to be included in the API docs
>> (work is already happening to make this available for everyone), and
>> consistent input and output types for API endpoints (making it easier
>> for consumers of the API to use it and for implementers to validate
>> inputs and assume consistent defaults).
> I can't stand XML. I believe it should be retired to the dustbin of coding
> history, like Basic.

You've made that clear in the past. :-) I agree, for what it's worth. Some
of our users do seem to want it, and with WSME *you don't have to care*.

> That said, consumers of a RESTful API don't care how the API is
> implemented. They care that it's documented and consistent, and if WSME
> makes API documentation easier, then that's A Good Thing, agreed.

> It's true that WSME includes some stuff to make validating inputs
> "easier", but it does so, IMHO, at the expense of readability because
> everything is decorated and hidden away somewhere other than the models
> themselves. See note above...

I'm not sure what that means. Hidden where? The validation is either
described in the attribute specifier for the model, or in the model's
class, or in the controller (depending on the scope of the rule being

>  And, to be clear, Pecan and WSME are integrated by Pecan can definitely
>> be used without WSME. I included WSME in the proposal to replace the
>> home-grown WSGI framework because I thought it added significant
>> benefits, but it is not going to be appropriate for all situations
>> (streaming large image files is one example).
>>     * It doesn't need a configuration file, specifically a configuration
>>     file that is a Python file as opposed to an .ini file.
>> Pecan does not require a configuration file. It can use one, but we set
>> up the WSGI app factory in ceilometer to not use one and I expect the
>> other projects to work the same way.
> OK, that's good to know.
> Here's a third reason I don't care for Pecan/WSME: it uses Webob. Other
> than eventlet, I don't know of a single library that OpenStack projects
> have used over the years that we've had more issues with than Webob.

Yes, I felt the pain of updating us to the latest WebOb. The project has
evolved since those days, and the current maintainers are committed to not
breaking the API. That new attitude, combined with the long history of
addressing edge cases from misbehaving web client libraries makes m e less
reluctant to use WebOb than in the past.

>  Tuesday (today) at 2:00 there is a session in the Oslo track
>> (http://icehousedesignsummit.sched.org/event/
>> b2680d411aa7f5d432438a435ac21fee)
>> to discuss tips and pain points with Pecan & WSME.
> Unfortunately, I won't be at the summit. :(

OK, we'll be updating the etherpad and obviously we can continue this via

> >I didn't intend to
>> revisit the decision to start adopting either (we've spent an hour at
>> each of the last summits going over that, as well as many email
>> threads),
> Yeah, you know the argument that a couple dozen folks have discussed
> something at one or two of the past summits and therefore it should be
> considered settled fact is a bit annoying to me.

"as well as many email threads".

> Partly it is annoying because the summit has a vast number of sessions,
> with many tracks going simultaneously, making it impossible to get to all
> the sessions that people feel strongly about. I wish I could have been at
> several design summit discussions, especially around database issues, at
> the last couple summits, but I was in the QA track most of the summit, for
> good or bad.

Yep. There is not a single time slot this week where I don't want to be in
fewer than 3 places at one time.

> Secondly, it is the sign of strength that a contributor community consider
> -- and continually consider -- alternative libraries or implementations of
> various things. Change is good, and projects that are just starting off are
> a good place to experiment with newer libraries and see if something is
> better than what existed previously. I seem to remember that was your
> position on Pecan when the community was considering whether to standardize
> on some WSGI pipeline. Why didn't you just use what was in other projects
> instead of Pecan and WSME? Likely the same reasons that I'm saying "let's
> give Falcon a chance".

My main arguments were "do not build something new from scratch when there
are working tools" and "the existing library is too hard to use". We can
argue about whether Falcon or Pecan solves the second issue better, but
Pecan was used in production deployments far earlier than Falcon even

> I believe we discussed why Bazaar and Launchpad merge proposals were the
> code contribution platform at length for the Austin, Bexar and Cactus
> summits, with GitHub fans at every one of those discussions, and lo and
> behold, the community switched to Git and GitHub and Gerrit during the
> Diablo timeframe. So, long story short, stuff changes, opinions change, and
> that's totally fine...



> > but I do want to clear up any other misconceptions and discuss
>> issues with either tool so that feedback can be incorporated upstream.
>> Now that both Pecan and WSME are on stackforge, we have already had a
>> few patches from OpenStack developers intended to improve and adjust
>> them to meet our needs better.
> I think that is a fine thing to continue, and I don't believe this
> particular debate by the Solum project should affect that at all.
> Best,
> -jay
> [1] https://github.com/openstack/ceilometer/blob/master/
> ceilometer/storage/models.py
> [2] https://github.com/openstack/ceilometer/blob/master/
> ceilometer/api/controllers/v2.py
>  Doug
>>         I care less about the particular choice and more about
>>         consistency.  It
>>         brings a lot of value, such as making it a lot easier for
>>         developers to
>>         jump around between the OpenStack projects.  Can we first at
>>         least agree
>>         that there is value in standardizing on *something* for most
>>         OpenStack APIs?
>>     I completely understand the need for consistency. I pushed my patch
>>     as an example of how to do things the Falcon way. While I would
>>     prefer Falcon over Pecan (and certainly over Pecan+WSME), I will
>>     respect the push towards consistency if that's what is most important.
>>     That said, I also believe that the projects in Stackforge should be
>>     the "laboratories of experiment", and these projects may serve as a
>>     good playground for various implementations of things. I remind the
>>     reader that over time, the development community has standardized on
>>     various things, only to find a better implementation in an incubated
>>     project. Pecan+WSME is actually an example of that experimentation
>>     turned accepted standard.
>>     Best,
>>     -jay
>>         I understand that there may be cases where the needs for an API
>>         justify
>>         being different.  Marconi being more of a data-plane API vs
>>         control-plane means that performance concerns are much higher,
>>         for example.
>>         If we agree that consistency is good, does Solum have needs that
>>         make it
>>         different than the majority of OpenStack APIs?  IMO, it does not.
>>         Can someone lay out a case for why all OpenStack projects should
>> be
>>         using Falcon, if that's what you think Solum should use?
>>         Also, is anyone willing to put up the equivalent of Jay's review
>>         [3],
>>         but with Pecan+WSME, to help facilitate the discussion?
>>         [1]
>>         http://icehousedesignsummit.__sched.org/event/__
>> b2680d411aa7f5d432438a435ac21f__ee
>>         <http://icehousedesignsummit.sched.org/event/
>> b2680d411aa7f5d432438a435ac21fee>
>>         [2]
>>         http://icehousedesignsummit.__sched.org/event/__
>> 4a7316a4f5c6f783e362cbba2644ba__e2
>>         <http://icehousedesignsummit.sched.org/event/
>> 4a7316a4f5c6f783e362cbba2644bae2>
>>         [3] https://review.openstack.org/#__/c/55040/
>>         <https://review.openstack.org/#/c/55040/>
>>     _________________________________________________
>>     OpenStack-dev mailing list
>>     OpenStack-dev at lists.openstack.__org
>>     <mailto:OpenStack-dev at lists.openstack.org>
>>     http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev<
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131105/a133e578/attachment.html>

More information about the OpenStack-dev mailing list