Open Stack

On 03/03/2014 06:32 PM, Russell Bryant wrote:
> There has been quite a bit of discussion about the future of the v3 API
> recently.  There has been growing support for the idea that we should
> change course and focus on evolving the existing v2 API instead of
> putting out a new major revision.  This message is a more complete
> presentation of that proposal that concludes that we can do what we
> really need to do with only the v2 API.
> 
> Keeping only the v2 API requires some confidence that we can stick with
> it for years to come.  We don't want to be revisiting this any time
> soon.  This message addresses a bunch of different questions about how
> things would work if we only had v2.
> 
> 1) What about tasks?
> 
> In some cases, the proposed integration of tasks is backwards
> compatible.  A task ID will be added to a header.  The biggest point of
> debate was if and how we would change the response for creating a
> server.  For tasks in v2, we would not change the response by default.
> The task ID would just be in a header.  However, if and when the client
> starts exposing version support information, we can provide an
> alternative/preferred response based on tasks.
> 
> For example:
> 
>    Accept: application/json;type=task
> 

I feel that ability to expose tasks is the single most important thing
we need to do from the API semantics standpoint, unless we redesign the
API from scratch (see below). Just looking at how awkward and edge-case
ridden the interaction between components that use each others APIs in
OpenStack is, should be enough to convince anyone that this needs fixing.

I am not sure if "tasks" will solve this, but the fact that we have
tried to solve this in several different ways up until now, and the
effort was being driven by large deployers, mostly tells me that this is
an issue people need solved.

>From that point of view - if we can do this with V2 we absolutely should.

> 2) Versioning extensions
> 
> One of the points being addressed in the v3 API was the ability to
> version extensions.  In v2, we have historically required new API
> extensions, even for changes that are backwards compatible.  We propose
> the following:
> 
>  - Add a version number to v2 API extensions
>  - Allow backwards compatible changes to these API extensions,
> accompanied by a version number increase
>  - Add the option to advertise an extension as deprecated, which can be
> used for all those extensions created only to advertise the availability
> of new input parameters
> 
> 3) Core versioning
> 
> Another pain point in API maintenance has been having to create API
> extensions for every small addition to the core API.  We propose that a
> version number be exposed for the core API that exposes the revision of
> the core API in use.  With that in place, backwards compatible changes
> such as adding a new property to a resource would be allowed when
> accompanied by a version number increase.
> 
> With versioning of the core and API extensions, we will be able to cut
> down significantly on the number of changes that require an API
> extension without sacrificing the ability of a client to discover
> whether the addition is present or not.
> 

The whole extensions vs. core discussion has been confusing me since the
beginning, and I can't say it has changed much.

After thinking about this for some time I've decided :) that I think
Nova needs 2 APIs. Amazon EC2 was always meant to be exposed as a web
service to it's users and having a REST API that exposes "resources"
without actually going into details about what is happening is fine, and
it's fine for people using Nova in a similar manner. It is clear from
this that I think the Nova API borrows a lot from EC2.

But I think nova would benefit from having a lower level API, just as a
well designed software library would, that let's people build services
on top of it, that might provide different stability guarantees, as it's
customers would not be cloud applications, but rather other cloud
infrastructure. I think that having things tired in this manner would
answer a lot of questions about what is and isn't "core" and what we
mean by "extensions".

If I were to attempt a new API for Nova - I would start from the above.

> 4) API Proxying
> 
> We don't see proxying APIs as a problem.  It is the cost we pay for
> choosing to split apart projects after they are released.  We don't
> think it's fair to break users just because we have chosen to split
> apart the backend implementation.
> 
> Further, the APIs that are proxied are frozen while those in the other
> projects are evolving.  We believe that as more features are available
> only via the native APIs in Cinder, Glance, and Neutron, users will
> naturally migrate over to the native APIs.
> 
> Over time, we can ensure clients are able to query the API without the
> need to proxy by adding new formats or extensions that don't return data
> that needed to be proxied.
> 

Proxying is fine, and conveniently fits in the above story about tiered
APIs :)

> 8) Input Validation
> 
> Previously, much of our input validation happened at the database driver
> layer for various reasons. This means that behavior of the API depends
> on such user-invisible things as which RDBM is in use. Thus, our input
> validation is already inconsistent across deployments. Further, the move
> to objects as the communication mechanism between the API and the
> backend means that more input validation is going on than we once had
> anyway, and this is not something we can avoid unless we freeze the
> backend anyway.
> 
> Exposing a definition like jsonschema is something that we should do
> anyway, and the process of doing so should allow us to define what was
> previously undefined. Given the variance of the behavior depending on
> the database used on the backend, getting something reasonably strict
> should be roughly equivalent (user-wise) to working against a provider
> that was using something other than MySQL.
> 

This is obviously fixing broken things that people's code has been
hacking around (I imagine). We should do this.

> 9) v3 ... if not now, when?
> 
> Not in the foreseeable future.  We don't see it happening unless some
> group of people wanted to rebuild the API from scratch in such a way
> that it doesn't look anything like the current API.  The new API would
> have to be different and compelling enough that we'd be willing to
> maintain it along side the current API for several years.
> 
> 10) Conclusion and Proposal
> 
> The v3 API effort has produced a lot of excellent work.  However, the
> majority opinion seems to be that we should avoid the cost of
> maintaining two APIs if at all possible.  We should apply what has been
> learned to the existing API where we can and focus on making v2
> something that we can continue to maintain for years to come.
> 
> We recognize and accept that it is a failure of Nova project leadership
> that we did not come to this conclusion much sooner.  We hope to have
> learned from the experience to help avoiding a situation like this
> happening again in the future.
> 

A few words on the final call about the v3 API:

As many have responded above, I also believe that going back on
decisions is very important for building software, and "guaranteeing"
things to "the management" has been a constant source of fail in the
history of software, so from that point - yes we should absolutely be OK
with doing it if it makes sense (and yes we should have probably done it
sooner if we are going to do it :)).

But as was also pointed out, if we are going to make people hate hacking
on Nova because they feel that there is a danger of their work being
dropped on (what may appear as a) whim, then we are for sure going to
fail. OpenStack in general has been very happy to celebrate success, but
we should also be as "happy" to own up to our failures.

What struck me as wrong about the v3 situation in particular was that
one of the issues that were held up against v3 was that v3 just has no
"killer" features that will make people want to switch, yet not many
people stood up and tried to see what those features could be, and what
people who are not directly hacking on the v3 API can do to help make v3
a success. Now don't get me wrong - I am all for pragmatism, but it
comes with a price, and in this case the price is coming off as an
unsupportive community. I have no way of knowing how Chris and other
folks feel, but I imagine this is how I would feel.

Thanks,

Nikola