Open Stack

On 20 February 2014 14:55, Christopher Yeoh <cbkyeoh at gmail.com> wrote:
> On Thu, 20 Feb 2014 08:22:57 -0500
> Sean Dague <sean at dague.net> wrote:
>>
>> We're also duplicating a lot of test and review energy in having 2 API
>> stacks. Even before v3 has come out of experimental it's consumed a
>> huge amount of review resource on both the Nova and Tempest sides to
>> get it to it's current state.
>>
>> So my feeling is that in order to get more energy and focus on the
>> API, we need some kind of game plan to get us to a single API
>> version, with a single data payload in L (or on the outside, M). If
>> the decision is v2 must be in both those releases (and possibly
>> beyond), then it seems like asking other hard questions.
>>
>> * why do a v3 at all? instead do we figure out a way to be able to
>> evolve v2 in a backwards compatible way.
>
> So there's lots of changes (cleanups) made between v2 and v3 which are
> really not possible to do in a backwards compatible way. One example
> is that we're a lot stricter and consistent on input validation in v3
> than v2 which is better both from a user and server point of view.
> Another is that the tasks API would be a lot uglier and really look
> "bolted on" if we tried to do so. Also doing so doesn't actually reduce
> the test load as if we're still supporting the old 'look' of the api we
> still need to test for it separately to the new 'look' even if we don't
> bump the api major version.
>
> In terms of code sharing (and we've experimented a bit with this for
> v2/v3) I think in most cases ends up actually being easier having two
> quite completely separate trees because it ends up diverging so much
> that having if statements around everywhere to handle the different
> cases is actually a higher maintenance burden (much harder to read)
> than just knowing that you might have to make changes in two quite
> separate places.

Maybe, but what about a slightly less different v3, that would enable
such an approach?

>> * if we aren't doing a v3, can we deprecate XML in v2 in Icehouse so
>> that working around all that code isn't a velocity inhibitor in the
>> cleanups required in v2? Because some of the crazy hacks that exist to
>> make XML structures work for the json in v2 is kind of special.
>
> So I don't think we can do that for similar reasons we can't just drop
> V2 after a couple of cycles. We should be encouraging people off, not
> forcing them off.

We could look to remove XML support, with the same cycle we considered
dropping v2.

>> This big bang approach to API development may just have run it's
>> course, and no longer be a useful development model. Which is good to
>> find out. Would have been nice to find out earlier... but not all
>> lessons are easy or cheap. :)
>
> So I think what the v3 gives us is much more consistent and clean
> API base to start from. It's a clean break from the past. But we have to
> be much more careful about any future API changes/enhancements than we
> traditionally have done in the past especially with any changes which
> affect the core. I think we've already significantly raised the quality
> bar in what we allow for both v2 and v3 in Icehouse compared to previous
> releases (those frustrated with trying to get API changes in will
> probably agree) but I'd like us to get even stricter about it in the
> future because getting it wrong in the API design has a MUCH higher
> long term impact than bugs in most other areas. Requiring an API spec
> upfront (and reviewing it) with a blueprint for any new API features
> should IMO be compulsory before a blueprint is approved.

I think we need to go down this (slightly painful) path.

Particularly because there is so much continuous deployment going on.

> Also micro and extension versioning is not the magic bullet which will
> get us out of trouble in the future. Especially with the core changes.
> Because even though versioning allows us to make changes, for similar
> reasons to not being able to just drop V2 after a couple of cycles
> we'll still need to keep supporting (and testing) the old behaviour for
> a significant period of time (we have often quietly ignored
> this issue in the past).

I thought the versions were all backwards compatible (for some
definition of that).

> Ultimately the only way to free ourselves from the maintenance of two
> API versions (and I'll claim this is rather misleading as it actually
> has more dimensions to it than this) is to convince users to move from
> the V2 API to the "new one". And it doesn't make much difference
> whether we call it V3 or V2.1 we still have very similar maintenance
> burdens if we want to make the sorts of API changes that we have done
> for V3.

I did agree with you before now, but I maybe we have "too many" people
using v2 already. I have been wondering about a half way house...

So, changes in v3 I would love to keep (highest priority first, as I see it):
* versioning extensions
* task API
* internal wiring fix ups (policy, everything is an extension, split
up extensions)
* return code fix ups
* better input validation
* url (consistency) changes
* document consistency changes

Assuming we are happy to retro-fix versions, the question is how much
do we allow to change between those versions.

I am good with stuff that would not break a "correct" old API client:
* allow an API to warn users it is "deprecated"
* extra attributes may be added in the return document
* return codes for error cases might get updated
* the x in 2xx might change for success cases
* stricter validation of inputs might occur

Ideally, we only do this once, like at the same time we add in the
versioning of extensions.

Having two urls for a single extension seems like quite a low cost
"fix up", that we can keep in tree. Unit tests should be able to cover
that, or at least only a small amount of functional tests.

The clean ups to the format of documents returned, this one is harder:
* let the client (somehow) choose the new version, maybe a new URL, or
maybe an Accepts header
* keep the old version via some "converter" that can be easily unit
tested in isolation

The general idea, is to get the fix ups we want, but make it easier to
maintain, and try to reduce the functional test version, by using the
knowledge there is so much common code, we don't need a full
duplication of tests.

As far as implementation, I think it would be to make the v3 code
support v2 and v3, in the process change v3 so thats possible, then
drop the v2 code.

John

>
> Chris
>
>>
>>       -Sean
>>
>> On 02/19/2014 12:36 PM, Russell Bryant wrote:
>> > Greetings,
>> >
>> > The v3 API effort has been going for a few release cycles now.  As
>> > we approach the Icehouse release, we are faced with the following
>> > question: "Is it time to mark v3 stable?"
>> >
>> > My opinion is that I think we need to leave v3 marked as
>> > experimental for Icehouse.
>> >
>> > There are a number of reasons for this:
>> >
>> > 1) Discussions about the v2 and v3 APIs at the in-person Nova meetup
>> > last week made me come to the realization that v2 won't be going
>> > away *any* time soon.  In some cases, users have long term API
>> > support expectations (perhaps based on experience with EC2).  In
>> > the best case, we have to get all of the SDKs updated to the new
>> > API, and then get to the point where everyone is using a new enough
>> > version of all of these SDKs to use the new API.  I don't think
>> > that's going to be quick.
>> >
>> > We really don't want to be in a situation where we're having to
>> > force any sort of migration to a new API.  The new API should be
>> > compelling enough that everyone *wants* to migrate to it.  If
>> > that's not the case, we haven't done our job.
>> >
>> > 2) There's actually quite a bit still left on the existing v3 todo
>> > list. We have some notes here:
>> >
>> > https://etherpad.openstack.org/p/NovaV3APIDoneCriteria
>> >
>> > One thing is nova-network support.  Since nova-network is still not
>> > deprecated, we certainly can't deprecate the v2 API without
>> > nova-network support in v3.  We removed it from v3 assuming
>> > nova-network would be deprecated in time.
>> >
>> > Another issue is that we discussed the tasks API as the big new API
>> > feature we would include in v3.  Unfortunately, it's not going to be
>> > complete for Icehouse.  It's possible we may have some initial parts
>> > merged, but it's much smaller scope than what we originally
>> > envisioned. Without this, I honestly worry that there's not quite
>> > enough compelling functionality yet to encourage a lot of people to
>> > migrate.
>> >
>> > 3) v3 has taken a lot more time and a lot more effort than anyone
>> > thought.  This makes it even more important that we're not going to
>> > need a v4 any time soon.  Due to various things still not quite
>> > wrapped up, I'm just not confident enough that what we have is
>> > something we all feel is Nova's API of the future.
>> >
>> >
>> > Let's all take some time to reflect on what has happened with v3 so
>> > far and what it means for how we should move forward.  We can
>> > regroup for Juno.
>> >
>> > Finally, I would like to thank everyone who has helped with the
>> > effort so far.  Many hours have been put in to code and reviews for
>> > this.  I would like to specifically thank Christopher Yeoh for his
>> > work here. Chris has done an *enormous* amount of work on this and
>> > deserves credit for it.  He has taken on a task much bigger than
>> > anyone anticipated. Thanks, Chris!
>> >
>>
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev