Open Stack

On Tue, 25 Feb 2014 09:26:09 -0500
Sean Dague <sean at dague.net> wrote:

> On 02/25/2014 08:17 AM, Ken'ichi Ohmichi wrote:
> > 2014-02-25 19:48 GMT+09:00 Thierry Carrez <thierry at openstack.org>:
> >> Sean Dague wrote:
> >>> So, that begs a new approach. Because I think at this point even
> >>> if we did put out Nova v3, there can never be a v4. It's too
> >>> much, too big, and doesn't fit in the incremental nature of the
> >>> project. So whatever gets decided about v3, the thing that's
> >>> important to me is a sane way to be able to add backwards
> >>> compatible changes (which we actually don't have today, and I
> >>> don't think any other service in OpenStack does either), as well
> >>> a mechanism for deprecating parts of the API. With some future
> >>> decision about whether removing them makes sense.
> >>
> >> I agree with Sean. Whatever solution we pick, we need to make sure
> >> it's solid enough that it can handle further evolutions of the
> >> Nova API without repeating this dilemma tomorrow. V2 or V3, we
> >> would stick to it for the foreseeable future.
> >>
> >> Between the cleanup of the API, the drop of XML support, and
> >> including a sane mechanism for supporting further changes without
> >> major bumps of the API, we may have enough to technically justify
> >> v3 at this point. However from a user standpoint, given the
> >> surface of the API, it can't be deprecated fast -- so this ideal
> >> solution only works in a world with infinite maintenance resources.
> >>
> >> Keeping V2 forever is more like a trade-off, taking into account
> >> the available maintenance resources and the reality of Nova's API
> >> huge surface. It's less satisfying technically, especially if
> >> you're deeply aware of the API incoherent bits, and the prospect
> >> of living with some of this incoherence forever is not really
> >> appealing.
> > 
> > What is the maintenance cost for keeping both APIs?
> > I think Chris and his team have already paid most part of it, the
> > works for porting
> > the existing v2 APIs to v3 APIs is almost done.
> > So I'd like to clarify the maintenance cost we are discussing.
> > 
> > If the cost means that we should implement both API methods when
> > creating a new API, how about implementing internal proxy from v2
> > to v3 API? When creating a new API, it is enough to implement API
> > method for v3 API. and when receiving a v2 request, Nova translates
> > it to v3 API. The request styles(url, body) of v2 and v3 are
> > different and this idea makes new v2 APIs v3 style. but now v2 API
> > has already a lot of inconsistencies. so it does not seem so big
> > problem.
> > 
> > 
> > From the viewpoint of OpenStack interoperability also, I believe we
> > need a new API.
> > Many v2 API parameters are not validated. If implementing strict
> > validation for v2 API,
> > incompatibility issues happen. That is why we are implementing input
> > validation for
> > v3 API. If staying v2 API forever, we should have this kind of
> > problem forever. v2 API is fragile now. So the interoperability
> > should depend on v2 API, that seems
> > sandbox.. (I know that it is a little overstatement, but we have
> > found a lot of this kind
> > of problem already..)
> 
> So I think this remains a good question about what keeping v2 forever
> means. Because it does mean keeping the fact that we don't validate
> input at the surface and depend on database specific errors to trickle
> back up correctly. So if MySQL changes how it handles certain things,
> you'll get different errors on the surface.
> 
> I'm gong to non-sequitor for a minute, because I think it's important
> to step back some times.
> 
> What I want out of Nova API at the end of the day:
> 
> 1. a way to discover what the API is
> 
> because this massively simplifies writing clients, SDKs, tests, and
> documentation. All those pipelines are terribly manual, and have
> errors in them because of it. Like has been said before you actually
> need to read the Nova source code to figure out how to use parts of
> the API.
> 
> I think this is a great example of that -
> https://blog.heroku.com/archives/2014/1/8/json_schema_for_heroku_platform_api?utm_source=newsletter&utm_medium=email&utm_campaign=januarynewsletter&mkt_tok=3RkMMJWWfF9wsRonuKzNZKXonjHpfsX57OQtX6SxlMI%2F0ER3fOvrPUfGjI4AScJrI%2BSLDwEYGJlv6SgFQrjAMapmyLgLUhE%3D
>

So from what I understand I think the jsonschema work that Ken'ichi has
been working on for V3 goes a fair way in being able to support this
sort of thing. The jsonschema we'd be able to provide for V2 however is
a bit trickier as we'd have to leave the input validation pretty loose
(and probably in rather wierdly inconsistent ways because that's how
its implemented) in most cases.

> Extensions thus far have largely just been used as a cheat to get
> around API compatibility changes based on the theory that users could
> list extensions to figure out what the API would look like. It's a bad
> theory, and not even nova command line does this. So users will get
> errors on nova cli with clouds because features aren't enabled, and
> the user has no idea why their commands don't work. Because it's
> right there in the nova help.

So I think extensions in the past have also been used as an excuse to
get code in fast. Eg. we're in a rush to get feature "foo" in as fast as
possible. So we add it as an extension and don't look at it too
closely, because hey if someone doesn't like it, they just don't load
it. And now we discover we might just have to live with it forever.

I think tightening down on this is part of Nova maturing and accepting
a slower velocity. That being said I think its a discussion which is
kind of orthogonal to V2/V3. If we want to say that we have a fixed API
surface then we need to have the giant debate about what is and what is
out (settling on a fairly small core for V3 took a while initself). And
I suspect its a debate that needs a wider audience than we can rely
on turning up at a summit session.

> 3. a solid validation surface
> 
> We really need to be far more defensive on our API validation surface.
> Right now bad data makes it far too far down the code stack. That's
> just a recipe for security issues.

Definitely. Ken'ichi's json schema input validation is really
tightening down on what is accepted. Because the way that input
validation is separated from what the API layer needs to do its a lot
easier to review.

We've also been working on ensuring that problems (permission/data etc)
are picked up in the api layer rather than a lot further down, where
unwinding what has already been done can be more complicated and error
prone.

> = Current tensions =
> 
> Cloud providers want v2 "for a long time" (which, honestly, was news).
> I'm not sure I completely know what that means. Are we talking about
> the bug for bug surface that is today? Are we talking about not
> wanting to change endpoints?

So the problem here is what we consider a "bug" becomes a feature from
a user of the API point of view. Eg they really shouldn't be passing
some data in a request, but its ignored and doesn't cause any issues
and the request ends up doing what they expect. 

Now if we tighten up input validation or otherwise change it, then when
the cloud providers end up deploying the new version client apps are
likely to break. And for CD providers they won't really be able to warn
people in advance like say some may be able to do for a
Havana->Icehouse upgrade.

I'd love to be corrected but I suspect that most if not all cloud
providers don't have the sort of regression tests that would pick up on
these sorts of issues (if they do, please donate the tests to
tempest :-) For most v2 API features we don't do that great a job of
testing around invalid data (especially unexpected data) either in
tempest or nova unittests (mostly because its really quite hard to do
manually).

Just as an example, one thing the input validation work has picked up
along the way is that we have had api samples (copied/modified from V2
and used for V3) which are incorrect. Now these are most often written
by the people who write the extensions. So not only have doc examples
been wrong in the past, but if the author of the extension makes
mistakes about what is valid input and its not picked, what are the
chances that users of the API that just read the doco get it
exactly right all the time?

> If we are talking about exactly the v2 API as is, then I think we have
> an answer, which is push forward with v3, because they don't actually
> want us to fix issues in v2, as that might break existing users. Even
> subtle things might do that. At which point we treat the whole thing
> like grandma's china, and just don't touch it, at all.

That's one of the other factors which concerns me about the "backport to
V2" approach or try "to encapsulate V3 and V2 in the same code". Making
major modifications to the V2 code means considerable risk in
accidentally making changes which will cause breakage for users of the
V2 API. And we often don't have the backup of unittests or tempest
tests to catch it so they'll get merged.

> No matter the results or verdicts on v3, we need to figure out some
> way to make incremental progress. Microversioning, accept types, ...
> something. And it needs to not just be a surface, but also some
> internals mechanism so we can sanely handle this internally.

So microversioning is something we definitely need for V3 if we're
not going to expose the version information for core API components.
And I think from a user point of view its much easier concept to manage
than lots of different individual versions.

> Because the Nova API (including extensions) is huge. I challenge
> anyone to put the whole thing in their head at the moment and
> understand the implications between the pieces. You can't even use
> big chunks of it if you use cells, which we know of many large
> implementations out there.

Agreed the API is huge. The smaller we can reasonably make the API the
better because it is hard to grasp (and things like image/volume
proxying make its needlessly bigger). 

I think we're getting to the stage of being a lot better of reviewing
API changes on a standalone basis (eg do the calls make sense, is there
good input validation etc, self consistency) rather than just
verifying it "works". But we still have a long way to go of ensuring
that new API features fit well into how the API as a whole looks and
feels. So it doesn't look like 50 different developers have
written their own parts without looking at what everyone else is doing.

Chris