Open Stack

On 06/08/15 10:08, Dan Smith wrote:
>> This is, I believe, sufficient to solve our entire problem.
>> Specifically, we have no need for an indirection API that rebroadcasts
>> messages that are too new (since that can't happen with pinning) and no
>> need for Versioned Objects in the RPC layer. (Versioned objects for the
>> DB are still critical, and we are very much better off for all the hard
>> work that Michal and others have put into them. Thanks!)
>
> So all your calls have simple types for all the arguments? Meaning,
> everything looks like this:
>
>    do_thing(uuid, 'foo', 'bar', 123)

Mostly.

> and not:
>
>    do_thing(uuid, params, data, dict_of_stuff)
>
> ?

We do have that, but dict_of_stuff is always just data that was provided 
to us by the user, verbatim. e.g. it's the contents of a template or an 
environment file. We can't control what the user sends us, so pretending 
to 'version' it is meaningless. We just pass it on without modification, 
and the engine either handles it or raises an exception so we can 
provide a 40x error to the user.

This isn't actually the interesting part though, because you're still 
thinking of it backwards - in Heat (unlike Nova) the API has no access 
to the DB, so it's not like dict_of_stuff could contain any internal 
data structures because there _are_ no internal data structures.

The interesting part is the *response* containing complex types. 
However, the same argument applies: the response just contains data that 
we are going to pass straight back to the user verbatim (at least in the 
native API), and comprises a mix of simple types and echoing data we 
originally received from the user.

> If you have the latter, then just doing RPC versioning is a mirage. Nova
> has had basic RPC versioning forever, but we didn't get actual upgrade
> ability until we tightened the screws on what we're actually sending
> over the wire. Just versioning the signatures of the calls doesn't help
> you if you're sending complex data structures (such as our Instance)
> over the wire.
>
> If you think that the object facade is necessary for insulating you from
> DB changes, I feel pretty confident that you need it for the RPC side
> for the same reason.

This assumes Nova's architecture.

> Unless you're going to unpack everything from the
> object into primitive call arguments and ensure that nobody ever changes
> one.

This is effectively what we do, although as noted above it's actually 
the response and not the arguments that we're talking about.

> If you pull things out of the DB and send them over the wire, then
> the DB schema affects your RPC API.

As I've been trying to explain, apparently unsuccessfully, we never ever 
ever pull things out of the DB and send them over the wire. Ever. Never 
have. Never will.

Here's an example of what we actually do:

http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/api.py?h=stable%2Fkilo#n158

This is how we show a resource. The function takes a Resource object, 
which in turn contains a VO with the DB representation of the resource. 
We extract various attributes and perform various calculations with the 
methods of the Resource object (all of which rely to some extent on data 
obtained from the DB). Each bit of data becomes an entry in a dict - 
this is actually the return value, but you could think of it as 
equivalent to each item in the dict as being an argument to call if the 
RPC were initiated from the opposite direction. The values are, for the 
most part, simple types, and the few exceptions are either very basic, 
well-defined and unchanging or they're just echoing data provided 
originally by the user.

The keys to the dict (~argument names) are well-defined in the 
heat.rpc.api module. We never remove a key, because that would break 
userspace. We never change the format of an item, because that would 
break userspace. Sometimes we add a key, but we always implement 
heat-api in such a way that it doesn't care whether the new key is 
present or not (i.e. it passes the data directly to the user without 
looking, or uses response.get(rpc_api.NEW_KEY, default) if it really 
needs to introspect it).

>> The nature of Heat's RPC API is that it is effectively user-facing - the
>> heat-api process is essentially a thin proxy between ReST and RPC. We
>> already have a translation layer between the internal representation(s)
>> of objects and the user-facing representation, in the form of
>> heat.engine.api, and the RPC API is firmly on the user-facing side. The
>> requirements for the content of these messages are actually much
>> stricter than anything we need for RPC API stability, since they need to
>> remain compatible not just with heat-api but with heatclient - and we
>> have *zero* control over when that gets upgraded. Despite that, we've
>> managed quite nicely for ~3 years without breaking changes afaik.
>
> I'm not sure how you evolve the internals without affecting the REST
> side if you don't have a translation layer. If you do, then the RPC API
> changes independently from the REST API.

We do have a translation layer (it's that whole file I linked above), 
but it's inside the engine (i.e. stuff gets translated *before* being 
sent over RPC, not after).

> Anyway, I don't really know anything about the internals of heat, and am
> completely willing to believe that it's fundamentally different in some
> way that makes it immune to the problems something like Nova has trying
> to make this work. I'm not sure I'm convinced of that so far, but that's
> fine :)

Here's the Heat architecture:

           ReST                RPC
   User --------> heat-api ---------> heat-engine ---------> DB

   [-----------------------------------][-------*-------------]
             User-facing data                Internal data

* = boundary where VO are needed

versus Nova (warning: handwaving - I don't really know anything about 
the internals of Nova):

                       DB
                       ^
           ReST        |       RPC
   User --------> nova-api ----------> nova-conductor, &c.
   [---------------][--*---*---------*-------------------]
      User-facing         Internal data structures

The difference is that Nova has internal data structures (specifically 
ones based on DB tables, although that's not necessarily relevant) that 
span multiple processes with an RPC bridge between them. And Heat does 
not. That isn't what we use RPC for.

It isn't that backwards/upwards compat over RPC isn't important, it's 
that we already have a much, much, much stronger contract between 
heat-engine and the *user* than we would ever need to enforce between 
heat-engine and heat-api, and therefore the issue just never comes up.

cheers,
Zane.