Open Stack

On 06/08/15 08:20, Grasza, Grzegorz wrote:
>
>
>> -----Original Message-----
>> From: Zane Bitter [mailto:zbitter at redhat.com]
>> Sent: Thursday, 6 August, 2015 2:57
>> To: OpenStack Development Mailing List
>> Subject: [openstack-dev] [Heat] RPC API versioning
>>
>> We've been talking about this since before summit without much consensus.
>> I think a large part of the problem is that very few people have deep
>> knowledge of both Heat and Versioned Objects. However, I think we are at a
>> point where we should be able to settle on an approach at least for the API<-
>>> engine RPC interface. I've been talking to Dan Smith about the Nova team's
>> plan for upgrades, which goes something like this:
>>
>> * Specify a max RPC API version in the config file
>> * In the RPC client lib, add code to handle versions as far back as the previous
>> release
>> * The operator rolls out the updated code, keeping the existing config file
>> with the pin to the previous release's RPC version
>> * Once all services are upgraded, the operator rolls out a new config file
>> shifting the pin
>> * The backwards compat code to handle release N-1 is removed in the N+1
>> release
>>
>> This is, I believe, sufficient to solve our entire problem.
>> Specifically, we have no need for an indirection API that rebroadcasts
>> messages that are too new (since that can't happen with pinning) and no
>> need for Versioned Objects in the RPC layer. (Versioned objects for the DB
>> are still critical, and we are very much better off for all the hard work that
>> Michal and others have put into them. Thanks!)
>
> What is the use of versioned objects outside of RPC?
> I've written some documentation for Oslo VO and helped in introducing them in Heat.
> As I understand it, the only use cases for VO are
> * to serialize objects to dicts with version information when they are sent over RPC
> * handle version dependent code inside the objects (instead of scattering it around the codebase)

This is what we currently have them for AIUI - so that we can do updates 
to the DB schema without downtime. The VO abstracts the 
version-dependent parts in a single place, so the rest of the code can 
continue to work even with an old DB schema. This means that we can roll 
out new code and only update the schema once it is all running.

> * provide an object oriented and transparent access to the resources represented by the objects to services which don't have direct access to that resource (via RPC) - the indirection API
>
> The last point was not yet discussed in Heat as far as I know, but the indirection API also contains an interface for backporting objects, which is something that is currently only used in Nova, and as you say, doesn't have a use when version pinning is in place.
>
>>
>> The nature of Heat's RPC API is that it is effectively user-facing - the heat-api
>> process is essentially a thin proxy between ReST and RPC. We already have a
>> translation layer between the internal representation(s) of objects and the
>> user-facing representation, in the form of heat.engine.api, and the RPC API is
>> firmly on the user-facing side. The requirements for the content of these
>> messages are actually much stricter than anything we need for RPC API
>> stability, since they need to remain compatible not just with heat-api but
>> with heatclient - and we have *zero* control over when that gets upgraded.
>> Despite that, we've managed quite nicely for ~3 years without breaking
>> changes afaik.
>>
>> Versioned Objects is a great way of retaining control when you need to share
>> internal data structures between processes. Fortunately the architecture of
>> Heat makes that unnecessary. That was a good design decision. We are not
>> going to reverse that design decision in order to use Versioned Objects. (In
>> the interest of making sure everyone uses their time productively, perhaps I
>> should clarify that to: "your patch is subject to -2 on sight if it introduces
>> internal engine data structures to heat-api/heat-cfn-api".)
>>
>> Hopefully I've convinced you of the sufficiency of this plan for the API<-
>>> engine interface specifically. If anyone disagrees, let them speak now, &c.
>
> I don't understand - what is the distinction between internal and external data structures?

Internal data structures are just whatever it's most convenient for the 
application to work with. They're subject to change/refactoring at any time.

External data structures are ones that have an explicit stability 
guarantee, in this case because they're part of the user interface.

>  From what I understand, versioned objects were introduced in Heat to represent objects which are sent over RPC between Heat services.

I believe you have been misinformed. All of the versioned objects that 
have been created so far are representations of tables in the database. 
However, we never send these over RPC: pre-VO it would obviously have 
been dumb to couple the user interface to the representation in the 
database, so we just didn't do that and therefore that was never a 
motivation for creating those VOs. And given that we already have a 
functioning abstraction layer in the engine, I don't think it makes 
sense to rewrite it and move it to a different process now.

>> I think there is still a case that could be made for a different approach to the
>> RPC API for convergence, which is engine->engine and
>> (probably) doesn't yet have a formal translation layer of the same kind.
>> At a minimum, obviously, we should do the same stuff listed above (though I
>> don't think we need to declare that interface stable until the first release
>> where we enable convergence by default).
>>
>
> I agree this could be a good use case for VO.
>
>
>> There's probably places where versioned objects could benefit us. For
>> example, when we trigger a check on a resource we pass it a bundle of data
>> containing all the attributes and IDs it might need from resources it depends
>> on. It definitely makes sense to me that that bundle would be a Versioned
>> Object. (In fact, that data gets stored in the DB - as SyncPoint in the
>> prototype - so we wouldn't even need to create a new object type. This
>> seems like a clear win.)
>>
>> What I do NOT want to do is to e.g. replace the resource_id of the resource
>> to check with a versioned object containing the DB representation of a
>> resource. In most cases that doesn't save a DB lookup, it just moves it to the
>> other end of the RPC call. And as long as primary keys are unique, this can
>> never create a compatibility problem :)
>
> Agreed. In a case it saves additional DB lookups though, sending an object can be a good option.

Yeah, I agree that it could be in that case. I don't think that's the 
case, but it would be nice to have the flexibility to adopt that in the 
future if it turns out that it is. Do you think your proposed solution 
(i.e. the indirection API mentioned below) would make it easier to e.g. 
swap the resource key for a versioned object in a future version of the 
message?

>> Grzegorz mentioned in https://review.openstack.org/#/c/196670/ the
>> possibility of using the indirection API to handle changes to the message
>> format - I guess this is an alternative to the version pinning approach
>> mentioned by Dan? I'm currently leaning toward thinking that this is
>> unnecessary, but I'm not sure I understand it completely so I'd need to see a
>> patch to form an opinion. I'm wide open to persuasion on this part. However
>> I'd be opposed to this *replacing* (rather than
>> supplementing) the version pinning behaviour described above - the whole
>> retransmission of too-new messages thing sounds like it could have any
>> number of pathological corner cases and I'd prefer to avoid it.
>>
>
> The only difference in my proposed solution is that the "if version == 'x.x'" code would be constrained to the version object implementation (inside the heat/objects directory), and the rest of the code could use the object properties and methods without being concerned about versioning.

OK, thanks for clarifying.

cheers,
Zane.