Open Stack

> From: Andrew Laski [mailto:andrew at lascii.com]
> Sent: November 23, 2015 17:03
> 
> On 11/23/15 at 08:54am, Ryan Rossiter wrote:
> >
> >
> >On 11/23/2015 5:33 AM, John Garbutt wrote:
> >>On 20 November 2015 at 09:37, Balázs Gibizer
> >><balazs.gibizer at ericsson.com> wrote:
> >>><snip>
> >>>Minor version change shall not cause any problem for the consumer as
> the payload is backward compatible between minor versions. So if the
> consumer does not need the new field then he/she does not need to
> change anything in his/her parser. As far as I know we had a single major
> object version in nova so far so this is not a frequent event.  In case of major
> change I think we can offer version pinning from nova via configuration as a
> future step.
> >>>
> >>>The library idea has the problem that it would be python lib and
> consumers can be in any language. For me lib would be used for compat code
> but as I mentioned above incompatibility is not that frequent. In the other
> hand discoverability of notifications are more important for me. For that I can
> suggest providing notification samples as a first step so the consumer can see
> in the source tree what notifications are provided by the nova. As a natural
> next step would be to provide not just samples but schemas for the
> notifications. I haven't looked it too deep but I feel if we provide json schema
> for the notifications then the consumer can generate an object model from
> that schema without too much effort. It might not help directly with
> backporting the payload but at least automate things around it. The json
> schema has the benefit that it is language independent too.
> >>>
> >>So, on re-reading, I think we are missing some of the context from the
> >>summit session in this spec:
> >>http://specs.openstack.org/openstack/nova-
> specs/specs/mitaka/approved/
> >>versioned-notification-api.html
> >>
> >>The aim of notifications is to focus more on structured logging, for
> >>the operator.
> >>
> >>It is not trying to provide an end-user async API, such as:
> >>https://etherpad.openstack.org/p/liberty-cross-project-user-notificati
> >>ons
> >>
> >>As Gibi mentions, the idea was to add a version, so you can tell when
> >>the content has changed, generally in a backwards compatible way. The
> >>other big parts, like properly testing the notifications to ensure
> >>they stay compatible.
> >>
> >>It uses o.vo, with the intent we avoid a major version bump. Should we
> >>need one, its likely to end up with a similar transition phase to the
> >>current un-versioned->versioned transition phase (effectively the
> >>deployer decides which major versions will be emitted at any one
> >>time).
> >>
> >>The interface is a stable JSON structure, with some handy version and
> >>identification metadata. It happens to use o.vo to generate that
> >>format, and we happen to have python code to read that format, but
> >>thats not the aim.
> >This is a stupid question but... who's the object versioning designed
> >for? Is it for the consumer, so they know what they're getting? Or is
> >it for Nova, so it has a contract to fill, instead of tossing a big bag
> >of dicts across the wire? Those sound the same, but, suddenly in my
> >head, they aren't.
> >
> >In the case of the former, if we add a new field, we bump the version,
> >so the consumer knows when it can start looking for new things when it
> >gets that version.
> >
> >In the case of the latter, the fields defined in version 1.0 is the
> >only guaranteed law until the end of the universe. Every field added
> >after 1.0 needs to be nullable to allow backversioned computes to still
> >work, right? We're still handcuffed on what we can actually truly
> >guarantee the consumer.
> >>
> >>There is a bit I am conflicted/worried about, and thats when we start
> >>including verbatim, DB objects into the notifications. At least you
> >>can now quickly detect if that blob is something compatible with your
> >>current parsing code. My preference is really to keep the
> >>Notifications as a totally separate object tree, but I am sure there
> >>are many cases where that ends up being seemingly stupid duplicate
> >>work. I am not expressing this well in text form :(
> >Are you saying we don't want to be willy-nilly tossing DB objects
> >across the wire? Yeah that was part of the rug-pulling of just having
> >the payload contain an object. We're automatically tossing everything
> >with the object then, whether or not some of that was supposed to be a
> >secret. We could add some sort of property to the field like
> >dont_put_me_on_the_wire=True (or I guess a notification_ready()
> >function that helps an object sanitize itself?) that the notifications
> >will look at to know if it puts that on the wire-serialized dict, but
> >that's adding a lot more complexity and work to a pile that's already
> >growing rapidly.
> 
> I don't want to be tossing db objects across the wire.  But I also am not
> convinced that we should be tossing the current objects over the wire either.
> You make the point that there may be things in the object that shouldn't be
> exposed, and I think object version bumps is another thing to watch out for.
> So far the only object that has been bumped is Instance but in doing so no
> notifications needed to change.  I think if we just put objects into
> notifications we're coupling the notification versions to db or RPC changes
> unnecessarily.  Some times they'll move together but other times, like
> moving flavor into instance_extra, there's no reason to bump notifications.


Sanitizing existing versioned objects before putting them to the wire is not hard to do. 
You can see an example of doing it in
 https://review.openstack.org/#/c/245678/8/nova/objects/service.py,cm L382. 
We don't need extra effort to take care of minor version bumps because that does not
 break a well written consumer. We do have to take care of the major version bumps 
but that is a rare event and therefore can be handled one by one in a way John
suggested, by keep sending the previous major version for a while too. 


> Note that I'm not against using objects for notifications and versioning, but I
> picture having something like an InstanceNotification object which can
> handle converting an Instance object into a suitable notification
> version/format.

Converting between two object models is doable what I'm afraid of is that it 
means we have to maintain two object models. Which also means If a new
 field added to an internal object which needs to be in the related notification 
then we have to add them in the notification model as well. So one field but 
two places. That seems to be a duplication for me. I think we have to guess which
 case will be more frequent
a) adding a new field to an internal object that does not need to be added to the
related notification. In this case having two separate models is better. 
OR b) adding a new field to an internal object that needs to be added to the related
notification. In this case having one model and a filtering mechanism is better.

> 
> >>
> >>Thanks,
> >>John
> >>
> >>>Cheers,
> >>>Gibi
> >>>>--
> >>>>
> >>>>Thanks,
> >>>>
> >>>>Matt Riedemann
> >>>
> >>>_______________________________________________________
> ___________________
> >>>OpenStack Development Mailing List (not for usage questions)
> >>>Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>________________________________________________________
> __________________
> >>OpenStack Development Mailing List (not for usage questions)
> >>Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >--
> >Thanks,
> >
> >Ryan Rossiter (rlrossit)
> >
> >
> >_________________________________________________________
> _________________
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __________________________________________________________
> ________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev