On Fri, 2018-11-30 at 20:57 +0000, Sean Mooney wrote:

Hi

I wanted to raise the topic of os-vif, its public api what client are allowed to do and how that related to serialisation, persistence of os-vif objects and version compatibility.

sorry this is a bit of a long mail so here is a TL;DR and I have separated the mail into a background and future section.

- we are planning to refine our data model in os-vf for hardware offloads. https://review.openstack.org/#/c/607610/ - we have been talking on and off about using os-vif on for nova neutron port binding and negotiation. to do that we will need to support serialisation in the future. - there are some uses of os-vif in kuryr-kubenetes that are currently out of contract that we would like to correct as they may break in the future. https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/o... - what would we like to see in an os-vif 2.0

background ========== Before I get to the topic of os-vif I wanted to take an aside to set the basis of how I wanted to frame the topic.

So in C++ land, there is an international standard that defines the language and the standard library for the C++ language. one of the stipulations of that standard is that users of the standard library are allowed to construct types defined by the standard library and you can call functions and methods defined by the library. C++ has different semantics then python so the implication of the statement you can call functions may not be apparent. In C++ it is technically undefined behaviour to take the address of a standard library function you may only call it. This allows the library author to implement it as a macro,lambda, function, function object or any other callable. So, in other words, there is a clear delineation between the public API and host that API is implemented.

Bringing this back to os-vif we also have a finite public API and seperate internal implementation and I want to talk about internal changes that may affect observable features from the client perspective. specifically things that client can depend on and things they cannot.

first of all the public api of os-vif is defined by https://github.com/openstack/os-vif/blob/master/os_vif/plugin.py

this module defined the base class of all plugins and is the only class a plugin must inherit from and the only class defined in os-vif that a plugin or client of os-vif may inherit from outside of the exception classes defined in https://github.com/openstack/os-vif/blob/master/os_vif/exception.py.

the second part of os-vif public api is a collection of data structures defined in https://github.com/openstack/os-vif/tree/master/os_vif/objects that clients are expected to _construct_ and plugins may _read_ from.

I highlight construct and read as that is the extent to which we currently promise to clients or plugins. specifically today our version compatibility for objects is partially enabled by our use of oslo versioned objects. since os-vif objects are OVOs they have additional methods such as obj_to_primitive and obj_from_primitive that exist but that are _not_ part of the public api.

Because serialisation (out side of serialisation via privsep) and persistence of os-vif longer then the lifetime of a function call, is not today part of the supported use cases of os-vif, that has enabled us to be less strict with version compatible then we would have to be If we supported those features.

For example, we have in general added obj_make_compatible methods to objects when we modify os-vif objects https://github.com/openstack/os-vif/blob/master/os_vif/objects/vif.py#L120-L... but we have often discussed should we add these functions as there are no current uses.

future ====== That brings me to the topic of how os-vif is used in kuryr and how we would like to use it in neutron in the future. Part of the reason we do not allow clients to extend VIF objects is to ensure that os-vif plugins are not required on controller nodes and so we can control the set of vocabulary types that are exchanged and version them. To help us move to passing os-vif object via the api we need to remove the out of tree vif types that in use in kuryr and move them to os-vif. https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/o... we had mentioned this 2 or 3 releases ago but the change that prompted this email was https://github.com/openstack/kuryr-kubernetes/commit/7cc187806b42fee5ea660f8...

I think we can handle that through port profiles. Anyway it's definitely a smaller issue that we can figure out. Moreover the change you mention actually allows us to do modifications in the data structure we save with keeping the backward compatibility, so definitely we have some options.

i would like to know how kuryr-kubernetes is currently using the serialised os-vif objects. - are they being stored? - to disk - in a db - in etcd

We save it in K8s objects annotations, through K8s API, which saves it into the etcd. We assume no access to the etcd itself, so communication happens only through K8s mechanisms.

- are they bining sent to other services

No, but we read the VIF's from both kuryr-kubernetes and kuryr-daemon services.

- are how are you serialising them. - obj_to_primitive - other

It's obj_to_primitive(), see [1].

- are you depending on obj_make_compatible

There's no code for that yet, but we definitely planned that to happen to enable us to do changes in o.vo's we save in a backwards-compatible manner.

- how can we help you not use them until we actually support this use case.

Well, we need to use them. :( Either we use os-vif's o.vo's or we transition to anything else we can own now.

In the future, we will support serialising os-vif object but the topic of should we use ovo has been raised. Initially, we had planned to serialise os-vif objects by calling obj_to_primitive and then passing that to oslo.serialsiation json utils.

For python client, this makes it relatively trivial to serialise and deserialise the objects when calling into neutron but it has a disadvantage in that the format of the payload crated is specific to OpenStack, it is very verbose and it is not the most friendly format for human consumption.

The other options that have come up in the past few months have been removing OVOs and validating with json schema or more recently replacing OVOs with protobufs.

now we won't be doing either in the stein cycle but we had considered flattening our data models once https://review.openstack.org/#/c/607610/ is implemented and removing some unused filed as part of a 2.0 release in preparation for future neutron integration in Train. To that end, I would like to start the discussion about what we would want from a 2.0 release if and when we create one, how we can better docuemnt and enformce our public api/ supported usage patterens and are there any other users of os-vif outside of nova and kuryr today.

I believe we need to list the changes you want to make and work out how to keep compatibility. Kuryr itself doesn't use a lot of the fields, but we need to keep compatibility with objects saved on K8s resources by older versions of Kuryr-Kubernetes. This means that we still need a way to deserialize old o.vo's with the newer version of os-vif.

regards sean.

[1] https://github.com/openstack/kuryr-kubernetes/blob/38f541604ff490fbc381f78a2...