[SDK][CLI][API][OpenAPI] Machine readable OpenStack API spec - time to do a next step?
Hey,
Some of you may know that for certain time I was doing experiments on the sdk/cli/ansible_mod side with regards to the code generation. Reason for that is simple: huge codebases and too few resources. While not being the primary goal of this initiative, but still a pointer into the same direction is that most of the tools we need to maintain do the same stuff with some adaptations necessary for the framework in question.
I did a lot of diverse attempts trying to take current OpenStackSDK codebase as a “source of truth” regarding what exists in which service and use it to generate code (OSC, AnsibleModule, etc). Pretty quickly it became clear that we do not have enough information to cover all the different needs: - when query parameter has been added into or dropped from the API - what is the type of the query parameter and whether in the case it is list it should be rendered as "?tags=a,b" or "?tags=a&tags=b” - which fields are necessary to be send in the request, which are optional and how to deal when you send name but get back UUID in the same field - what are the parameter/resource fields types in detail with respect to strongly typed programming languages (“int” is not good to differentiate between u8 and f64) - how to determine list of supported “actions” on the resource which are above the plain CRUD
Firstly I tried to extend our current model of things in SDK, but even that was not so helpful. And then I decided to give another look at OpenAPI and our desires since multiple years to move into that direction. First of all it provides all the required information to do what is necessary. And so I tried to gather all the things we said about OpenAPI at OpenStack and which challenges we have and compare it against the OpenAPI 3.1. Problems:
1) OpenAPI does not support model of microversions. To be more precise on that you can’t describe different request/response body based on the header differentiator 2) URL + Method are representing unique combination making it impossible to represent different actions based on the content 3) All relevant Headers are supposed to be described in the spec. For example Swift is using X-Object-Meta-* prefix to allow user customisations what is not currently supported by the spec
Actually that’s it, everything else was doable. Have I missed something? So what can we do about it:
1) OpenAPI 3.1 is now based upon jsonschema, what allows huge flexibility in describing object schema including inheritance and polymorphism (with composition, discriminator, etc). So we can describe the necessary Microversion Header and schema using “oneOf” whenever necessary. Moreover it is possible to extend type schema with “x-“ prefixed custom properties. So I went and started adding “x-openstack-“ prefixed properties where necessary 2) I personally agree with OpenAPI that URL+method should be enough to uniquely identify what is expected (while not a big deal for the API consumer, it would have been easier on the server side to have URL uniquely tied with the data and operation instead of dynamically looking at the request to figure out what to do. This is actually a most complex thing to deal with and the only solution that comes to my mind would be not to try to put everything into single spec, but rather have a dedicated specs for different operations. It makes it anyway easier to maintain such spec. Maybe at some next OpenAPI iteration this gets resolved and we can adopt changes. 3) A tricky thing, but there is nothing preventing us specifying headers as “x-object-meta-*” and defining custom processors for things like that to for example compose/decompose this into dictionary on the client side
OpenAPI now even allows defining spec extensions if necessary, but I think at the moment we can cover all issues without that.
Demo time: - server live migration (in my eyes one of the trickiest things I have seen so far from the OpenStack APIs). Here you have an action requiring a dedicated spec, multiple revisions with micro versions and even parameter type bool which can be a string as well: [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ More real-life examples in [4] - image create and upload: [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ Here it is possible to put few methods into the single spec. You can also see here samples of different media-types, capture headers, readOnly/writeOnly properties (You can actually take those specs and paste them into https://editor-next.swagger.io/ and see how it looks like)
Closing notes: I have implemented a WIP change [1] in SDK that consumes list of specs and performs requested operation. Series of SDK commands are converted to the new way dealing with APIs. This is backwards compatible, but opens a road to heavily reduce amount of code in SDK (precisely by dropping all those customisations that are necessary now). Honestly speaking my drafts even show performance improvement on SDK side in the area of 100-300ms for diverse API operations, but I am still not really able to explain why. Maybe because of avoiding most black magic of SDK. I assume after completion we could get another performance boost by dropping other unnecessary logic and abstractions. The work is not completed as the change title states.
Once we have specs for the commands code generators can be updated to consume this data and produce much better code. BTW, I have now also a prototype of the new CLI written with Rust and this is a hypersonic bullet compared to the current OSC. But please do not push me more on that, it is still in a very early stages and depends heavily on the available specs. I can only say that it is now designed in the way that every API call can have a thin CLI coverage just by providing a spec, when additional logic is desired - surely will require human implementation. Code generators in the pipe: OSC, AnsibleModules, RustSDK (sync/async), RustCLI. Next thing that are on the radar: gopher cloud, terraform modules, async python sdk, JS SDK(?)
If all of that gets executed properly and with some community traction we can all have following things covered:
- improve standardisation of OpenStack internals and externals: glance and nova (at least those 2) are already using jsonschema internally in different areas to describe requests/responses. Why not to make this standard reaching the service consumers? - getting rid of api-ref work by updating our sphinx machinery to consume our customised specs and produce nice docs matching the reality - sharing specs between teams to improve interface (not like currently we need to read the api-ref with tons of bugs plus source code to understand how to cover new feature in service X). Maybe even a central repo with the specs per release. - there are plenty of code generators and server bindings for OpenAPI specs so that we can potentially align frameworks used by different teams to maintain less - less work for all of us who needs services talking to each other (not immediately right now, but once the code is switched on consuming specs) - request verification already on the client side not waiting for the response - finally show something to customers often annoying asking “where are your openapi specs” (no offence here ;-))?
I know it is a long message. But I am pretty excited with the progress and would like to hear community opinions. For the more detailed discussion consider this as a pre-announcement of the topic for PTG in sdk/cli slots.
Huge invest but huge outcome
P.S. it can result in a good chunk of relatively easy work for students
Regards, Artem
[1] https://review.opendev.org/c/openstack/openstacksdk/+/892161 [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ [4] https://review.opendev.org/c/openstack/openstacksdk/+/893365
On Thu, Aug 31, 2023 at 3:47 PM Artem Goncharov artem.goncharov@gmail.com wrote:
Hey,
Some of you may know that for certain time I was doing experiments on the sdk/cli/ansible_mod side with regards to the code generation. Reason for that is simple: huge codebases and too few resources. While not being the primary goal of this initiative, but still a pointer into the same direction is that most of the tools we need to maintain do the same stuff with some adaptations necessary for the framework in question.
I did a lot of diverse attempts trying to take current OpenStackSDK codebase as a “source of truth” regarding what exists in which service and use it to generate code (OSC, AnsibleModule, etc). Pretty quickly it became clear that we do not have enough information to cover all the different needs:
- when query parameter has been added into or dropped from the API
- what is the type of the query parameter and whether in the case it is list it should be rendered as "?tags=a,b" or "?tags=a&tags=b”
- which fields are necessary to be send in the request, which are optional and how to deal when you send name but get back UUID in the same field
- what are the parameter/resource fields types in detail with respect to strongly typed programming languages (“int” is not good to differentiate between u8 and f64)
- how to determine list of supported “actions” on the resource which are above the plain CRUD
Firstly I tried to extend our current model of things in SDK, but even that was not so helpful. And then I decided to give another look at OpenAPI and our desires since multiple years to move into that direction. First of all it provides all the required information to do what is necessary. And so I tried to gather all the things we said about OpenAPI at OpenStack and which challenges we have and compare it against the OpenAPI 3.1. Problems:
- OpenAPI does not support model of microversions. To be more precise on that you can’t describe different request/response body based on the header differentiator
- URL + Method are representing unique combination making it impossible to represent different actions based on the content
- All relevant Headers are supposed to be described in the spec. For example Swift is using X-Object-Meta-* prefix to allow user customisations what is not currently supported by the spec
Actually that’s it, everything else was doable. Have I missed something? So what can we do about it:
- OpenAPI 3.1 is now based upon jsonschema, what allows huge flexibility in describing object schema including inheritance and polymorphism (with composition, discriminator, etc). So we can describe the necessary Microversion Header and schema using “oneOf” whenever necessary. Moreover it is possible to extend type schema with “x-“ prefixed custom properties. So I went and started adding “x-openstack-“ prefixed properties where necessary
- I personally agree with OpenAPI that URL+method should be enough to uniquely identify what is expected (while not a big deal for the API consumer, it would have been easier on the server side to have URL uniquely tied with the data and operation instead of dynamically looking at the request to figure out what to do. This is actually a most complex thing to deal with and the only solution that comes to my mind would be not to try to put everything into single spec, but rather have a dedicated specs for different operations. It makes it anyway easier to maintain such spec. Maybe at some next OpenAPI iteration this gets resolved and we can adopt changes.
- A tricky thing, but there is nothing preventing us specifying headers as “x-object-meta-*” and defining custom processors for things like that to for example compose/decompose this into dictionary on the client side
OpenAPI now even allows defining spec extensions if necessary, but I think at the moment we can cover all issues without that.
Demo time:
- server live migration (in my eyes one of the trickiest things I have seen so far from the OpenStack APIs). Here you have an action requiring a dedicated spec, multiple revisions with micro versions and even parameter type bool which can be a string as well: [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ More real-life examples in [4]
Just an fyi we have jsonschema schmas stored in python dicts for every api in nova https://github.com/openstack/nova/blob/4490c8bc8461a56a96cdaa08020c9e25ebe8f...
we use them in our api tests with eh API scamples which are use both as test and to generate docs https://github.com/openstack/nova/tree/master/doc/api_samples/server-migrati...
every time we make an api change we add an api sample and add a secma for that micorversion.
These schemas are also use in the api to validate the requests https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/mig...
so a lot of the data (boht on the spec side an confromance side) exits but not in a form that is directly usabel to export openapi formated documents.
- image create and upload: [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ Here it is possible to put few methods into the single spec. You can also see here samples of different media-types, capture headers, readOnly/writeOnly properties
(You can actually take those specs and paste them into https://editor-next.swagger.io/ and see how it looks like)
Closing notes: I have implemented a WIP change [1] in SDK that consumes list of specs and performs requested operation. Series of SDK commands are converted to the new way dealing with APIs. This is backwards compatible, but opens a road to heavily reduce amount of code in SDK (precisely by dropping all those customisations that are necessary now). Honestly speaking my drafts even show performance improvement on SDK side in the area of 100-300ms for diverse API operations, but I am still not really able to explain why. Maybe because of avoiding most black magic of SDK. I assume after completion we could get another performance boost by dropping other unnecessary logic and abstractions. The work is not completed as the change title states.
Once we have specs for the commands code generators can be updated to consume this data and produce much better code. BTW, I have now also a prototype of the new CLI written with Rust and this is a hypersonic bullet compared to the current OSC. But please do not push me more on that, it is still in a very early stages and depends heavily on the available specs. I can only say that it is now designed in the way that every API call can have a thin CLI coverage just by providing a spec, when additional logic is desired - surely will require human implementation. Code generators in the pipe: OSC, AnsibleModules, RustSDK (sync/async), RustCLI. Next thing that are on the radar: gopher cloud, terraform modules, async python sdk, JS SDK(?)
If all of that gets executed properly and with some community traction we can all have following things covered:
- improve standardisation of OpenStack internals and externals: glance and nova (at least those 2) are already using jsonschema internally in different areas to describe requests/responses. Why not to make this standard reaching the service consumers?
- getting rid of api-ref work by updating our sphinx machinery to consume our customised specs and produce nice docs matching the reality
- sharing specs between teams to improve interface (not like currently we need to read the api-ref with tons of bugs plus source code to understand how to cover new feature in service X). Maybe even a central repo with the specs per release.
- there are plenty of code generators and server bindings for OpenAPI specs so that we can potentially align frameworks used by different teams to maintain less
- less work for all of us who needs services talking to each other (not immediately right now, but once the code is switched on consuming specs)
- request verification already on the client side not waiting for the response
- finally show something to customers often annoying asking “where are your openapi specs” (no offence here ;-))?
I know it is a long message. But I am pretty excited with the progress and would like to hear community opinions. For the more detailed discussion consider this as a pre-announcement of the topic for PTG in sdk/cli slots.
Huge invest but huge outcome
P.S. it can result in a good chunk of relatively easy work for students
Regards, Artem
[1] https://review.opendev.org/c/openstack/openstacksdk/+/892161 [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ [4] https://review.opendev.org/c/openstack/openstacksdk/+/893365
Just an fyi we have jsonschema schmas stored in python dicts for every api in nova https://github.com/openstack/nova/blob/4490c8bc8461a56a96cdaa08020c9e25ebe8f...
we use them in our api tests with eh API scamples which are use both as test and to generate docs https://github.com/openstack/nova/tree/master/doc/api_samples/server-migrati...
every time we make an api change we add an api sample and add a secma for that micorversion.
These schemas are also use in the api to validate the requests https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/mig...
so a lot of the data (boht on the spec side an confromance side) exits but not in a form that is directly usabel to export openapi formated documents.
Thanks Sean, I know that and because of that I mentioned that nova is already using jsonschema. That is more or less exactly the point that I want to have a discussion on making this standard by the projects and do this generally with the openapi spec instead of bunch of unconnected schemas so that our own lives become hopefully easier.
On Thu, 2023-08-31 at 18:36 +0200, Artem Goncharov wrote:
Just an fyi we have jsonschema schmas stored in python dicts for every api in nova https://github.com/openstack/nova/blob/4490c8bc8461a56a96cdaa08020c9e25ebe8f...
we use them in our api tests with eh API scamples which are use both as test and to generate docs https://github.com/openstack/nova/tree/master/doc/api_samples/server-migrati...
every time we make an api change we add an api sample and add a secma for that micorversion.
These schemas are also use in the api to validate the requests https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/mig...
so a lot of the data (boht on the spec side an confromance side) exits but not in a form that is directly usabel to export openapi formated documents.
Thanks Sean, I know that and because of that I mentioned that nova is already using jsonschema. That is more or less exactly the point that I want to have a discussion on making this standard by the projects and do this generally with the openapi spec instead of bunch of unconnected schemas so that our own lives become hopefully easier.
ya i wanted to highlight that for you and others to see how they can and are currently used. if we were to standatise this acroos services what i would want to see is the exesitign schemas ported to openapi schema files which we would then import and uses for our validation.
that is a non trivaial amount of work but a the end of the process we woudl have a set of schma files that could be used to generate client.
we coudld also add a midelware to oslo mideelware that would optionally be able to enumarate the schemas and server them form the rest api if loaded.
im not against proting to a well know standard provided we do not loose any fo the test coverage we or validation capablities we have today as a result.
in nova the schmas benifit us today by narrowing the domain of check we need to preform in the python code directly i.e. we know if you get to the live_migrate handeler in the nova api that the body is syntacticaly valid due to the schema vlaidation and that you have permissiosn to perform the operators due to the keystoen auth and policy checks that have already been done.
that means we only need to validate the semantic i.e. does the instance you asked us to migrate exist? is it in a migratble state ectra. the schemas get rid of a lot of boilerplate in that regard so i think it would be useful for other too. and if other project were to add schmea validation today it would make sense ot align to a standard rather then inventing there own or copyting hte nova code. the nova code works but its kindof our own thign even if it is jasonschma based so it requires more knowledge to extend and uses then following a standard.
On Fri, Sep 1, 2023, 15:42 smooney@redhat.com wrote:
On Thu, 2023-08-31 at 18:36 +0200, Artem Goncharov wrote:
Just an fyi we have jsonschema schmas stored in python dicts for every api in nova
https://github.com/openstack/nova/blob/4490c8bc8461a56a96cdaa08020c9e25ebe8f...
we use them in our api tests with eh API scamples which are use both as test and to generate docs
https://github.com/openstack/nova/tree/master/doc/api_samples/server-migrati...
every time we make an api change we add an api sample and add a secma for that micorversion.
These schemas are also use in the api to validate the requests
https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/mig...
so a lot of the data (boht on the spec side an confromance side) exits but not in a form that is directly usabel to export openapi formated documents.
Thanks Sean, I know that and because of that I mentioned that nova is
already using jsonschema. That is more or less
exactly the point that I want to have a discussion on making this
standard by the projects and do this generally with
the openapi spec instead of bunch of unconnected schemas so that our own
lives become hopefully easier. ya i wanted to highlight that for you and others to see how they can and are currently used. if we were to standatise this acroos services what i would want to see is the exesitign schemas ported to openapi schema files which we would then import and uses for our validation.
Actually I have done in my poc nearly that: taken schema (as example, not literally) from nova code and integrated it into the spec. Surely it would be possible as you suggested to take schemas from the current code and construct them into the spec while we learn on how to have our middleware to locate them from the overall spec.
Generally openapi helps us (from 3.1) not to loose info, but vice versa extend and combine schemas into a full description of the operation how consumer is supposed to use it (what is the url, what are the parameters, what is the response(s)) with descriptions for each of those elements that are used for rendering docs and helpstrings.
On 31.08.23 16:39, Artem Goncharov wrote:
Once we have specs for the commands code generators can be updated to consume this data and produce much better code. BTW, I have now also a prototype of the new CLI written with Rust and this is a hypersonic bullet compared to the current OSC. But please do not push me more on that, it is still in a very early stages and depends heavily on the available specs. I can only say that it is now designed in the way that every API call can have a thin CLI coverage just by providing a spec, when additional logic is desired - surely will require human implementation. Code generators in the pipe: OSC, AnsibleModules, RustSDK (sync/async), RustCLI. Next thing that are on the radar: gopher cloud, terraform modules, async python sdk, JS SDK(?)
If all of that gets executed properly and with some community traction we can all have following things covered:
- improve standardisation of OpenStack internals and externals: glance
and nova (at least those 2) are already using jsonschema internally in different areas to describe requests/responses. Why not to make this standard reaching the service consumers?
- getting rid of api-ref work by updating our sphinx machinery to
consume our customised specs and produce nice docs matching the reality
- sharing specs between teams to improve interface (not like currently
we need to read the api-ref with tons of bugs plus source code to understand how to cover new feature in service X). Maybe even a central repo with the specs per release.
- there are plenty of code generators and server bindings for OpenAPI
specs so that we can potentially align frameworks used by different teams to maintain less
- less work for all of us who needs services talking to each other
(not immediately right now, but once the code is switched on consuming specs)
- request verification already on the client side not waiting for the
response
- finally show something to customers often annoying asking “where are
your openapi specs” (no offence here ;-))?
I know it is a long message. But I am pretty excited with the progress and would like to hear community opinions. For the more detailed discussion consider this as a pre-announcement of the topic for PTG in sdk/cli slots.
You have every reason to be excited - this sounds simply awesome!
I am a huge fan of OpenStackSDK and Gophercloud and the progress of aligning the contact surface to OpenStack APIs. Moving away from all the individual clients and different usage patterns ... But your new goals bring things to a whole new level! This is how a modern API landscape should look like! Validated and standardized schemas + code auto-generation for all sorts of API clients!
Let's go!
Christian
participants (4)
-
Artem Goncharov
-
Christian Rohmann
-
Sean Mooney
-
smooney＠redhat.com