[TC][openapi] Request for new projects under the OpenStack hat
Hi TC representatives, Some time ago I was communicating about my POC on generating OpenAPI schemas from the source code of services. I continued working on that and have now few projects (that I was developing on GitHub to make life easier) that I would like to bring under the OpenStack hood (and place them under the SDK team). I have also figured out that Nova is now reviewing spec on starting using supporting this (thanks Stephen) and Neutron folks expressed also interest. === codegenerator https://gtema.github.io/openstack-codegenerator/ This project is responsible for generation of the OpenAPI specs and generate other (currently user-facing) code ouf of them === openstack-openapi https://gtema.github.io/openstack-openapi/ This repo hosts rendered specs and provides analogue to the os-api-ref Sphinx extension that renders them into the html (what you see under the link is a mix of Swagger and os-api-ref style). I was not really effectively working on that rendering lately, the repo right now is more or less a storage for the specs themselves with a first sketch of sphinx extension of rendering and basic OpenAPI flavoring doc. === openstack-rs https://gtema.github.io/openstack/ This is a Rust monorepo project with few items: - SDK for OpenStack out of the rendered OpenAPI specs - experimental CLI following very different approach compared to the python- openstackclient. It is compiled into single binary (what helps those not willing to pull whole python with all deps and the insanity we experience lately or those who build docker containers with only OSC inside to bypass the mentioned hell) and purely auto-generated from the specs - [future] openstack-tui (similar to k9s for Kubernetes) Here it is all really oriented only on the user facing tooling. I am very convinced in this project and will continue working on it independent on the outcome. Any objections/ideas/comments? Should I just go and create 3 project-config changes pending the approval? ====== Since I am on this topic anyway I would also like to address another related question. While working for OpenTelekomCloud we developed HashiVault plugin for managing OpenStack credentials as secrets (I had a talk on that in Vancouver). We tried to bring this to OpenDev but were unsure under which namespace. Pretty shortly a Hashi license issue started circling around and after few back and forth iterations we were not sure what to do with that project at all. Anyway the plugin itself is Apache-2 and working. It should be possible to also use it with OpenBAO (as a fork of Vault), but it was not tested so far. I am now not with Telekom anymore, but people there are still willing to contribute it under the OpenStack. Do you have any opinions on that? Is there interest in getting this under the OpenDev and if yes then which namespace (openstack or X)? ======= Thanks, Artem
On 2024-02-22 11:47:35 +0100 (+0100), gtema wrote: [...]
Since I am on this topic anyway I would also like to address another related question. While working for OpenTelekomCloud we developed HashiVault plugin for managing OpenStack credentials as secrets (I had a talk on that in Vancouver). We tried to bring this to OpenDev but were unsure under which namespace. Pretty shortly a Hashi license issue started circling around and after few back and forth iterations we were not sure what to do with that project at all. Anyway the plugin itself is Apache-2 and working. It should be possible to also use it with OpenBAO (as a fork of Vault), but it was not tested so far.
Possible you missed my comment on your abandoned project creation change, but OpenBao is an open source fork of Vault, if switching to that makes the problem simpler.
I am now not with Telekom anymore, but people there are still willing to contribute it under the OpenStack. Do you have any opinions on that? Is there interest in getting this under the OpenDev and if yes then which namespace (openstack or X)?
Please not the "x/" namespace, that was a dumping ground for all projects whose maintainers never got back to us specifying a preference for their individual namespaces. If it's not going into "openstack/" you can create whatever new namespace you want, but we should avoid putting new maintained things in "x/" when possible. -- Jeremy Stanley
On Thursday, 22 February 2024 14:49:09 CET Jeremy Stanley wrote:
On 2024-02-22 11:47:35 +0100 (+0100), gtema wrote: [...]
Since I am on this topic anyway I would also like to address another related question. While working for OpenTelekomCloud we developed HashiVault plugin for managing OpenStack credentials as secrets (I had a talk on that in Vancouver). We tried to bring this to OpenDev but were unsure under which namespace. Pretty shortly a Hashi license issue started circling around and after few back and forth iterations we were not sure what to do with that project at all. Anyway the plugin itself is Apache-2 and working. It should be possible to also use it with OpenBAO (as a fork of Vault), but it was not tested so far.
Possible you missed my comment on your abandoned project creation change, but OpenBao is an open source fork of Vault, ...
Neah, that actually triggered mentioning it here. After frickler again asked me about the change I decided to just abandon and forget it. Your comment showed me there might be still some interested parties and decided to raise this more publicly (before restoring the change). wrt "x/" vs non "x/": got it, thanks
On Thu, 2024-02-22 at 11:47 +0100, gtema wrote:
Hi TC representatives,
Some time ago I was communicating about my POC on generating OpenAPI schemas from the source code of services. I continued working on that and have now few projects (that I was developing on GitHub to make life easier) that I would like to bring under the OpenStack hood (and place them under the SDK team). I have also figured out that Nova is now reviewing spec on starting using supporting this (thanks Stephen) and Neutron folks expressed also interest.
=== codegenerator https://gtema.github.io/openstack-codegenerator/
This project is responsible for generation of the OpenAPI specs and generate other (currently user-facing) code ouf of them
=== openstack-openapi https://gtema.github.io/openstack-openapi/
This repo hosts rendered specs and provides analogue to the os-api-ref Sphinx extension that renders them into the html (what you see under the link is a mix of Swagger and os-api-ref style). I was not really effectively working on that rendering lately, the repo right now is more or less a storage for the specs themselves with a first sketch of sphinx extension of rendering and basic OpenAPI flavoring doc. i personally object to haveing a offical repo to define the open api scemas for offial projcet that scope of the relevent repo.
i.e. i do not agree that we shoudl have openapi defintion for the nova api ourside of the nova repo or a repo that is not a deliverable of the comptue project. stephen has proposed a spec to add openapi support to nova directly https://review.opendev.org/c/openstack/nova-specs/+/909448 what i really dont wnat to see is going forward an api change to nova woudl also requrie the oepnapi schema be updated by submiting a patch to a speratre repo. so im fine with haveing schemas in the nova repo but im not ok with having a https://opendev.org/openstack/openstack-openapi/ repo
=== openstack-rs https://gtema.github.io/openstack/
This is a Rust monorepo project with few items: - SDK for OpenStack out of the rendered OpenAPI specs - experimental CLI following very different approach compared to the python- openstackclient. It is compiled into single binary (what helps those not willing to pull whole python with all deps and the insanity we experience lately or those who build docker containers with only OSC inside to bypass the mentioned hell) and purely auto-generated from the specs - [future] openstack-tui (similar to k9s for Kubernetes)
i was actully playing with rust and pyo3 to try and generate rust data types form the jsonscheam defintion of novas oslo versioend objects https://github.com/SeanMooney/nova-oxide thats mainly to paly with rust and rust<>python interaction in partarlary. i was toying with the idea of writing a deamon that coudl comuncitate with nova comonets over the rpc bus effectivly a rust implemetiaon of a nova-comptue agent as way to learn rust. basically nova-comput ewould need effectivly a rewrite to remove our use of eventlet anyway so i was toying with the idea of if we are rewriting should we rewtien in something other then python. have you considerd using something like pyo3 to provide python binding around a rust inmpelmation of the sdk to support both lanageues form a singel codeabse. https://pyo3.rs/v0.20.3/
Here it is all really oriented only on the user facing tooling. I am very convinced in this project and will continue working on it independent on the outcome.
Any objections/ideas/comments?
yep see above. mainly just decopleing the schmas form the implemation. if we keep the schemas with the python code in the same repo or in a lib repo provded by each service project i.e. neturon-lib, nova-lib ectra then that i could be on board with.
Should I just go and create 3 project-config changes pending the approval?
====== Since I am on this topic anyway I would also like to address another related question. While working for OpenTelekomCloud we developed HashiVault plugin for managing OpenStack credentials as secrets (I had a talk on that in Vancouver). We tried to bring this to OpenDev but were unsure under which namespace. Pretty shortly a Hashi license issue started circling around and after few back and forth iterations we were not sure what to do with that project at all. Anyway the plugin itself is Apache-2 and working. It should be possible to also use it with OpenBAO (as a fork of Vault), but it was not tested so far. I am now not with Telekom anymore, but people there are still willing to contribute it under the OpenStack. Do you have any opinions on that? Is there interest in getting this under the OpenDev and if yes then which namespace (openstack or X)? =======
Thanks, Artem
On Fri, 2024-02-23 at 17:15 +0000, smooney@redhat.com wrote:
On Thu, 2024-02-22 at 11:47 +0100, gtema wrote:
Hi TC representatives,
Some time ago I was communicating about my POC on generating OpenAPI schemas from the source code of services. I continued working on that and have now few projects (that I was developing on GitHub to make life easier) that I would like to bring under the OpenStack hood (and place them under the SDK team). I have also figured out that Nova is now reviewing spec on starting using supporting this (thanks Stephen) and Neutron folks expressed also interest.
=== codegenerator https://gtema.github.io/openstack-codegenerator/
This project is responsible for generation of the OpenAPI specs and generate other (currently user-facing) code ouf of them
=== openstack-openapi https://gtema.github.io/openstack-openapi/
This repo hosts rendered specs and provides analogue to the os-api-ref Sphinx extension that renders them into the html (what you see under the link is a mix of Swagger and os-api-ref style). I was not really effectively working on that rendering lately, the repo right now is more or less a storage for the specs themselves with a first sketch of sphinx extension of rendering and basic OpenAPI flavoring doc. i personally object to haveing a offical repo to define the open api scemas for offial projcet that scope of the relevent repo.
i.e. i do not agree that we shoudl have openapi defintion for the nova api ourside of the nova repo or a repo that is not a deliverable of the comptue project.
stephen has proposed a spec to add openapi support to nova directly https://review.opendev.org/c/openstack/nova-specs/+/909448
what i really dont wnat to see is going forward an api change to nova woudl also requrie the oepnapi schema be updated by submiting a patch to a speratre repo. so im fine with haveing schemas in the nova repo but im not ok with having a https://opendev.org/openstack/openstack-openapi/ repo
In case it's not obvious, I am a big advocate of these things living in-tree in the projects. However, this only works if the projects teams work with us. There are going to be both specs and code changes (to add schemas) to review. Those will either need reviewer attention or some other mechanism to get them in. If we don't get that, we're going to need to explore alternatives.
=== openstack-rs https://gtema.github.io/openstack/
This is a Rust monorepo project with few items: - SDK for OpenStack out of the rendered OpenAPI specs - experimental CLI following very different approach compared to the python- openstackclient. It is compiled into single binary (what helps those not willing to pull whole python with all deps and the insanity we experience lately or those who build docker containers with only OSC inside to bypass the mentioned hell) and purely auto-generated from the specs - [future] openstack-tui (similar to k9s for Kubernetes)
i was actully playing with rust and pyo3 to try and generate rust data types form the jsonscheam defintion of novas oslo versioend objects https://github.com/SeanMooney/nova-oxide thats mainly to paly with rust and rust<>python interaction in partarlary.
i was toying with the idea of writing a deamon that coudl comuncitate with nova comonets over the rpc bus effectivly a rust implemetiaon of a nova-comptue agent as way to learn rust. basically nova-comput ewould need effectivly a rewrite to remove our use of eventlet anyway so i was toying with the idea of if we are rewriting should we rewtien in something other then python.
have you considerd using something like pyo3 to provide python binding around a rust inmpelmation of the sdk to support both lanageues form a singel codeabse. https://pyo3.rs/v0.20.3/
The way that the SDK is currently implemented leaves too much ambiguity and relies too much on magic (setattr and getattr calls _everywhere_) to be useful as a basis for anything else. If anything, Gophercloud, while far less complete, would be a better basis. We can fix this, and I have some thoughts for how to do so, but that's going to take time. On the subject of Rust and Rust-based OpenStack components, it's worth noting that an API spec would allow for easy reimplementation of the user-facing API components in other frameworks or languages. I'm not saying that's a good idea or something we want, but it does become slightly more viable. Go- or Rust-based nova-api, anyone? Stephen
Here it is all really oriented only on the user facing tooling. I am very convinced in this project and will continue working on it independent on the outcome.
Any objections/ideas/comments?
yep see above. mainly just decopleing the schmas form the implemation. if we keep the schemas with the python code in the same repo or in a lib repo provded by each service project
i.e. neturon-lib, nova-lib ectra then that i could be on board with.
Should I just go and create 3 project-config changes pending the approval?
====== Since I am on this topic anyway I would also like to address another related question. While working for OpenTelekomCloud we developed HashiVault plugin for managing OpenStack credentials as secrets (I had a talk on that in Vancouver). We tried to bring this to OpenDev but were unsure under which namespace. Pretty shortly a Hashi license issue started circling around and after few back and forth iterations we were not sure what to do with that project at all. Anyway the plugin itself is Apache-2 and working. It should be possible to also use it with OpenBAO (as a fork of Vault), but it was not tested so far. I am now not with Telekom anymore, but people there are still willing to contribute it under the OpenStack. Do you have any opinions on that? Is there interest in getting this under the OpenDev and if yes then which namespace (openstack or X)? =======
Thanks, Artem
Thanks a lot, Stephen, for an explicit response On Friday, 1 March 2024 14:20:57 CET Stephen Finucane wrote:
On Fri, 2024-02-23 at 17:15 +0000, smooney@redhat.com wrote:
On Thu, 2024-02-22 at 11:47 +0100, gtema wrote:
Hi TC representatives,
Some time ago I was communicating about my POC on generating OpenAPI schemas from the source code of services. I continued working on that and have now few projects (that I was developing on GitHub to make life easier) that I would like to bring under the OpenStack hood (and place them under the SDK team). I have also figured out that Nova is now reviewing spec on starting using supporting this (thanks Stephen) and Neutron folks expressed also interest.
=== codegenerator https://gtema.github.io/openstack-codegenerator/
This project is responsible for generation of the OpenAPI specs and generate other (currently user-facing) code ouf of them
=== openstack-openapi https://gtema.github.io/openstack-openapi/
This repo hosts rendered specs and provides analogue to the os-api-ref Sphinx extension that renders them into the html (what you see under the link is a mix of Swagger and os-api-ref style). I was not really effectively working on that rendering lately, the repo right now is more or less a storage for the specs themselves with a first sketch of sphinx extension of rendering and basic OpenAPI flavoring doc.
i personally object to haveing a offical repo to define the open api scemas for offial projcet that scope of the relevent repo.
i.e. i do not agree that we shoudl have openapi defintion for the nova api ourside of the nova repo or a repo that is not a deliverable of the comptue project.
stephen has proposed a spec to add openapi support to nova directly https://review.opendev.org/c/openstack/nova-specs/+/909448
what i really dont wnat to see is going forward an api change to nova woudl also requrie the oepnapi schema be updated by submiting a patch to a speratre repo. so im fine with haveing schemas in the nova repo but im not ok with having a https://opendev.org/openstack/openstack-openapi/ repo
In case it's not obvious, I am a big advocate of these things living in-tree in the projects. However, this only works if the projects teams work with us. There are going to be both specs and code changes (to add schemas) to review. Those will either need reviewer attention or some other mechanism to get them in. If we don't get that, we're going to need to explore alternatives.
I would also prefer if we can get specs out of service repo as easy as possible. The only thing that disturbs me is that if spec is an artifact of some process it is going to be unnecessary complex to get it under the same git. We do not store rendered HTML docs in git and the same should be done with spec I am explicitly talking about spec and not schemas). But anyway, if we are not able to get changes into the repos then it just stays like it is right now - part of the codegenerator. I stared populating it with schemas in hope to have them merged sometime back into the corresponding repos. I need to have something to work with. btw, I proposed a very similar spec to Keystone so that I can move the schemas directly into service code next cycle(s).
=== openstack-rs https://gtema.github.io/openstack/
This is a Rust monorepo project with few items: - SDK for OpenStack out of the rendered OpenAPI specs - experimental CLI following very different approach compared to the python- openstackclient. It is compiled into single binary (what helps those not willing to pull whole python with all deps and the insanity we experience lately or those who build docker containers with only OSC inside to bypass the mentioned hell) and purely auto-generated from the specs - [future] openstack-tui (similar to k9s for Kubernetes)
i was actully playing with rust and pyo3 to try and generate rust data types form the jsonscheam defintion of novas oslo versioend objects https://github.com/SeanMooney/nova-oxide thats mainly to paly with rust and rust<>python interaction in partarlary.
i was toying with the idea of writing a deamon that coudl comuncitate with nova comonets over the rpc bus effectivly a rust implemetiaon of a nova-comptue agent as way to learn rust. basically nova-comput ewould need effectivly a rewrite to remove our use of eventlet anyway so i was toying with the idea of if we are rewriting should we rewtien in something other then python.
have you considerd using something like pyo3 to provide python binding around a rust inmpelmation of the sdk to support both lanageues form a singel codeabse. https://pyo3.rs/v0.20.3/
The way that the SDK is currently implemented leaves too much ambiguity and relies too much on magic (setattr and getattr calls _everywhere_) to be useful as a basis for anything else. If anything, Gophercloud, while far less complete, would be a better basis. We can fix this, and I have some thoughts for how to do so, but that's going to take time.
Same on my side - ideas are there, but current focus is on putting the whole idea on the feet. In Rust I have used completely different design of SDK and as of now it looks definitely much better than in the current OpenStackSDK
On the subject of Rust and Rust-based OpenStack components, it's worth noting that an API spec would allow for easy reimplementation of the user-facing API components in other frameworks or languages. I'm not saying that's a good idea or something we want, but it does become slightly more viable. Go- or Rust-based nova-api, anyone?
Would be really awesome.
Stephen
Here it is all really oriented only on the user facing tooling. I am very convinced in this project and will continue working on it independent on the outcome.
Any objections/ideas/comments?
yep see above. mainly just decopleing the schmas form the implemation. if we keep the schemas with the python code in the same repo or in a lib repo provded by each service project
i.e. neturon-lib, nova-lib ectra then that i could be on board with.
Should I just go and create 3 project-config changes pending the approval?
For good or bad I opened change to project-config with adding 2 new repos: generator and doc/sphinx extension and a corresponding change to governance repo. Let's see when they get processed. For Rust I generally disagree on direct requirement of adding new runtime under the OpenStack since we talk about client bindings for different programming languages and not runtime of OpenStack services. Imagine someone will now want to build sdk for Haskel - should we go through all formalities of adding new programming language under the governance just because of that? I will definitely support adding Rust as a new language (since its programming safety is so freaking good as I have ensured myself during sdk/cli development), I just disagree with policy requiring that (most policies were created from the service perspective and not from the user). I will be clarifying this more actively with TC directly after TC elections since it looks TC is on mute in front of elections.
participants (4)
-
gtema
-
Jeremy Stanley
-
smooney＠redhat.com
-
Stephen Finucane