August 2023 - openstack-discuss - lists.openstack.org

[SDK][CLI][API][OpenAPI] Machine readable OpenStack API spec - time to do a next step?
by Artem Goncharov 01 Sep '23

01 Sep '23

Hey, Some of you may know that for certain time I was doing experiments on the sdk/cli/ansible_mod side with regards to the code generation. Reason for that is simple: huge codebases and too few resources. While not being the primary goal of this initiative, but still a pointer into the same direction is that most of the tools we need to maintain do the same stuff with some adaptations necessary for the framework in question. I did a lot of diverse attempts trying to take current OpenStackSDK codebase as a “source of truth” regarding what exists in which service and use it to generate code (OSC, AnsibleModule, etc). Pretty quickly it became clear that we do not have enough information to cover all the different needs: - when query parameter has been added into or dropped from the API - what is the type of the query parameter and whether in the case it is list it should be rendered as "?tags=a,b" or "?tags=a&tags=b” - which fields are necessary to be send in the request, which are optional and how to deal when you send name but get back UUID in the same field - what are the parameter/resource fields types in detail with respect to strongly typed programming languages (“int” is not good to differentiate between u8 and f64) - how to determine list of supported “actions” on the resource which are above the plain CRUD Firstly I tried to extend our current model of things in SDK, but even that was not so helpful. And then I decided to give another look at OpenAPI and our desires since multiple years to move into that direction. First of all it provides all the required information to do what is necessary. And so I tried to gather all the things we said about OpenAPI at OpenStack and which challenges we have and compare it against the OpenAPI 3.1. Problems: 1) OpenAPI does not support model of microversions. To be more precise on that you can’t describe different request/response body based on the header differentiator 2) URL + Method are representing unique combination making it impossible to represent different actions based on the content 3) All relevant Headers are supposed to be described in the spec. For example Swift is using X-Object-Meta-* prefix to allow user customisations what is not currently supported by the spec Actually that’s it, everything else was doable. Have I missed something? So what can we do about it: 1) OpenAPI 3.1 is now based upon jsonschema, what allows huge flexibility in describing object schema including inheritance and polymorphism (with composition, discriminator, etc). So we can describe the necessary Microversion Header and schema using “oneOf” whenever necessary. Moreover it is possible to extend type schema with “x-“ prefixed custom properties. So I went and started adding “x-openstack-“ prefixed properties where necessary 2) I personally agree with OpenAPI that URL+method should be enough to uniquely identify what is expected (while not a big deal for the API consumer, it would have been easier on the server side to have URL uniquely tied with the data and operation instead of dynamically looking at the request to figure out what to do. This is actually a most complex thing to deal with and the only solution that comes to my mind would be not to try to put everything into single spec, but rather have a dedicated specs for different operations. It makes it anyway easier to maintain such spec. Maybe at some next OpenAPI iteration this gets resolved and we can adopt changes. 3) A tricky thing, but there is nothing preventing us specifying headers as “x-object-meta-*” and defining custom processors for things like that to for example compose/decompose this into dictionary on the client side OpenAPI now even allows defining spec extensions if necessary, but I think at the moment we can cover all issues without that. Demo time: - server live migration (in my eyes one of the trickiest things I have seen so far from the OpenStack APIs). Here you have an action requiring a dedicated spec, multiple revisions with micro versions and even parameter type bool which can be a string as well: [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ More real-life examples in [4] - image create and upload: [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ Here it is possible to put few methods into the single spec. You can also see here samples of different media-types, capture headers, readOnly/writeOnly properties (You can actually take those specs and paste them into https://editor-next.swagger.io/ and see how it looks like) Closing notes: I have implemented a WIP change [1] in SDK that consumes list of specs and performs requested operation. Series of SDK commands are converted to the new way dealing with APIs. This is backwards compatible, but opens a road to heavily reduce amount of code in SDK (precisely by dropping all those customisations that are necessary now). Honestly speaking my drafts even show performance improvement on SDK side in the area of 100-300ms for diverse API operations, but I am still not really able to explain why. Maybe because of avoiding most black magic of SDK. I assume after completion we could get another performance boost by dropping other unnecessary logic and abstractions. The work is not completed as the change title states. Once we have specs for the commands code generators can be updated to consume this data and produce much better code. BTW, I have now also a prototype of the new CLI written with Rust and this is a hypersonic bullet compared to the current OSC. But please do not push me more on that, it is still in a very early stages and depends heavily on the available specs. I can only say that it is now designed in the way that every API call can have a thin CLI coverage just by providing a spec, when additional logic is desired - surely will require human implementation. Code generators in the pipe: OSC, AnsibleModules, RustSDK (sync/async), RustCLI. Next thing that are on the radar: gopher cloud, terraform modules, async python sdk, JS SDK(?) If all of that gets executed properly and with some community traction we can all have following things covered: - improve standardisation of OpenStack internals and externals: glance and nova (at least those 2) are already using jsonschema internally in different areas to describe requests/responses. Why not to make this standard reaching the service consumers? - getting rid of api-ref work by updating our sphinx machinery to consume our customised specs and produce nice docs matching the reality - sharing specs between teams to improve interface (not like currently we need to read the api-ref with tons of bugs plus source code to understand how to cover new feature in service X). Maybe even a central repo with the specs per release. - there are plenty of code generators and server bindings for OpenAPI specs so that we can potentially align frameworks used by different teams to maintain less - less work for all of us who needs services talking to each other (not immediately right now, but once the code is switched on consuming specs) - request verification already on the client side not waiting for the response - finally show something to customers often annoying asking “where are your openapi specs” (no offence here ;-))? I know it is a long message. But I am pretty excited with the progress and would like to hear community opinions. For the more detailed discussion consider this as a pre-announcement of the topic for PTG in sdk/cli slots. Huge invest but huge outcome P.S. it can result in a good chunk of relatively easy work for students Regards, Artem [1] https://review.opendev.org/c/openstack/openstacksdk/+/892161 [2] https://paste.opendev.org/show/bcxFi2CrNX1YNEuTzoEh/ [3] https://paste.opendev.org/show/bNN1t0qVmHEdVHu0Mtrn/ [4] https://review.opendev.org/c/openstack/openstacksdk/+/893365

4 5

[all] osprofiler requirements on opentelemetry: is this reasonable?
by Thomas Goirand 01 Sep '23

01 Sep '23

Hi, I just saw that the last version of osprofiler requires opentelemetry-exporter-otlp and opentelemetry-sdk. Since osprofiler is used in almost all projects, and that opentelemetry is really HUGE, I wonder if this is all reasonable. This forces downstream distros to do a lot of packaging only for that. Is this avoidable? I started doing the packaging, and then stopped to write this message, seeing how much work that would represent. Your thoughts? Cheers, Thomas Goirand (zigo)

3 3

[Rally]
by Aayushi Gautam 31 Aug '23

31 Aug '23

Hello, I am Aayushi , an intern with Redhat working on ESI ( Elastic secure Infracstructure) group. We were trying to use Rally to test the performance of our codebase. I have created a Plugin and task . And getting an error. The same bug is also asked on the bug page of Rally, but it was not answered. I have attached the code of the plugin and task and the error message. Looking forward to hearing from you.

2 5

Issue with Live Migration
by Chris DiLorenzo 31 Aug '23

31 Aug '23

We are running zed deployed with kolla-ansible. We are running into an issue with Live Migration. These are the steps to reproduce the issue: - Create a VM - Attach a Cinder Volume. This will create a new Storage Controller. - Start writing data to the new disk. We used FIO. Let FIO run until it starts reporting IOPS data. - Live migrate the VM - The VM will migrate and FIO will continue to run. FIO will show 0 or no IOPS - lspci output will show that the new storage device is in error. - The VM will remain in this state until the user issues a hard reset command - Once the VM is rebooted, the issue can not be reproduced until another device is attached This also happens if we attach a new VNIC. Prior to migration, this is what we see: root@test-vm-cd:~# cat before_migration_lspci.txt | grep block 05:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) Subsystem: Red Hat, Inc. Virtio block device 08:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) Subsystem: Red Hat, Inc. Virtio block device Post migration we see this: 08:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev ff) (prog-if ff) !!! Unknown header type 7f Kernel driver in use: virtio-pci The original SCSI controller is fine. Outside of the above error, we don't see any other errors. From Openstack's perspective, the migration went fine. Any suggestions on where to look? Has anyone else seen this? Thanks, Chris

1 0

[elections][sahara] PTL Candidacy for 2024.1
by Jerry Zhou (周鑫勇) 31 Aug '23

31 Aug '23

Hello everyone! I would like to announce my candidacy for the role of Sahara PTL during the 2024.1 cycle. I have been in this role for the last cycle, and I would like to continue serving as PTL. During the past cycles, there was a lot of effort put into promoting contributor collaboration through various activities. We also revised unstable unit/functional tests to improve the quality of development, and created documents for users to demonstrate use cases with practical examples. In the next cycle, I'd like to continue improving unit testing and documentation quality.

1 0

TripleO centos9 wallaby IPA job is failing
by Bhagyashri Shewale 31 Aug '23

31 Aug '23

Hi folks, All the centos9 wallaby IPA jobs are failing due to [1] and blocking gate. We are currently working on it. [1]: https://bugs.launchpad.net/tripleo/+bug/2033357 Thanks and regards Bhagyashri Shewale

2 1

Openstack Horizon Login Issues
by Asma Naz Shariq 31 Aug '23

31 Aug '23

Hi Team, Can anyone help me to resolve this issue of openstack dashboard-Yoga release? You do not have permission to access the resource: /horizon/ Login as different user or go back to home page. How to resolve this issue in openstack while accessing web interface. Can anyone guide me how to resolve this issue? when i entered the credentials to access to openstack dashboard, it display mesasge: Something went wrong! An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator. Regards, Asma

2 1

migration error
by Karera Tony 31 Aug '23

31 Aug '23

Hello Team, I am migrating an instance from one Cpmpute to another but I am getting this error *Error: *Failed to live migrate instance to host "compute1". Details <http://10.10.13.31/admin/instances/#message_details> Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-6bc103a8-f1f1-489c-a620-ec535b651280) Please advise Regards Tony Karera

1 0

Re: openstack-discuss Digest, Vol 58, Issue 108
by Karera Tony 31 Aug '23

31 Aug '23

My issue was resolved after upgrading kolla-ansible. Regards Tony Karera On Wed, Aug 30, 2023 at 3:53 PM < openstack-discuss-request(a)lists.openstack.org> wrote: > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. [keystone][election] Self-nomination for Keystone PTL for > 2024.1 cycle (Dave Wilde) > 2. Re: openstack-discuss Digest, Vol 58, Issue 107 (Karera Tony) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 30 Aug 2023 08:35:00 -0500 > From: Dave Wilde <dwilde(a)redhat.com> > To: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> > Subject: [keystone][election] Self-nomination for Keystone PTL for > 2024.1 cycle > Message-ID: <54d82977-aee0-4df2-a7c3-2be3b8c2788c@Spark> > Content-Type: text/plain; charset="utf-8" > > Hey folks, > > It's Dave here, your current OpenStack Keystone PTL. I'd like to submit my > candidacy to again act as the PTL for the 2024.1 cycle.??We're making great > progress with Keystone, and I would like to continue with that excellent > work. > > As PTL in this cycle here are some of the things I'd like to focus on: > > ??- Finish the manager role and ensure that the SRBAC implied roles are > correct > ?? ?for manager and member > ??- Continue the OAuth 2.0 Implementation > ??- Start a known issues section in the Keystone documentation > ??- Start a documentation audit to ensure that our documentation is of the > ?? ?highest quality > > Of course we will continue the weekly meetings and the reviewathons.??I > think > the reviewathons have been very successful and I'm keen on keeping them > going. > > I'm looking forward to another successful cycle and working with everyone > again! > > Thank you! > > /Dave >

1 0

[elections][venus] PTL Candidacy for 2024.1
by Eric Zhang (张连法)-浪潮信息 30 Aug '23

30 Aug '23

Hello everyone! I would like to announce my candidacy for the role of Venus PTL during the Caracal/2024.1 cycle. I have been in this role for the last 2 cycles and I would like to continue serving as PTL. In the upcoming cycle, I would like to focus on the following: * Continuous improvement in documents, unit tests, clients and dashboard. * Enhance log anomaly detection, such as more matching rules, improving performance. Thank you for your consideration! Lianfa Zhang

1 0