Thank you Pierre! That fixed the "long" issue for now. When I look up the placement CLI I find this document: https://docs.openstack.org/placement/latest/cli/index.html I don't see anything here about getting hypervisor stats. What's the recommended new method? Albert On Thursday, October 10, 2024 at 06:13:35 AM EDT, Pierre Riteau <pierre@stackhpc.com> wrote: Hello, I don't know about your first issue, but your second one is due to a Nova microversion change in Wallaby which removed various fields from the os-hypervisors API: https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht... is recommended to use Placement instead, but you can still get long output using any API version below 2.88 (e.g. --os-compute-api-version 2.87). Cheers,Pierre Riteau (priteau) On Wed, 9 Oct 2024 at 22:49, Albert Braden <ozzzo@yahoo.com> wrote: After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list: Train: openstack server list --all-projects --locked --os-compute-api-version 2.73 <empty list because nothing is locked> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> Wallaby (locked flag is ignored): openstack server list --all-projects --locked --os-compute-api-version 2.73 <list of every VM in the cluster> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> The second problem is the hypervisor list. We can’t get “long” output in Wallaby: Train: openstack hypervisor list --long |  ID | Hypervisor Hostname             | Hypervisor Type | Host IP        | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | Wallaby (long flag is ignored): openstack hypervisor list --long | ID                                   | Hypervisor Hostname             | Hypervisor Type | Host IP        | State |

Regarding the "locked" issue, when I run debug, I see that, without an API specified, both Train and Wallaby use compute API version 2.1. Train will allow versions up to 2.79 and Wallaby will allow up to 2,.88. The obvious difference is that Train sends 'tenant_id' and 'all_tenants' to the API, whereas Wallaby sends 'project_id' and 'all_projects'. Could this be what causes the "locked" flag to be ignored? The other obvious difference is that the Wallaby client doesn't complain when I used "locked" and don't specify an API version. That seems like a bug, but it is probably not causing this issue. Train: Without --os-compute-api-version 2.73: openstack server list --all-projects --locked --debug compute API version 2.1, cmd group openstack.compute.v2 --os-compute-api-version 2.73 or greater is required to support the --locked option With --os-compute-api-version 2.73: openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug compute API version 2.73, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'instance_name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'tenant_id': None, 'all_tenants': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True} <returns correct list of only locked VMs> Wallaby: Without --os-compute-api-version 2.73: openstack server list --all-projects --locked --debug compute API version 2.1, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True} <returns all VMs> With --os-compute-api-version 2.73: openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug compute API version 2.73, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True} <returns all VMs> With --os-compute-api-version 2.88: openstack server list --all-projects --locked --os-compute-api-version 2.88 --debug compute API version 2.88, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True} <returns all VMs> On Thursday, October 10, 2024 at 06:13:35 AM EDT, Pierre Riteau <pierre@stackhpc.com> wrote: Hello, I don't know about your first issue, but your second one is due to a Nova microversion change in Wallaby which removed various fields from the os-hypervisors API: https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht... is recommended to use Placement instead, but you can still get long output using any API version below 2.88 (e.g. --os-compute-api-version 2.87). Cheers,Pierre Riteau (priteau) On Wed, 9 Oct 2024 at 22:49, Albert Braden <ozzzo@yahoo.com> wrote: After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list: Train: openstack server list --all-projects --locked --os-compute-api-version 2.73 <empty list because nothing is locked> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> Wallaby (locked flag is ignored): openstack server list --all-projects --locked --os-compute-api-version 2.73 <list of every VM in the cluster> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> The second problem is the hypervisor list. We can’t get “long” output in Wallaby: Train: openstack hypervisor list --long |  ID | Hypervisor Hostname             | Hypervisor Type | Host IP        | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | Wallaby (long flag is ignored): openstack hypervisor list --long | ID                                   | Hypervisor Hostname             | Hypervisor Type | Host IP        | State |

---- On Fri, 11 Oct 2024 10:07:38 -0700 Sean Mooney wrote ---

On 11/10/2024 17:51, Albert Braden wrote:

...

Regarding the "locked" issue, when I run debug, I see that, without an API specified, both Train and Wallaby use compute API version 2.1. Train will allow versions up to 2.79 and Wallaby will allow up to 2,.88. The obvious difference is that Train sends 'tenant_id' and 'all_tenants' to the API, whereas Wallaby sends 'project_id' and 'all_projects'. Could this be what causes the "locked" flag to be ignored?

so there was a propsoal to move nova's api to project_id isntead of teant_id

https://review.opendev.org/q/topic:%22bp/remove-tenant-id%22

the specs were approved but never implemted form waht i can see

so if osc or the sdk is sending project_id then that a bug

OSC also always send tenant-id and all_tenants in case project-id is passed: - https://github.com/openstack/python-openstackclient/blob/wallaby-eom/opensta... -gmann

this was the patch for server list https://review.opendev.org/c/openstack/nova/+/765311

...

The other obvious difference is that the Wallaby client doesn't complain when I used "locked" and don't specify an API version. That seems like a bug, but it is probably not causing this issue.

Train: Without --os-compute-api-version 2.73: openstack server list --all-projects --locked --debug compute API version 2.1, cmd group openstack.compute.v2 --os-compute-api-version 2.73 or greater is required to support the --locked option

With --os-compute-api-version 2.73: openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug compute API version 2.73, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'instance_name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'tenant_id': None, 'all_tenants': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

Wallaby: Without --os-compute-api-version 2.73: openstack server list --all-projects --locked --debug compute API version 2.1, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

With --os-compute-api-version 2.73: openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug compute API version 2.73, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

With --os-compute-api-version 2.88: openstack server list --all-projects --locked --os-compute-api-version 2.88 --debug compute API version 2.88, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

On Thursday, October 10, 2024 at 06:13:35 AM EDT, Pierre Riteau pierre@stackhpc.com> wrote:

Hello,

I don't know about your first issue, but your second one is due to a Nova microversion change in Wallaby which removed various fields from the os-hypervisors API: https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht... It is recommended to use Placement instead, but you can still get long output using any API version below 2.88 (e.g. --os-compute-api-version 2.87).

Cheers, Pierre Riteau (priteau)

On Wed, 9 Oct 2024 at 22:49, Albert Braden ozzzo@yahoo.com> wrote:

After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list:

Train: openstack server list --all-projects --locked --os-compute-api-version 2.73

openstack server list --all-projects --unlocked --os-compute-api-version 2.73

Wallaby (locked flag is ignored): openstack server list --all-projects --locked --os-compute-api-version 2.73

openstack server list --all-projects --unlocked --os-compute-api-version 2.73

The second problem is the hypervisor list. We can’t get “long” output in Wallaby:

Train: openstack hypervisor list --long |  ID | Hypervisor Hostname             | Hypervisor Type | Host IP        | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |

Wallaby (long flag is ignored): openstack hypervisor list --long | ID                                   | Hypervisor Hostname             | Hypervisor Type | Host IP        | State |

If I type "openstack --version" I get 6.6.0 On Friday, October 11, 2024 at 01:28:01 PM EDT, Ghanshyam Mann <gmann@ghanshyammann.com> wrote: ---- On Fri, 11 Oct 2024 09:51:32 -0700  Albert Braden  wrote ---

                Regarding the "locked" issue, when I run debug, I see that, without an API specified, both Train and Wallaby use compute API version 2.1. Train will allow versions up to 2.79 and Wallaby will allow up to 2,.88. The obvious difference is that Train sends 'tenant_id' and 'all_tenants' to the API, whereas Wallaby sends 'project_id' and 'all_projects'. Could this be what causes the "locked" flag to be ignored?

This should not cause the issue, as both should be same and supported.

The other obvious difference is that the Wallaby client doesn't complain when I used "locked" and don't specify an API version. That seems like a bug, but it is probably not causing this issue.

Till microversion 2.75, Nova ignored a few of the invalid query parameters, which means it does not raise errors, and no filtering is done based on those invalid parameters and 'locked' was one of them. Though API server still ignores them if passed before microversion  2.73 but on the client side, the 'locked' parameter is passed with microversion <2.73, then an error is always raised. I am not sure why you are not seeing the error for Wallaby when requesting locked with v2.1. Can you please let me know the client version? On another issue where you are seeing all VMs when requesting the locked one only on Wallaby, I am trying to reproduce that and will see what exactly causing it. Can you please let me know the client version which will help. Nova API server side, cannot find anything which can cause this, but maybe client implementation is causing it? -gman

Train: Without --os-compute-api-version 2.73: openstack server list --all-projects --locked --debug compute API version 2.1, cmd group openstack.compute.v2 --os-compute-api-version 2.73 or greater is required to support the --locked option

With --os-compute-api-version 2.73: openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug compute API version 2.73, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'instance_name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'tenant_id': None, 'all_tenants': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

Wallaby: Without --os-compute-api-version 2.73: openstack server list --all-projects --locked --debug compute API version 2.1, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

With --os-compute-api-version 2.73: openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug compute API version 2.73, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

With --os-compute-api-version 2.88: openstack server list --all-projects --locked --os-compute-api-version 2.88 --debug compute API version 2.88, cmd group openstack.compute.v2 search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}

                                                                                                        On Thursday, October 10, 2024 at 06:13:35 AM EDT, Pierre Riteau pierre@stackhpc.com> wrote:                                                                                                            Hello, I don't know about your first issue, but your second one is due to a Nova microversion change in Wallaby which removed various fields from the os-hypervisors API: https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht... is recommended to use Placement instead, but you can still get long output using any API version below 2.88 (e.g. --os-compute-api-version 2.87). Cheers,Pierre Riteau (priteau) On Wed, 9 Oct 2024 at 22:49, Albert Braden ozzzo@yahoo.com> wrote: After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list:

Train: openstack server list --all-projects --locked --os-compute-api-version 2.73

openstack server list --all-projects --unlocked --os-compute-api-version 2.73

Wallaby (locked flag is ignored): openstack server list --all-projects --locked --os-compute-api-version 2.73

openstack server list --all-projects --unlocked --os-compute-api-version 2.73

The second problem is the hypervisor list. We can’t get “long” output in Wallaby:

Train: openstack hypervisor list --long |  ID | Hypervisor Hostname             | Hypervisor Type | Host IP        | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |

Wallaby (long flag is ignored): openstack hypervisor list --long | ID                                   | Hypervisor Hostname             | Hypervisor Type | Host IP        | State |

                           

That was the issue. If I use the 5.5.0 client the problem goes away. Thanks for helping me find that! On Friday, October 11, 2024 at 02:37:38 PM EDT, Ghanshyam Mann <gmann@ghanshyammann.com> wrote: ---- On Fri, 11 Oct 2024 10:50:56 -0700  Albert Braden  wrote ---

                If I type "openstack --version" I get 6.6.0

Ok,  you are using the Caracal (2024.1) client on train and wallaby. I am not sure if those are guaranteed to work on the old OpenStack services. Can you try the below tested/compatible released version corresponding to Wallaby and Train: Wallaby: 5.5.0 Train: 4.0.0 -gmann

                                                                                                        On Friday, October 11, 2024 at 01:28:01 PM EDT, Ghanshyam Mann gmann@ghanshyammann.com> wrote:                                                                                                              ---- On Fri, 11 Oct 2024 09:51:32 -0700  Albert Braden  wrote ---   >                 Regarding the "locked" issue, when I run debug, I see that, without an API specified, both Train and Wallaby use compute API version 2.1. Train will allow versions up to 2.79 and Wallaby will allow up to 2,.88. The obvious difference is that Train sends 'tenant_id' and 'all_tenants' to the API, whereas Wallaby sends 'project_id' and 'all_projects'. Could this be what causes the "locked" flag to be ignored?

This should not cause the issue, as both should be same and supported.

  >   > The other obvious difference is that the Wallaby client doesn't complain when I used "locked" and don't specify an API version. That seems like a bug, but it is probably not causing this issue.

Till microversion 2.75, Nova ignored a few of the invalid query parameters, which means it does not raise errors, and no filtering is done based on those invalid parameters and 'locked' was one of them. Though API server still ignores them if passed before microversion  2.73 but on the client side, the 'locked' parameter is passed with microversion <2.73, then an error is always raised. I am not sure why you are not seeing the error for Wallaby when requesting locked with v2.1. Can you please let me know the client version?

On another issue where you are seeing all VMs when requesting the locked one only on Wallaby, I am trying to reproduce that and will see what exactly causing it. Can you please let me know the client version which will help. Nova API server side, cannot find anything which can cause this, but maybe client implementation is causing it?

-gman

  >   > Train:   > Without --os-compute-api-version 2.73:   > openstack server list --all-projects --locked --debug   > compute API version 2.1, cmd group openstack.compute.v2   > --os-compute-api-version 2.73 or greater is required to support the --locked option   >   > With --os-compute-api-version 2.73:   > openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug   > compute API version 2.73, cmd group openstack.compute.v2   > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'instance_name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'tenant_id': None, 'all_tenants': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}   >   >   > Wallaby:   > Without --os-compute-api-version 2.73:   > openstack server list --all-projects --locked --debug   > compute API version 2.1, cmd group openstack.compute.v2   > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}   >   >   > With --os-compute-api-version 2.73:   > openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug   > compute API version 2.73, cmd group openstack.compute.v2   > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}   >   >   > With --os-compute-api-version 2.88:   > openstack server list --all-projects --locked --os-compute-api-version 2.88 --debug   > compute API version 2.88, cmd group openstack.compute.v2   > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}   >   >                                                                                                         On Thursday, October 10, 2024 at 06:13:35 AM EDT, Pierre Riteau pierre@stackhpc.com> wrote:                                          >                       >                                                 Hello,   > I don't know about your first issue, but your second one is due to a Nova microversion change in Wallaby which removed various fields from the os-hypervisors API: https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht... is recommended to use Placement instead, but you can still get long output using any API version below 2.88 (e.g. --os-compute-api-version 2.87).   > Cheers,Pierre Riteau (priteau)   > On Wed, 9 Oct 2024 at 22:49, Albert Braden ozzzo@yahoo.com> wrote:   > After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list:   >   > Train:   > openstack server list --all-projects --locked --os-compute-api-version 2.73   >   > openstack server list --all-projects --unlocked --os-compute-api-version 2.73   >   >   > Wallaby (locked flag is ignored):   > openstack server list --all-projects --locked --os-compute-api-version 2.73   >   > openstack server list --all-projects --unlocked --os-compute-api-version 2.73   >   >   > The second problem is the hypervisor list. We can’t get “long” output in Wallaby:   >   > Train:   > openstack hypervisor list --long   > |  ID | Hypervisor Hostname             | Hypervisor Type | Host IP        | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |   >   > Wallaby (long flag is ignored):   > openstack hypervisor list --long   > | ID                                   | Hypervisor Hostname             | Hypervisor Type | Host IP        | State |   >   >   >   >   >                                                        

On Fri, 2024-10-11 at 21:50 +0000, Jeremy Stanley wrote:

On 2024-10-11 20:30:37 +0000 (+0000), Albert Braden wrote:

...

That was the issue. If I use the 5.5.0 client the problem goes away. Thanks for helping me find that! [...]

For what it's worth, I think the OpenStackClient maintainers would consider that a bug worth fixing. They generally want people to be able to use newer client versions with older existing OpenStack deployments, but it becomes increasingly hard to deploy older and older vintages of OpenStack services for automated testing of new changes in development, so sometimes regressions do slip in.

Yes, please. I took a quick look at this and spotted that the 'locked' filter is not currently supported by openstacksdk. I have proposed a fix and will backport that once merged: https://review.opendev.org/c/openstack/openstacksdk/+/932400 https://review.opendev.org/c/openstack/openstacksdk/+/932401 https://review.opendev.org/c/openstack/openstacksdk/+/932402 https://review.opendev.org/c/openstack/openstacksdk/+/932403 https://review.opendev.org/c/openstack/openstacksdk/+/932404 In general, openstackclient and openstacksdk should support everything but the most ancient OpenStack releases (we technically still support things like the identity v2 and block storage v2 APIs, as well as features like nova-network, though of course we have no ability to test these nowadays). It should be safe to use either library from pip to administer clouds from Mitaka through to Dalmatian. If you notice this isn't the case, meaning a feature stops working between releases and there's nothing in the release notes about it, then it's most likely a bug and should be reported. As for why you don't get warning on the Wallaby OSC version when you don't pass '--os-compute-api-version': we migrated the 'server list' command from using novaclient under the hood to using openstack. openstacksdk encodes a maximum API microversion for each resource (in this case, the openstack.compute.v2.server.Server resource) and will attempt to use this unless told otherwise or if the server does not support it. In the latter case, it will downgrade the version unless the field is not supported. Hope this helps, Stephen