That was the issue. If I use the 5.5.0 client the problem goes away. Thanks for helping me find that!
On Friday, October 11, 2024 at 02:37:38 PM EDT, Ghanshyam Mann <gmann@ghanshyammann.com> wrote:


---- On Fri, 11 Oct 2024 10:50:56 -0700  Albert Braden  wrote ---
>                If I type "openstack --version" I get 6.6.0

Ok,  you are using the Caracal (2024.1) client on train and wallaby. I am not sure
if those are guaranteed to work on the old OpenStack services. Can you try the below
tested/compatible released version corresponding to Wallaby and Train:

Wallaby: 5.5.0
Train: 4.0.0

-gmann



>                                                                                                        On Friday, October 11, 2024 at 01:28:01 PM EDT, Ghanshyam Mann gmann@ghanshyammann.com> wrote:                                       
>                   
>                                                  ---- On Fri, 11 Oct 2024 09:51:32 -0700  Albert Braden  wrote ---
>  >                 Regarding the "locked" issue, when I run debug, I see that, without an API specified, both Train and Wallaby use compute API version 2.1. Train will allow versions up to 2.79 and Wallaby will allow up to 2,.88. The obvious difference is that Train sends 'tenant_id' and 'all_tenants' to the API, whereas Wallaby sends 'project_id' and 'all_projects'. Could this be what causes the "locked" flag to be ignored?
>
> This should not cause the issue, as both should be same and supported.
>
>  >
>  > The other obvious difference is that the Wallaby client doesn't complain when I used "locked" and don't specify an API version. That seems like a bug, but it is probably not causing this issue.
>
> Till microversion 2.75, Nova ignored a few of the invalid query parameters, which means it does not raise errors,
> and no filtering is done based on those invalid parameters and 'locked' was one of them. Though API server still
> ignores them if passed before microversion  2.73 but on the client side, the 'locked' parameter is passed with
> microversion <2.73, then an error is always raised. I am not sure why you are not seeing the error for Wallaby
> when requesting locked with v2.1. Can you please let me know the client version?
>
> On another issue where you are seeing all VMs when requesting the locked one only on Wallaby, I am trying to
> reproduce that and will see what exactly causing it. Can you please let me know the client version which will
> help. Nova API server side, cannot find anything which can cause this, but maybe client implementation is
> causing it?
>
> -gman
>
>
>  >
>  > Train:
>  > Without --os-compute-api-version 2.73:
>  > openstack server list --all-projects --locked --debug
>  > compute API version 2.1, cmd group openstack.compute.v2
>  > --os-compute-api-version 2.73 or greater is required to support the --locked option
>  >
>  > With --os-compute-api-version 2.73:
>  > openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug
>  > compute API version 2.73, cmd group openstack.compute.v2
>  > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'instance_name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'tenant_id': None, 'all_tenants': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}
>  >
>  >
>  > Wallaby:
>  > Without --os-compute-api-version 2.73:
>  > openstack server list --all-projects --locked --debug
>  > compute API version 2.1, cmd group openstack.compute.v2
>  > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}
>  >
>  >
>  > With --os-compute-api-version 2.73:
>  > openstack server list --all-projects --locked --os-compute-api-version 2.73 --debug
>  > compute API version 2.73, cmd group openstack.compute.v2
>  > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}
>  >
>  >
>  > With --os-compute-api-version 2.88:
>  > openstack server list --all-projects --locked --os-compute-api-version 2.88 --debug
>  > compute API version 2.88, cmd group openstack.compute.v2
>  > search options: {'reservation_id': None, 'ip': None, 'ip6': None, 'name': None, 'status': None, 'flavor': None, 'image': None, 'host': None, 'project_id': None, 'all_projects': True, 'user_id': None, 'deleted': False, 'changes-before': None, 'changes-since': None, 'locked': True}
>  >
>  >                                                                                                         On Thursday, October 10, 2024 at 06:13:35 AM EDT, Pierre Riteau pierre@stackhpc.com> wrote:                                       
>  >                    
>  >                                                 Hello,
>  > I don't know about your first issue, but your second one is due to a Nova microversion change in Wallaby which removed various fields from the os-hypervisors API: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-wallabyIt is recommended to use Placement instead, but you can still get long output using any API version below 2.88 (e.g. --os-compute-api-version 2.87).
>  > Cheers,Pierre Riteau (priteau)
>  > On Wed, 9 Oct 2024 at 22:49, Albert Braden ozzzo@yahoo.com> wrote:
>  > After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list:
>  >
>  > Train:
>  > openstack server list --all-projects --locked --os-compute-api-version 2.73
>  >
>  > openstack server list --all-projects --unlocked --os-compute-api-version 2.73
>  >
>  >
>  > Wallaby (locked flag is ignored):
>  > openstack server list --all-projects --locked --os-compute-api-version 2.73
>  >
>  > openstack server list --all-projects --unlocked --os-compute-api-version 2.73
>  >
>  >
>  > The second problem is the hypervisor list. We can’t get “long” output in Wallaby:
>  >
>  > Train:
>  > openstack hypervisor list --long
>  > |  ID | Hypervisor Hostname             | Hypervisor Type | Host IP        | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |
>  >
>  > Wallaby (long flag is ignored):
>  > openstack hypervisor list --long
>  > | ID                                   | Hypervisor Hostname             | Hypervisor Type | Host IP        | State |
>  >
>  >
>  >
>  >
>  >                            
>