so the openstack client is calling the nova simple tenant usage api https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... maps to https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant the openstack client does not have a --all-tenants flag or simplar to call https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for... so the difference is the openstack client uses /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone with the member or reader role in a given project the all tenats version is /os-simple-tenant-usage and that endpoint is admin only. osc has only ever support the tenant specific version so if you wantted to supprot the admin version you would need to modify osc to add a --all-tenant option to the cli. horizon presumably support the multi tenant view as an admin usign the /os-simple-tenant-usage endpoint. On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote:

Hi,

I've been struggling today to understand the problem. I wrote a (Python) script that uses directly the API instead of going through the "openstack usage list". I had some hope of something related to the limit parameter but unfortunately it is not the case and the result using the API is the same as using the openstack client. Only horizon returns all the information.

Does somebody has any clue about what is done differently by Horizon compared to the openstack client (I assume it is using the API rather than the client)?

Any input would be very much appreciated as going month by month (we need a month-level detail) through the dashboard is not very sustainable and potentially very error prone...

Best regards,

Michel

Le 27/11/2023 à 18:39, Michel Jouvin a écrit :

...

Hi,

We are using "openstack usage list" to follow resource usage by projects at our cloud. We recently discovered that ~1/3 of the resource consumed (mostly "big projects") are missing from the JSON/CSV produced where the Admin Overview menu in the Horizon dashboard displays everything. Is there a trivial reason for this?

We tried both with a pretty old openstackclient (5.8) and the last one after applying the patch in https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15...  to fix the problem with date conversion. The result is the same.

Thank you in advance for any help. Cheers,

Michel

Are you able to provide a code snippet? Might help, but also understand if you can not. My team has dozens of python scripts for operational use, reporting etc, on a number of clouds. We haven't observed any limits. Our scripts are using the "admin-rc" so assuming the user account you run the script as is scoped, it shouldn't have any issues. Cheers Michael On Tue, Nov 28, 2023 at 5:08 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:

Hi,

Thanks for the answer. But in fact I am using the /os-simple-tenant-usage API as an admin so I'd really expected it to list all projects/tenants like Horizon does... I'm surely missing something but not clear what...

Best regards,

Michel Sent from my mobile

Le 28 novembre 2023 19:27:33 smooney@redhat.com a écrit :

so the openstack client is calling the nova simple tenant usage api

...

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... maps to

https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant

the openstack client does not have a --all-tenants flag or simplar to call

https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for...

so the difference is the openstack client uses /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone with the member or reader role in a given project

the all tenats version is /os-simple-tenant-usage and that endpoint is admin only.

osc has only ever support the tenant specific version so if you wantted to supprot the admin version you would need to modify osc to add a --all-tenant option to the cli.

horizon presumably support the multi tenant view as an admin usign the /os-simple-tenant-usage endpoint.

On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote:

...

Hi,

I've been struggling today to understand the problem. I wrote a (Python) script that uses directly the API instead of going through the "openstack usage list". I had some hope of something related to the limit parameter but unfortunately it is not the case and the result using the API is the same as using the openstack client. Only horizon returns all the information.

Does somebody has any clue about what is done differently by Horizon compared to the openstack client (I assume it is using the API rather than the client)?

Any input would be very much appreciated as going month by month (we need a month-level detail) through the dashboard is not very sustainable and potentially very error prone...

Best regards,

Michel

Le 27/11/2023 à 18:39, Michel Jouvin a écrit :

...

Hi,

We are using "openstack usage list" to follow resource usage by projects at our cloud. We recently discovered that ~1/3 of the resource consumed (mostly "big projects") are missing from the JSON/CSV produced where the Admin Overview menu in the Horizon dashboard displays everything. Is there a trivial reason for this?

We tried both with a pretty old openstackclient (5.8) and the last one after applying the patch in

https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15...

to fix the problem with date conversion. The result is the same.

Thank you in advance for any help. Cheers,

Michel

Hi Michel, Long time no speak. We use this to generate accounting data and squirt it into InfluxDB: https://github.com/stfc/SCD-OpenStack-Utils/tree/master/OpenStack-accounting It might be a bit more heavy weight than you are after at the moment but we have found it scales really well. Regards Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc.atlassian.net/wiki/spaces/CLOUDKB/overview To raise a support ticket with the cloud team please email cloud-support@stfc.ac.uk<mailto:cloud-support@stfc.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ From: Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> Sent: Wednesday, November 29, 2023 8:56 AM To: Michael Knox <michael@knox.net.nz>; Openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: "openstack usage list" (and Nova API) doesn't return information about all projects Hi Michael, Sure there is nothing secret or fancy in this script, I attach it! It is typically run with something like python get_accounting_data.py --first-y 2022 --first-m 12 --last-m 11 --output openstack-accounting_data.csv [--debug] Adding --debug produces a very verbose (so better redirected to a file) but allow to get the raw data received from the API for each request, before all the data manipulations with Pandas. Not sure how you can assess if it works properly for you. Probably the easiest is to count the lines in the CSV file produced and compare with the lines returned by Overview menu in Dashboard admin section. In my case, I could easily identify some big projects present in the dashboard output and missing in the CSV file. After checking with --debug, I saw that these projects are present in the project list but have no entries in any month (where they are among our bigger consumers!). I really have no clue what I'm doing wrong. The API has not so many parameters so it's pretty easy to use if you respect the date format. One obvious difference between my script/openstack client and the dashboard is the Python version. Dashboard is run under Python 3.6 where my interactive environment is Python 3.11. But I cannot imagine any real influence on the API responses because of this... Thanks in advance for your help. Cheers, Michel Le 29/11/2023 à 00:23, Michael Knox a écrit : Are you able to provide a code snippet? Might help, but also understand if you can not. My team has dozens of python scripts for operational use, reporting etc, on a number of clouds. We haven't observed any limits. Our scripts are using the "admin-rc" so assuming the user account you run the script as is scoped, it shouldn't have any issues. Cheers Michael On Tue, Nov 28, 2023 at 5:08 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr<mailto:michel.jouvin@ijclab.in2p3.fr>> wrote: Hi, Thanks for the answer. But in fact I am using the /os-simple-tenant-usage API as an admin so I'd really expected it to list all projects/tenants like Horizon does... I'm surely missing something but not clear what... Best regards, Michel Sent from my mobile Le 28 novembre 2023 19:27:33 smooney@redhat.com<mailto:smooney@redhat.com> a écrit : so the openstack client is calling the nova simple tenant usage api https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... maps to https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant the openstack client does not have a --all-tenants flag or simplar to call https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for... so the difference is the openstack client uses /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone with the member or reader role in a given project the all tenats version is /os-simple-tenant-usage and that endpoint is admin only. osc has only ever support the tenant specific version so if you wantted to supprot the admin version you would need to modify osc to add a --all-tenant option to the cli. horizon presumably support the multi tenant view as an admin usign the /os-simple-tenant-usage endpoint. On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote: Hi, I've been struggling today to understand the problem. I wrote a (Python) script that uses directly the API instead of going through the "openstack usage list". I had some hope of something related to the limit parameter but unfortunately it is not the case and the result using the API is the same as using the openstack client. Only horizon returns all the information. Does somebody has any clue about what is done differently by Horizon compared to the openstack client (I assume it is using the API rather than the client)? Any input would be very much appreciated as going month by month (we need a month-level detail) through the dashboard is not very sustainable and potentially very error prone... Best regards, Michel Le 27/11/2023 à 18:39, Michel Jouvin a écrit : Hi, We are using "openstack usage list" to follow resource usage by projects at our cloud. We recently discovered that ~1/3 of the resource consumed (mostly "big projects") are missing from the JSON/CSV produced where the Admin Overview menu in the Horizon dashboard displays everything. Is there a trivial reason for this? We tried both with a pretty old openstackclient (5.8) and the last one after applying the patch in https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15... to fix the problem with date conversion. The result is the same. Thank you in advance for any help. Cheers, Michel

thanks Michel, are you running this script with an admin role? Cloud admin that is, not project admin. Cheers Michael On Wed, Nov 29, 2023 at 5:30 AM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:

To add one more information. I just ran the script on the Horizon server, in the same Python environment where I run the dashboard and using the Nova admin or internal interface rather than the public one. It makes absolutely no difference... I'm really wondering what could be done differently by the dashboard...

Best regards,

Michel Le 29/11/2023 à 09:56, Michel Jouvin a écrit :

Hi Michael,

Sure there is nothing secret or fancy in this script, I attach it! It is typically run with something like

python get_accounting_data.py --first-y 2022 --first-m 12 --last-m 11 --output openstack-accounting_data.csv [--debug]

Adding --debug produces a very verbose (so better redirected to a file) but allow to get the raw data received from the API for each request, before all the data manipulations with Pandas.

Not sure how you can assess if it works properly for you. Probably the easiest is to count the lines in the CSV file produced and compare with the lines returned by Overview menu in Dashboard admin section. In my case, I could easily identify some big projects present in the dashboard output and missing in the CSV file. After checking with --debug, I saw that these projects are present in the project list but have no entries in any month (where they are among our bigger consumers!).

I really have no clue what I'm doing wrong. The API has not so many parameters so it's pretty easy to use if you respect the date format. One obvious difference between my script/openstack client and the dashboard is the Python version. Dashboard is run under Python 3.6 where my interactive environment is Python 3.11. But I cannot imagine any real influence on the API responses because of this...

Thanks in advance for your help. Cheers,

Michel Le 29/11/2023 à 00:23, Michael Knox a écrit :

Are you able to provide a code snippet? Might help, but also understand if you can not.

My team has dozens of python scripts for operational use, reporting etc, on a number of clouds. We haven't observed any limits. Our scripts are using the "admin-rc" so assuming the user account you run the script as is scoped, it shouldn't have any issues.

Cheers Michael

On Tue, Nov 28, 2023 at 5:08 PM Michel Jouvin < michel.jouvin@ijclab.in2p3.fr> wrote:

...

Hi,

Thanks for the answer. But in fact I am using the /os-simple-tenant-usage API as an admin so I'd really expected it to list all projects/tenants like Horizon does... I'm surely missing something but not clear what...

Best regards,

Michel Sent from my mobile

Le 28 novembre 2023 19:27:33 smooney@redhat.com a écrit :

so the openstack client is calling the nova simple tenant usage api

...

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... maps to

https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant

the openstack client does not have a --all-tenants flag or simplar to call

https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for...

so the difference is the openstack client uses /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone with the member or reader role in a given project

the all tenats version is /os-simple-tenant-usage and that endpoint is admin only.

osc has only ever support the tenant specific version so if you wantted to supprot the admin version you would need to modify osc to add a --all-tenant option to the cli.

horizon presumably support the multi tenant view as an admin usign the /os-simple-tenant-usage endpoint.

On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote:

...

Hi,

I've been struggling today to understand the problem. I wrote a (Python) script that uses directly the API instead of going through the "openstack usage list". I had some hope of something related to the limit parameter but unfortunately it is not the case and the result using the API is the same as using the openstack client. Only horizon returns all the information.

Does somebody has any clue about what is done differently by Horizon compared to the openstack client (I assume it is using the API rather than the client)?

Any input would be very much appreciated as going month by month (we need a month-level detail) through the dashboard is not very sustainable and potentially very error prone...

Best regards,

Michel

Le 27/11/2023 à 18:39, Michel Jouvin a écrit :

...

Hi,

We are using "openstack usage list" to follow resource usage by projects at our cloud. We recently discovered that ~1/3 of the resource consumed (mostly "big projects") are missing from the JSON/CSV produced where the Admin Overview menu in the Horizon dashboard displays everything. Is there a trivial reason for this?

We tried both with a pretty old openstackclient (5.8) and the last one after applying the patch in

https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15...

to fix the problem with date conversion. The result is the same.

Thank you in advance for any help. Cheers,

Michel

Hi Michael, Thanks for the feedback. Yes, I'm using an admin account. I may have found a possible reason but still have to confirm it. When I look in nova-api.log at the requests issued by the dashboard, I see that it uses the "marker" attribute that is used for paging the responses... And the number of requests actually issued by the dashboard increases with the period duration. It may be that with the level of activity we have, we cannot get the full answer with only one request... Stay tuned... I'll post the solution if I find it! Cheers, Michel Le 30/11/2023 à 01:25, Michael Knox a écrit :

Hi Michel,

I can run the script as is and scrape all projects in one of our clouds. The account rc used was an admin. My suggestion would be to ensure the user you are using to run is scoped correctly. The API behaviour is as expected with admin.

Cheers Michael

On Wed, Nov 29, 2023 at 6:16 PM Michael Knox <michael@knox.net.nz> wrote:

thanks Michel, are you running this script with an admin role? Cloud admin that is, not project admin.

Cheers Michael

On Wed, Nov 29, 2023 at 5:30 AM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:

To add one more information. I just ran the script on the Horizon server, in the same Python environment where I run the dashboard and using the Nova admin or internal interface rather than the public one. It makes absolutely no difference... I'm really wondering what could be done differently by the dashboard...

Best regards,

Michel

Le 29/11/2023 à 09:56, Michel Jouvin a écrit :

...

Hi Michael,

Sure there is nothing secret or fancy in this script, I attach it! It is typically run with something like

python get_accounting_data.py --first-y 2022 --first-m 12 --last-m 11 --output openstack-accounting_data.csv [--debug]

Adding --debug produces a very verbose (so better redirected to a file) but allow to get the raw data received from the API for each request, before all the data manipulations with Pandas.

Not sure how you can assess if it works properly for you. Probably the easiest is to count the lines in the CSV file produced and compare with the lines returned by Overview menu in Dashboard admin section. In my case, I could easily identify some big projects present in the dashboard output and missing in the CSV file. After checking with --debug, I saw that these projects are present in the project list but have no entries in any month (where they are among our bigger consumers!).

I really have no clue what I'm doing wrong. The API has not so many parameters so it's pretty easy to use if you respect the date format. One obvious difference between my script/openstack client and the dashboard is the Python version. Dashboard is run under Python 3.6 where my interactive environment is Python 3.11. But I cannot imagine any real influence on the API responses because of this...

Thanks in advance for your help. Cheers,

Michel

Le 29/11/2023 à 00:23, Michael Knox a écrit :

...

Are you able to provide a code snippet? Might help, but also understand if you can not.

My team has dozens of python scripts for operational use, reporting etc, on a number of clouds. We haven't observed any limits. Our scripts are using the "admin-rc" so assuming the user account you run the script as is scoped, it shouldn't have any issues.

Cheers Michael

On Tue, Nov 28, 2023 at 5:08 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:

Hi,

Thanks for the answer. But in fact I am using the /os-simple-tenant-usage API as an admin so I'd really expected it to list all projects/tenants like Horizon does... I'm surely missing something but not clear what...

Best regards,

Michel Sent from my mobile

Le 28 novembre 2023 19:27:33 smooney@redhat.com a écrit :

...

so the openstack client is calling the nova simple tenant usage api

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... maps to https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant

the openstack client does not have a --all-tenants flag or simplar to call

https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for...

so the difference is the openstack client uses /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone with the member or reader role in a given project

the all tenats version is /os-simple-tenant-usage and that endpoint is admin only.

osc has only ever support the tenant specific version so if you wantted to supprot the admin version you would need to modify osc to add a --all-tenant option to the cli.

horizon presumably support the multi tenant view as an admin usign the /os-simple-tenant-usage endpoint.

On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote:

...

Hi,

I've been struggling today to understand the problem. I wrote a (Python) script that uses directly the API instead of going through the "openstack usage list". I had some hope of something related to the limit parameter but unfortunately it is not the case and the result using the API is the same as using the openstack client. Only horizon returns all the information.

Does somebody has any clue about what is done differently by Horizon compared to the openstack client (I assume it is using the API rather than the client)?

Any input would be very much appreciated as going month by month (we need a month-level detail) through the dashboard is not very sustainable and potentially very error prone...

Best regards,

Michel

Le 27/11/2023 à 18:39, Michel Jouvin a écrit :

...

Hi,

We are using "openstack usage list" to follow resource usage by projects at our cloud. We recently discovered that ~1/3 of the resource consumed (mostly "big projects") are missing from the JSON/CSV produced where the Admin Overview menu in the Horizon dashboard displays everything. Is there a trivial reason for this?

We tried both with a pretty old openstackclient (5.8) and the last one after applying the patch in https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15...

to fix the problem with date conversion. The result is the same.

Thank you in advance for any help. Cheers,

Michel

Hi, I manage to sort out the problem with response paging. It was stupid! I was using OpenStack-API-Version header as suggested by https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for... but running Ussuri, it seems that the microversion used by default is < 2.27 and it is necessary to use the "historical header" X-OpenStack-Nova-API-Version instead (as explained in the same page when microversion is < 2.27). My script is now perfectly working. A bit more explanation in the Nova API documentation on how to handle paging (I used the presence of "tenant-usages-links" in the response as an indication more data where available, not sure if it is the right indicator, getting the last seen instance ID for the last entry in the tenant-usages list). BTW, as for me, I think "openstack use list" is buggy as doesn't return the proper number of entries when you have a number of instances involved in the period larger than max_limit. Paging seems to occur only once, I didn't look at the code to understand what happens (as the API itself is working fine and easy to use once the described problems have been sorted out). Hope it helps others... Best regards, Michel Le 30/11/2023 à 11:05, Michel Jouvin a écrit :

Hi,

I made significant progresses! I identified the source of the problem and found a workaround. Still fighting to implement the proper solution.

The reason of my problem is response pagination as documented in Nova API documentation. I was able to identify this when I realized I could identify the requests done by the dashboard by looking at nova-api.log rather than digging into the code: I saw that the requests were including "marker=xxx" parameters and that the number of requests was proportional to the period length. For the time being, as a workaround, I was able to get the appropriate accounting data after setting Nova API max_limit=10000: I now have the same number of vCPUsxhours reported as the dashboard.

I tried to validate this with curl but so far I have not been able to get limit and marker parameters be taken into account, not clear why. Also I saw that the dashboard is not using limit parameter and I don't know how it decides another request is needed (may be just doing a new request until the last response returns nothing). One thing that could be better documented is that the limit applies to the number of instances processed, even if none are returned is you don't use detailed=1 (this is hidding the problem in fact, making more difficult to understand that there is a limit problem).

Cheers,

Michel

Le 29/11/2023 à 11:21, Michel Jouvin a écrit :

...

To add one more information. I just ran the script on the Horizon server, in the same Python environment where I run the dashboard and using the Nova admin or internal interface rather than the public one. It makes absolutely no difference... I'm really wondering what could be done differently by the dashboard...

Best regards,

Michel

Le 29/11/2023 à 09:56, Michel Jouvin a écrit :

...

Hi Michael,

Sure there is nothing secret or fancy in this script, I attach it! It is typically run with something like

python get_accounting_data.py --first-y 2022 --first-m 12 --last-m 11 --output openstack-accounting_data.csv [--debug]

Adding --debug produces a very verbose (so better redirected to a file) but allow to get the raw data received from the API for each request, before all the data manipulations with Pandas.

Not sure how you can assess if it works properly for you. Probably the easiest is to count the lines in the CSV file produced and compare with the lines returned by Overview menu in Dashboard admin section. In my case, I could easily identify some big projects present in the dashboard output and missing in the CSV file. After checking with --debug, I saw that these projects are present in the project list but have no entries in any month (where they are among our bigger consumers!).

I really have no clue what I'm doing wrong. The API has not so many parameters so it's pretty easy to use if you respect the date format. One obvious difference between my script/openstack client and the dashboard is the Python version. Dashboard is run under Python 3.6 where my interactive environment is Python 3.11. But I cannot imagine any real influence on the API responses because of this...

Thanks in advance for your help. Cheers,

Michel

Le 29/11/2023 à 00:23, Michael Knox a écrit :

...

Are you able to provide a code snippet? Might help, but also understand if you can not.

My team has dozens of python scripts for operational use, reporting etc, on a number of clouds. We haven't observed any limits. Our scripts are using the "admin-rc" so assuming the user account you run the script as is scoped, it shouldn't have any issues.

Cheers Michael

On Tue, Nov 28, 2023 at 5:08 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:

Hi,

Thanks for the answer. But in fact I am using the /os-simple-tenant-usage API as an admin so I'd really expected it to list all projects/tenants like Horizon does... I'm surely missing something but not clear what...

Best regards,

Michel Sent from my mobile

Le 28 novembre 2023 19:27:33 smooney@redhat.com a écrit :

...

so the openstack client is calling the nova simple tenant usage api

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... maps to https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant

the openstack client does not have a --all-tenants flag or simplar to call

https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for...

so the difference is the openstack client uses /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone with the member or reader role in a given project

the all tenats version is /os-simple-tenant-usage and that endpoint is admin only.

osc has only ever support the tenant specific version so if you wantted to supprot the admin version you would need to modify osc to add a --all-tenant option to the cli.

horizon presumably support the multi tenant view as an admin usign the /os-simple-tenant-usage endpoint.

On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote:

...

Hi,

I've been struggling today to understand the problem. I wrote a (Python) script that uses directly the API instead of going through the "openstack usage list". I had some hope of something related to the limit parameter but unfortunately it is not the case and the result using the API is the same as using the openstack client. Only horizon returns all the information.

Does somebody has any clue about what is done differently by Horizon compared to the openstack client (I assume it is using the API rather than the client)?

Any input would be very much appreciated as going month by month (we need a month-level detail) through the dashboard is not very sustainable and potentially very error prone...

Best regards,

Michel

Le 27/11/2023 à 18:39, Michel Jouvin a écrit : > Hi, > > We are using "openstack usage list" to follow resource usage by > projects at our cloud. We recently discovered that ~1/3 of the > resource consumed (mostly "big projects") are missing from the > JSON/CSV produced where the Admin Overview menu in the Horizon > dashboard displays everything. Is there a trivial reason for > this? > > We tried both with a pretty old openstackclient (5.8) and > the last one > after applying the patch in > https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15... > > to fix the problem with date conversion. The result is the same. > > Thank you in advance for any help. Cheers, > > Michel >

Thanks for your detailed answer. I'll try to contribute something to the docs, probably not immediately. I tend to think that despite the limits you described of the Nova usage API, if it is exposed in some way by OSC, it should be done properly. And for me, it is not the case currenty and should be fixed at some point to implement the same paging as Horizon (it is not much more than adding a while loop around the request... In my script it is < 10 lines). Best regards, Michel Le 30/11/2023 à 15:33, smooney@redhat.com a écrit :

...

Hi,

I manage to sort out the problem with response paging. It was stupid! I was using OpenStack-API-Version header as suggested by https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for... but running Ussuri, it seems that the microversion used by default is < 2.27 and it is necessary to use the "historical header" X-OpenStack-Nova-API-Version instead (as explained in the same page when microversion is < 2.27). My script is now perfectly working. A bit more explanation in the Nova API documentation on how to handle paging (I used the presence of "tenant-usages-links" in the response as an indication more data where available, not sure if it is the right indicator, getting the last seen instance ID for the last entry in the tenant-usages list).

BTW, as for me, I think "openstack use list" is buggy as doesn't return the proper number of entries when you have a number of instances involved in the period larger than max_limit. Paging seems to occur only once, I didn't look at the code to understand what happens (as the API itself is working fine and easy to use once the described problems have been sorted out). so i belive the way marker and paging works with the cli is you are intened to update

On Thu, 2023-11-30 at 14:47 +0100, Michel Jouvin wrote: the marker and make addtional calls passing the last server form the previous respocne as the marker. i dont recall the detail off the top of my head but that is basically what horizon was doing with the multipel calls.

there is no auto paging in osc to my knowlage.

this also explains why it was "working" fo melanie and most devs we typiclaly only test thigs kind fo thing in devstack ro a simialr small dev cloud wehre we never have more then a couple of instnace so never actully hit the paging code.

if you feel like you can impvoed the docs to cover these topics contirbutiosn are more then welcome.

one meta coment. the simple tenant usage api is one of the apis provided by nova for legacy reasons that we would remove if we could. billing is not in the socpe of the nova project to do https://github.com/openstack/nova/blob/master/doc/source/contributor/project... we are very unlikely to remove this api since it would violate https://github.com/openstack/nova/blob/master/doc/source/contributor/project... but we also done want ot extend it i nthe futruere.

there are better way to do billing and tack usage/quota in general and we try to discourage building on top of the simple tenant usage api because it can be very very slow it also will only provide a accurate view for relitively short term time periods

basicaly as instnace are deleted and you run archive deleted rows and purge archived rows you are removign the data that was trackign those deleted instances.

if you are building your own billing system then using the notification system like cloudkitty does may be a better option.https://wiki.openstack.org/wiki/CloudKitty https://opendev.org/openstack/cloudkitty

the notificaiotn message bus is best effort and we make no delivery guarantees. all notification are fire and forget. so that is also not a perfect approch. https://github.com/openstack/nova/blob/master/doc/source/admin/notifications...

the main isuse with the simple tenant usage api is it does not scale as clouds and workloads get larger and its not really fixable in a reasonable way. we would like horizon to swap to using placment for usage information in the future https://docs.openstack.org/api-ref/placement/#list-usages and we are plannign to add some more fucntionaltiy for capsity/usage on other granularity (agggrates or mulit tenant) for operatos to be abel to better plan. but the main delta with placement is its usage apis will only ever be point in time. ie. the current useage at the time you request it. it will not suport ranges or historical usage. so it will not really adress the billing usecase as that is alos not in placements project scope.

...

Hope it helps others... Best regards,

Michel

Le 30/11/2023 à 11:05, Michel Jouvin a écrit :

...

Hi,

I made significant progresses! I identified the source of the problem and found a workaround. Still fighting to implement the proper solution.

The reason of my problem is response pagination as documented in Nova API documentation. I was able to identify this when I realized I could identify the requests done by the dashboard by looking at nova-api.log rather than digging into the code: I saw that the requests were including "marker=xxx" parameters and that the number of requests was proportional to the period length. For the time being, as a workaround, I was able to get the appropriate accounting data after setting Nova API max_limit=10000: I now have the same number of vCPUsxhours reported as the dashboard.

I tried to validate this with curl but so far I have not been able to get limit and marker parameters be taken into account, not clear why. Also I saw that the dashboard is not using limit parameter and I don't know how it decides another request is needed (may be just doing a new request until the last response returns nothing). One thing that could be better documented is that the limit applies to the number of instances processed, even if none are returned is you don't use detailed=1 (this is hidding the problem in fact, making more difficult to understand that there is a limit problem).

Cheers,

Michel

Le 29/11/2023 à 11:21, Michel Jouvin a écrit :

...

To add one more information. I just ran the script on the Horizon server, in the same Python environment where I run the dashboard and using the Nova admin or internal interface rather than the public one. It makes absolutely no difference... I'm really wondering what could be done differently by the dashboard...

Best regards,

Michel

Le 29/11/2023 à 09:56, Michel Jouvin a écrit :

...

Hi Michael,

Sure there is nothing secret or fancy in this script, I attach it! It is typically run with something like

python get_accounting_data.py --first-y 2022 --first-m 12 --last-m 11 --output openstack-accounting_data.csv [--debug]

Adding --debug produces a very verbose (so better redirected to a file) but allow to get the raw data received from the API for each request, before all the data manipulations with Pandas.

Not sure how you can assess if it works properly for you. Probably the easiest is to count the lines in the CSV file produced and compare with the lines returned by Overview menu in Dashboard admin section. In my case, I could easily identify some big projects present in the dashboard output and missing in the CSV file. After checking with --debug, I saw that these projects are present in the project list but have no entries in any month (where they are among our bigger consumers!).

I really have no clue what I'm doing wrong. The API has not so many parameters so it's pretty easy to use if you respect the date format. One obvious difference between my script/openstack client and the dashboard is the Python version. Dashboard is run under Python 3.6 where my interactive environment is Python 3.11. But I cannot imagine any real influence on the API responses because of this...

Thanks in advance for your help. Cheers,

Michel

Le 29/11/2023 à 00:23, Michael Knox a écrit :

...

Are you able to provide a code snippet? Might help, but also understand if you can not.

My team has dozens of python scripts for operational use, reporting etc, on a number of clouds. We haven't observed any limits. Our scripts are using the "admin-rc" so assuming the user account you run the script as is scoped, it shouldn't have any issues.

Cheers Michael

On Tue, Nov 28, 2023 at 5:08 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:

    Hi,

    Thanks for the answer. But in fact I am using the     /os-simple-tenant-usage API as an admin so I'd really expected     it to list all projects/tenants like Horizon does... I'm surely     missing something but not clear what...

    Best regards,

    Michel     Sent from my mobile

    Le 28 novembre 2023 19:27:33 smooney@redhat.com a écrit :

>     so the openstack client is calling the nova simple tenant >     usage api > >     https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... >     maps to >     https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant > >     the openstack client does not have a --all-tenants flag or >     simplar to call > >     https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for... > >     so the difference is the openstack client uses >     /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone >     with the member or reader role in a given project > >     the all tenats version is /os-simple-tenant-usage >     and that endpoint is admin only. > >     osc has only ever support the tenant specific version so if >     you wantted to supprot the admin version >     you would need to modify osc to add a --all-tenant option to >     the cli. > >     horizon presumably support the multi tenant view as an admin >     usign the /os-simple-tenant-usage endpoint. > > > >     On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote: >>     Hi, >> >>     I've been struggling today to understand the problem. I wrote >>     a (Python) >>     script that uses directly the API instead of going through the >>     "openstack usage list". I had some hope of something related >>     to the >>     limit parameter but unfortunately it is not the case and the >>     result >>     using the API is the same as using the openstack client. Only >>     horizon >>     returns all the information. >> >>     Does somebody has any clue about what is done differently by >>     Horizon >>     compared to the openstack client (I assume it is using the >>     API rather >>     than the client)? >> >>     Any input would be very much appreciated as going month by >>     month (we >>     need a month-level detail) through the dashboard is not very >>     sustainable >>     and potentially very error prone... >> >>     Best regards, >> >>     Michel >> >>     Le 27/11/2023 à 18:39, Michel Jouvin a écrit : >>>     Hi, >>> >>>     We are using "openstack usage list" to follow resource usage by >>>     projects at our cloud. We recently discovered that ~1/3 of the >>>     resource consumed (mostly "big projects") are missing from the >>>     JSON/CSV produced where the Admin Overview menu in the Horizon >>>     dashboard displays everything. Is there a trivial reason for >>>     this? >>> >>>     We tried both with a pretty old openstackclient (5.8) and >>>     the last one >>>     after applying the patch in >>>     https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15... >>> >>>     to fix the problem with date conversion. The result is the same. >>> >>>     Thank you in advance for any help. Cheers, >>> >>>     Michel >>>