Hi Michael, Thanks for the feedback. Yes, I'm using an admin account. I may have found a possible reason but still have to confirm it. When I look in nova-api.log at the requests issued by the dashboard, I see that it uses the "marker" attribute that is used for paging the responses... And the number of requests actually issued by the dashboard increases with the period duration. It may be that with the level of activity we have, we cannot get the full answer with only one request... Stay tuned... I'll post the solution if I find it! Cheers, Michel Le 30/11/2023 à 01:25, Michael Knox a écrit :
Hi Michel,
I can run the script as is and scrape all projects in one of our clouds. The account rc used was an admin. My suggestion would be to ensure the user you are using to run is scoped correctly. The API behaviour is as expected with admin.
Cheers Michael
On Wed, Nov 29, 2023 at 6:16 PM Michael Knox <michael@knox.net.nz> wrote:
thanks Michel, are you running this script with an admin role? Cloud admin that is, not project admin.
Cheers Michael
On Wed, Nov 29, 2023 at 5:30 AM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
To add one more information. I just ran the script on the Horizon server, in the same Python environment where I run the dashboard and using the Nova admin or internal interface rather than the public one. It makes absolutely no difference... I'm really wondering what could be done differently by the dashboard...
Best regards,
Michel
Le 29/11/2023 à 09:56, Michel Jouvin a écrit :
Hi Michael,
Sure there is nothing secret or fancy in this script, I attach it! It is typically run with something like
python get_accounting_data.py --first-y 2022 --first-m 12 --last-m 11 --output openstack-accounting_data.csv [--debug]
Adding --debug produces a very verbose (so better redirected to a file) but allow to get the raw data received from the API for each request, before all the data manipulations with Pandas.
Not sure how you can assess if it works properly for you. Probably the easiest is to count the lines in the CSV file produced and compare with the lines returned by Overview menu in Dashboard admin section. In my case, I could easily identify some big projects present in the dashboard output and missing in the CSV file. After checking with --debug, I saw that these projects are present in the project list but have no entries in any month (where they are among our bigger consumers!).
I really have no clue what I'm doing wrong. The API has not so many parameters so it's pretty easy to use if you respect the date format. One obvious difference between my script/openstack client and the dashboard is the Python version. Dashboard is run under Python 3.6 where my interactive environment is Python 3.11. But I cannot imagine any real influence on the API responses because of this...
Thanks in advance for your help. Cheers,
Michel
Le 29/11/2023 à 00:23, Michael Knox a écrit :
Are you able to provide a code snippet? Might help, but also understand if you can not.
My team has dozens of python scripts for operational use, reporting etc, on a number of clouds. We haven't observed any limits. Our scripts are using the "admin-rc" so assuming the user account you run the script as is scoped, it shouldn't have any issues.
Cheers Michael
On Tue, Nov 28, 2023 at 5:08 PM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Hi,
Thanks for the answer. But in fact I am using the /os-simple-tenant-usage API as an admin so I'd really expected it to list all projects/tenants like Horizon does... I'm surely missing something but not clear what...
Best regards,
Michel Sent from my mobile
Le 28 novembre 2023 19:27:33 smooney@redhat.com a écrit :
so the openstack client is calling the nova simple tenant usage api
https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... maps to https://docs.openstack.org/api-ref/compute/#show-usage-statistics-for-tenant
the openstack client does not have a --all-tenants flag or simplar to call
https://docs.openstack.org/api-ref/compute/#list-tenant-usage-statistics-for...
so the difference is the openstack client uses /os-simple-tenant-usage/{tenant_id} which is aviabel to anyone with the member or reader role in a given project
the all tenats version is /os-simple-tenant-usage and that endpoint is admin only.
osc has only ever support the tenant specific version so if you wantted to supprot the admin version you would need to modify osc to add a --all-tenant option to the cli.
horizon presumably support the multi tenant view as an admin usign the /os-simple-tenant-usage endpoint.
On Tue, 2023-11-28 at 18:47 +0100, Michel Jouvin wrote:
Hi,
I've been struggling today to understand the problem. I wrote a (Python) script that uses directly the API instead of going through the "openstack usage list". I had some hope of something related to the limit parameter but unfortunately it is not the case and the result using the API is the same as using the openstack client. Only horizon returns all the information.
Does somebody has any clue about what is done differently by Horizon compared to the openstack client (I assume it is using the API rather than the client)?
Any input would be very much appreciated as going month by month (we need a month-level detail) through the dashboard is not very sustainable and potentially very error prone...
Best regards,
Michel
Le 27/11/2023 à 18:39, Michel Jouvin a écrit :
Hi,
We are using "openstack usage list" to follow resource usage by projects at our cloud. We recently discovered that ~1/3 of the resource consumed (mostly "big projects") are missing from the JSON/CSV produced where the Admin Overview menu in the Horizon dashboard displays everything. Is there a trivial reason for this?
We tried both with a pretty old openstackclient (5.8) and the last one after applying the patch in https://github.com/openstack/python-openstackclient/commit/30462f57d48162d15...
to fix the problem with date conversion. The result is the same.
Thank you in advance for any help. Cheers,
Michel