R: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi
This is the outcome of the gnocchi archive-policy list. All are created "automatically" during the Devstack configuration
stack@ubuntu-VirtualBox:~/devstack$ gnocchi archive-policy list +----------------------+-------------+-----------------------------------------------------------------------+---------------------------------+ | name | back_window | definition | aggregation_methods | +----------------------+-------------+-----------------------------------------------------------------------+---------------------------------+ | bool | 3600 | - timespan: 365 days, 0:00:00, granularity: 0:00:01, points: 31536000 | last | | ceilometer-high | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | mean | | | | - timespan: 1 day, 0:00:00, granularity: 0:01:00, points: 1440 | | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | | ceilometer-high-rate | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | rate:mean, mean | | | | - timespan: 1 day, 0:00:00, granularity: 0:01:00, points: 1440 | | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | | ceilometer-low | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | mean | | ceilometer-low-rate | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | rate:mean, mean | | high | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | count, min, mean, max, sum, std | | | | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | | low | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | count, min, mean, max, sum, std | | medium | 0 | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | count, min, mean, max, sum, std | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | +----------------------+-------------+-----------------------------------------------------------------------+---------------------------------+
Shall I delete them??
can you try it? Not all of them, but the "ceilometer-low" that is causing the conflict
On Tue, Mar 8, 2022 at 1:49 PM Molka Gharbaoui < Molka.Gharbaoui@santannapisa.it> wrote:
This is the outcome of the gnocchi archive-policy list. All are created "automatically" during the Devstack configuration
stack@ubuntu-VirtualBox:~/devstack$ gnocchi archive-policy list +----------------------+-------------+---------------------- -------------------------------------------------+---------- -----------------------+ | name | back_window | definition | aggregation_methods | +----------------------+-------------+---------------------- -------------------------------------------------+---------- -----------------------+ | bool | 3600 | - timespan: 365 days, 0:00:00, granularity: 0:00:01, points: 31536000 | last | | ceilometer-high | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | mean | | | | - timespan: 1 day, 0:00:00, granularity: 0:01:00, points: 1440 | | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | | ceilometer-high-rate | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | rate:mean, mean | | | | - timespan: 1 day, 0:00:00, granularity: 0:01:00, points: 1440 | | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | | ceilometer-low | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | mean | | ceilometer-low-rate | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | rate:mean, mean | | high | 0 | - timespan: 1:00:00, granularity: 0:00:01, points: 3600 | count, min, mean, max, sum, std | | | | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | | low | 0 | - timespan: 30 days, 0:00:00, granularity: 0:05:00, points: 8640 | count, min, mean, max, sum, std | | medium | 0 | - timespan: 7 days, 0:00:00, granularity: 0:01:00, points: 10080 | count, min, mean, max, sum, std | | | | - timespan: 365 days, 0:00:00, granularity: 1:00:00, points: 8760 | | +----------------------+-------------+---------------------- -------------------------------------------------+---------- -----------------------+
Shall I delete them??
I removed manually "ceilometer-low" and "ceilometer-high" from archive-policy list (both were creating a conflict) however nothing changes:
If I execute ceilometer-upgrade/ceilometer-polling nothing "new" happens.
When I execute ceilometer-agent-notification, ceilometer-low/ceilometer-high are readded to the archive-policy list and the error reappears.
I guess I should not execute those commands manually but I cannot find a way to push the metrics to the database.
Hey Molka, Rafael, all,
On 08/03/2022 21:02, Molka Gharbaoui wrote:
I removed manually "ceilometer-low" and "ceilometer-high" from archive-policy list (both were creating a conflict) however nothing changes: If I execute ceilometer-upgrade/ceilometer-polling nothing "new" happens. When I execute ceilometer-agent-notification, ceilometer-low/ceilometer-high are readded to the archive-policy list and the error reappears. I guess I should not execute those commands manually but I cannot find a way to push the metrics to the database.
I just ran into this issue when playing with Ceilometer + Gnocchi on a devstack running stable/yoga and tried the same things. To went through the cascade of calls happening in the code. First there is ceilometer "ensuring" the archive policy exists https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3...
This is done via python-gnocchiclient and the create method at https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f59622....
which then sends a POST to the create REST API endpoint of Gnocci at https://github.com/gnocchixyz/gnocchi/blob/f124cf76479720d3b9caf439241da12b2....
The mentioned "ensures_archives_policies" method then checks for the "ArchivePolicyAlreadyExists" exception (https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3...) and apparently this is the issue here - this exception is NOT thrown, but simply a 409 Conflict error.
Exceptions are created and then enriched via the https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f59622... method and in our case it should be https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f59622... .
So I first thought the string must have been changed somewhere somehow for it to not match anymore, but all that code is unchanged for years.
But, when looking at a trace between the apache2 and the unix socket exposing the gnocci-api I saw something strange:
--- SENT ---
.7....proxy-sendchunked..1..APACHE_RUN_USER..ubuntu..APACHE_RUN_GROUP..ubuntu .PATH_INFO../v1/archive_policy/ .HTTP_HOST .192.168.0.100..HTTP_USER_AGENTW.ceilometer-agent-notification keystoneauth1/4.5.0 python-requests/2.27.1 CPython/3.8.10..HTTP_ACCEPT_ENCODING .gzip, deflate..HTTP_ACCEPT..application/json, */*..HTTP_CONNECTION .keep-alive..CONTENT_TYPE..application/json..HTTP_X_AUTH_TOKEN..gAAAAABiscoSYYjDWLVJzL8V12knrSmvI5aRYFTH4cZyQ0HKPBrheWqbh-6PS6Cf56kNTuTw5hwhpYa6UNEXkA_aIYj2xk6hTchgFaBOSrIHq36opuo46ktJCXcsEBQnus1_IZnWwuy8RBlzso6LGKeo37dzT_zloUYj_XhaMRDlgUifeAgUESQ..CONTENT_LENGTH..245..PATHF./usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin..SERVER_SIGNATUREJ.<address>Apache/2.4.41 (Ubuntu) Server at 192.168.0.100 Port 80</address> ..SERVER_SOFTWARE..Apache/2.4.41 (Ubuntu)..SERVER_NAME .192.168.0.100..SERVER_ADDR .192.168.0.100..SERVER_PORT..80..REMOTE_ADDR .192.168.0.100 .DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..REQUEST_SCHEME..http..CONTEXT_PREFIX....CONTEXT_DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..SERVER_ADMIN..[no address given]..SCRIPT_FILENAME6.proxy:uwsgi://uwsgi-uds-gnocchi-api/v1/archive_policy/..REMOTE_PORT..42324..GATEWAY_INTERFACE..CGI/1.1..SERVER_PROTOCOL..HTTP/1.1..REQUEST_METHOD..POST..QUERY_STRING....REQUEST_URI../metric/v1/archive_policy/..SCRIPT_NAME../metric{"name":"ceilometer-high-rate","aggregation_methods":["mean","rate:mean"],"back_window":0,"definition":[{"granularity":"1 second","timespan":"1 hour"},{"granularity":"1 minute","timespan":"1 day"},{"granularity":"1 hour","timespan":"365 days"}]} --- SENT ---
--- RECEIVED --- HTTP/1.1 409 Conflict Content-Length: 228 Content-Type: text/html; charset=UTF-8 Connection: close
<html> <head> <title>409 Conflict</title> </head> <body> <h1>409 Conflict</h1> There was a conflict when trying to complete your request.<br /><br /> Archive policy ceilometer-high-rate already exists
</body> </html>
--- RECEIVED ---
When though a content type of "application/json" was sent, the gnocchi-api returned "Content-Type: text/html; charset=UTF-8" and a "human readable" error instead of the expected JSON. And that is why the exception is not properly casted and we end up in this archive_policy creation loop.
Why the response is html I was not yet able to find out.
Regards
Christian
On 22/06/2022 10:18, Christian Rohmann wrote:
a content type of "application/json" was sent, the gnocchi-api returned "Content-Type: text/html; charset=UTF-8" and a "human readable" error instead of the expected JSON. And that is why the exception is not properly casted and we end up in this archive_policy creation loop.
Why the response is html I was not yet able to find out.
I now opened an issue at https://github.com/gnocchixyz/gnocchi/issues/1261, but am wondering if this is maybe related to https://github.com/gnocchixyz/python-gnocchiclient/pull/110 ?
Regards
Christian
participants (3)
-
Christian Rohmann
-
Molka Gharbaoui
-
Rafael Weingärtner