R: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi
Christian Rohmann
christian.rohmann at inovex.de
Wed Jun 22 08:18:26 UTC 2022
Hey Molka, Rafael, all,
On 08/03/2022 21:02, Molka Gharbaoui wrote:
> I removed manually "ceilometer-low" and "ceilometer-high" from
> archive-policy list (both were creating a conflict) however nothing
> changes:
> If I execute ceilometer-upgrade/ceilometer-polling nothing "new" happens.
> When I execute ceilometer-agent-notification,
> ceilometer-low/ceilometer-high are readded to the archive-policy list
> and the error reappears.
> I guess I should not execute those commands manually but I cannot find
> a way to push the metrics to the database.
I just ran into this issue when playing with Ceilometer + Gnocchi on a
devstack running stable/yoga and tried the same things. To went through
the cascade of calls happening in the code. First there is ceilometer
"ensuring" the archive policy exists
https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L269
This is done via python-gnocchiclient and the create method at
https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/v1/archive_policy.py#L35.
which then sends a POST to the create REST API endpoint of Gnocci at
https://github.com/gnocchixyz/gnocchi/blob/f124cf76479720d3b9caf439241da12b2974ac90/gnocchi/rest/api.py#L302.
The mentioned "ensures_archives_policies" method then checks for the
"ArchivePolicyAlreadyExists" exception
(https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L274)
and apparently this is the issue here - this exception is NOT thrown,
but simply a 409 Conflict error.
Exceptions are created and then enriched via the
https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L217
method and in our case it should be
https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L161
.
So I first thought the string must have been changed somewhere somehow
for it to not match anymore, but all that code is unchanged for years.
But, when looking at a trace between the apache2 and the unix socket
exposing the gnocci-api I saw something strange:
--- SENT ---
.7....proxy-sendchunked..1..APACHE_RUN_USER..ubuntu..APACHE_RUN_GROUP..ubuntu
.PATH_INFO../v1/archive_policy/ .HTTP_HOST
.192.168.0.100..HTTP_USER_AGENTW.ceilometer-agent-notification
keystoneauth1/4.5.0 python-requests/2.27.1
CPython/3.8.10..HTTP_ACCEPT_ENCODING
.gzip, deflate..HTTP_ACCEPT..application/json, */*..HTTP_CONNECTION
.keep-alive..CONTENT_TYPE..application/json..HTTP_X_AUTH_TOKEN..gAAAAABiscoSYYjDWLVJzL8V12knrSmvI5aRYFTH4cZyQ0HKPBrheWqbh-6PS6Cf56kNTuTw5hwhpYa6UNEXkA_aIYj2xk6hTchgFaBOSrIHq36opuo46ktJCXcsEBQnus1_IZnWwuy8RBlzso6LGKeo37dzT_zloUYj_XhaMRDlgUifeAgUESQ..CONTENT_LENGTH..245..PATHF./usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin..SERVER_SIGNATUREJ.<address>Apache/2.4.41
(Ubuntu) Server at 192.168.0.100 Port 80</address>
..SERVER_SOFTWARE..Apache/2.4.41 (Ubuntu)..SERVER_NAME
.192.168.0.100..SERVER_ADDR
.192.168.0.100..SERVER_PORT..80..REMOTE_ADDR
.192.168.0.100
.DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..REQUEST_SCHEME..http..CONTEXT_PREFIX....CONTEXT_DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..SERVER_ADMIN..[no
address
given]..SCRIPT_FILENAME6.proxy:uwsgi://uwsgi-uds-gnocchi-api/v1/archive_policy/..REMOTE_PORT..42324..GATEWAY_INTERFACE..CGI/1.1..SERVER_PROTOCOL..HTTP/1.1..REQUEST_METHOD..POST..QUERY_STRING....REQUEST_URI../metric/v1/archive_policy/..SCRIPT_NAME../metric{"name":"ceilometer-high-rate","aggregation_methods":["mean","rate:mean"],"back_window":0,"definition":[{"granularity":"1
second","timespan":"1 hour"},{"granularity":"1 minute","timespan":"1
day"},{"granularity":"1 hour","timespan":"365 days"}]}
--- SENT ---
--- RECEIVED ---
HTTP/1.1 409 Conflict
Content-Length: 228
Content-Type: text/html; charset=UTF-8
Connection: close
<html>
<head>
<title>409 Conflict</title>
</head>
<body>
<h1>409 Conflict</h1>
There was a conflict when trying to complete your request.<br /><br />
Archive policy ceilometer-high-rate already exists
</body>
</html>
--- RECEIVED ---
When though a content type of "application/json" was sent, the
gnocchi-api returned "Content-Type: text/html; charset=UTF-8" and a
"human readable" error instead of the expected JSON. And that is why the
exception is not properly casted and we end up in this archive_policy
creation loop.
Why the response is html I was not yet able to find out.
Regards
Christian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220622/ffbf5fa7/attachment-0001.htm>
More information about the openstack-discuss
mailing list