Hey Molka, Rafael, all,

On 08/03/2022 21:02, Molka Gharbaoui wrote:

I removed manually "ceilometer-low" and "ceilometer-high" from archive-policy list (both were creating a conflict) however nothing changes:

If I execute ceilometer-upgrade/ceilometer-polling nothing "new" happens.

When I execute ceilometer-agent-notification, ceilometer-low/ceilometer-high are readded to the archive-policy list and the error reappears.

I guess I should not execute those commands manually but I cannot find a way to push the metrics to the database.

I just ran into this issue when playing with Ceilometer + Gnocchi on a devstack running stable/yoga and tried the same things. To went through the cascade of calls happening in the code. First there is ceilometer "ensuring" the archive policy exists https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L269

This is done via python-gnocchiclient and the create method at https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/v1/archive_policy.py#L35.

which then sends a POST to the create REST API endpoint of Gnocci at https://github.com/gnocchixyz/gnocchi/blob/f124cf76479720d3b9caf439241da12b2974ac90/gnocchi/rest/api.py#L302.

The mentioned "ensures_archives_policies" method then checks for the "ArchivePolicyAlreadyExists" exception (https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L274) and apparently this is the issue here - this exception is NOT thrown, but simply a 409 Conflict error.

Exceptions are created and then enriched via the https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L217 method and in our case it should be https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L161 .

So I first thought the string must have been changed somewhere somehow for it to not match anymore, but all that code is unchanged for years.

But, when looking at a trace between the apache2 and the unix socket exposing the gnocci-api I saw something strange:

--- SENT ---

.7....proxy-sendchunked..1..APACHE_RUN_USER..ubuntu..APACHE_RUN_GROUP..ubuntu .PATH_INFO../v1/archive_policy/ .HTTP_HOST
.192.168.0.100..HTTP_USER_AGENTW.ceilometer-agent-notification keystoneauth1/4.5.0 python-requests/2.27.1 CPython/3.8.10..HTTP_ACCEPT_ENCODING
.gzip, deflate..HTTP_ACCEPT..application/json, */*..HTTP_CONNECTION
.keep-alive..CONTENT_TYPE..application/json..HTTP_X_AUTH_TOKEN..gAAAAABiscoSYYjDWLVJzL8V12knrSmvI5aRYFTH4cZyQ0HKPBrheWqbh-6PS6Cf56kNTuTw5hwhpYa6UNEXkA_aIYj2xk6hTchgFaBOSrIHq36opuo46ktJCXcsEBQnus1_IZnWwuy8RBlzso6LGKeo37dzT_zloUYj_XhaMRDlgUifeAgUESQ..CONTENT_LENGTH..245..PATHF./usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin..SERVER_SIGNATUREJ.<address>Apache/2.4.41 (Ubuntu) Server at 192.168.0.100 Port 80</address>
..SERVER_SOFTWARE..Apache/2.4.41 (Ubuntu)..SERVER_NAME
.192.168.0.100..SERVER_ADDR
.192.168.0.100..SERVER_PORT..80..REMOTE_ADDR
.192.168.0.100
.DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..REQUEST_SCHEME..http..CONTEXT_PREFIX....CONTEXT_DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..SERVER_ADMIN..[no address given]..SCRIPT_FILENAME6.proxy:uwsgi://uwsgi-uds-gnocchi-api/v1/archive_policy/..REMOTE_PORT..42324..GATEWAY_INTERFACE..CGI/1.1..SERVER_PROTOCOL..HTTP/1.1..REQUEST_METHOD..POST..QUERY_STRING....REQUEST_URI../metric/v1/archive_policy/..SCRIPT_NAME../metric{"name":"ceilometer-high-rate","aggregation_methods":["mean","rate:mean"],"back_window":0,"definition":[{"granularity":"1 second","timespan":"1 hour"},{"granularity":"1 minute","timespan":"1 day"},{"granularity":"1 hour","timespan":"365 days"}]}
--- SENT ---

--- RECEIVED ---
HTTP/1.1 409 Conflict
Content-Length: 228
Content-Type: text/html; charset=UTF-8
Connection: close

<html>
<head>
<title>409 Conflict</title>
</head>
<body>
<h1>409 Conflict</h1>
There was a conflict when trying to complete your request.<br /><br />
Archive policy ceilometer-high-rate already exists

</body>
</html>

--- RECEIVED ---

When though a content type of "application/json" was sent, the gnocchi-api returned "Content-Type: text/html; charset=UTF-8" and a "human readable" error instead of the expected JSON. And that is why the exception is not properly casted and we end up in this archive_policy creation loop.

Why the response is html I was not yet able to find out.

Regards

Christian