<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hey Molka, Rafael, all,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 08/03/2022 21:02, Molka Gharbaoui
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:VE1PR03MB5550E97D3C346F4AE743B3D7EC099@VE1PR03MB5550.eurprd03.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
I removed manually "ceilometer-low" and "ceilometer-high" from
archive-policy list (both were creating a conflict) however
nothing changes:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
If I execute ceilometer-upgrade/ceilometer-polling nothing "new"
happens.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
When I execute ceilometer-agent-notification,
ceilometer-low/ceilometer-high are readded to the archive-policy
list and the error reappears.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
I guess I should not execute those commands manually but I
cannot find a way to push the metrics to the database.<br>
</div>
</blockquote>
<p>
I just ran into this issue when playing with Ceilometer + Gnocchi on a devstack running stable/yoga and tried the same things.
To went through the cascade of calls happening in the code. First there is ceilometer "ensuring" the archive policy exists <a class="moz-txt-link-freetext" href="https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L269">https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L269</a>
</p>
<p>This is done via python-gnocchiclient and the create method at <a class="moz-txt-link-freetext" href="https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/v1/archive_policy.py#L35">https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/v1/archive_policy.py#L35</a>.
</p>
<p>which then sends a POST to the create REST API endpoint of Gnocci at <a class="moz-txt-link-freetext" href="https://github.com/gnocchixyz/gnocchi/blob/f124cf76479720d3b9caf439241da12b2974ac90/gnocchi/rest/api.py#L302">https://github.com/gnocchixyz/gnocchi/blob/f124cf76479720d3b9caf439241da12b2974ac90/gnocchi/rest/api.py#L302</a>.
</p>
<p>
</p>
<p>The mentioned "<span class="pl-en"><span class="pl-token" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":6642735,"ref":"master","language":"Python","backend":"ALEPH_PRECISE","code_nav_context":"BLOB_VIEW","retry_backend":"","originating_url":"https://github.com/openstack/ceilometer/find-definition?q=ensures_archives_policies&blob_path=ceilometer%2Fpublisher%2Fgnocchi.py&ref=master&language=Python&row=268&col=8&code_nav_context=BLOB_VIEW","user_id":13763987}}" data-hydro-click-hmac="050f7cad16540a309462ed3b9604668471a133f9f8f168387d612f8633c867b6">ensures_archives_policies" method then checks for the "</span></span><span class="pl-v"><span class="pl-token" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":6642735,"ref":"master","language":"Python","backend":"ALEPH_PRECISE","code_nav_context":"BLOB_VIEW","retry_backend":"","originating_url":"https://github.com/openstack/ceilometer/find-definition?q=ArchivePolicyAlreadyExists&blob_path=ceilometer%2Fpublisher%2Fgnocchi.py&ref=master&language=Python&row=273&col=35&code_nav_context=BLOB_VIEW","user_id":13763987}}" data-hydro-click-hmac="2cc319df4748e8ccf9f676a98fccc22f26c1a6e92999c3233e1b893463d9b68c">ArchivePolicyAlreadyExists" exception (</span></span><span class="pl-v"><span class="pl-token" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":6642735,"ref":"master","language":"Python","backend":"ALEPH_PRECISE","code_nav_context":"BLOB_VIEW","retry_backend":"","originating_url":"https://github.com/openstack/ceilometer/find-definition?q=ArchivePolicyAlreadyExists&blob_path=ceilometer%2Fpublisher%2Fgnocchi.py&ref=master&language=Python&row=273&col=35&code_nav_context=BLOB_VIEW","user_id":13763987}}" data-hydro-click-hmac="2cc319df4748e8ccf9f676a98fccc22f26c1a6e92999c3233e1b893463d9b68c"><a class="moz-txt-link-freetext" href="https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L274">https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3bdf4f41e1/ceilometer/publisher/gnocchi.py#L274</a>)
and apparently this is the issue here - this exception is NOT thrown, but simply a 409 Conflict error.</span></span></p>
<p>
<span class="pl-v"><span class="pl-token" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":6642735,"ref":"master","language":"Python","backend":"ALEPH_PRECISE","code_nav_context":"BLOB_VIEW","retry_backend":"","originating_url":"https://github.com/openstack/ceilometer/find-definition?q=ArchivePolicyAlreadyExists&blob_path=ceilometer%2Fpublisher%2Fgnocchi.py&ref=master&language=Python&row=273&col=35&code_nav_context=BLOB_VIEW","user_id":13763987}}" data-hydro-click-hmac="2cc319df4748e8ccf9f676a98fccc22f26c1a6e92999c3233e1b893463d9b68c"></span></span></p>
<p><span class="pl-v"><span class="pl-token" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":6642735,"ref":"master","language":"Python","backend":"ALEPH_PRECISE","code_nav_context":"BLOB_VIEW","retry_backend":"","originating_url":"https://github.com/openstack/ceilometer/find-definition?q=ArchivePolicyAlreadyExists&blob_path=ceilometer%2Fpublisher%2Fgnocchi.py&ref=master&language=Python&row=273&col=35&code_nav_context=BLOB_VIEW","user_id":13763987}}" data-hydro-click-hmac="2cc319df4748e8ccf9f676a98fccc22f26c1a6e92999c3233e1b893463d9b68c">Exceptions are created and then enriched via the <a class="moz-txt-link-freetext" href="https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L217">https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L217</a> method and in our case it should be </span></span><a class="moz-txt-link-freetext" href="https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L161">https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f5962230a565574ce8c76c1caa/gnocchiclient/exceptions.py#L161</a> .</p>
<p>
</p>
<p>So I first thought the string must have been changed somewhere somehow for it to not match anymore, but all that code is unchanged for years.</p>
<p>
</p>
<p>But, when looking at a trace between the apache2 and the unix socket exposing the gnocci-api I saw something strange:
</p>
<br>
<p><style type="text/css">p, li { white-space: pre-wrap; }</style></p>
<p>
--- SENT ---
</p>
.7....proxy-sendchunked..1..APACHE_RUN_USER..ubuntu..APACHE_RUN_GROUP..ubuntu
.PATH_INFO../v1/archive_policy/ .HTTP_HOST<br>
.192.168.0.100..HTTP_USER_AGENTW.ceilometer-agent-notification
keystoneauth1/4.5.0 python-requests/2.27.1
CPython/3.8.10..HTTP_ACCEPT_ENCODING<br>
.gzip, deflate..HTTP_ACCEPT..application/json, */*..HTTP_CONNECTION<br>
.keep-alive..CONTENT_TYPE..application/json..HTTP_X_AUTH_TOKEN..gAAAAABiscoSYYjDWLVJzL8V12knrSmvI5aRYFTH4cZyQ0HKPBrheWqbh-6PS6Cf56kNTuTw5hwhpYa6UNEXkA_aIYj2xk6hTchgFaBOSrIHq36opuo46ktJCXcsEBQnus1_IZnWwuy8RBlzso6LGKeo37dzT_zloUYj_XhaMRDlgUifeAgUESQ..CONTENT_LENGTH..245..PATHF./usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin..SERVER_SIGNATUREJ.<address>Apache/2.4.41
(Ubuntu) Server at 192.168.0.100 Port 80</address><br>
..SERVER_SOFTWARE..Apache/2.4.41 (Ubuntu)..SERVER_NAME<br>
.192.168.0.100..SERVER_ADDR<br>
.192.168.0.100..SERVER_PORT..80..REMOTE_ADDR<br>
.192.168.0.100<br>
.DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..REQUEST_SCHEME..http..CONTEXT_PREFIX....CONTEXT_DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..SERVER_ADMIN..[no
address
given]..SCRIPT_FILENAME6.proxy:uwsgi://uwsgi-uds-gnocchi-api/v1/archive_policy/..REMOTE_PORT..42324..GATEWAY_INTERFACE..CGI/1.1..SERVER_PROTOCOL..HTTP/1.1..REQUEST_METHOD..POST..QUERY_STRING....REQUEST_URI../metric/v1/archive_policy/..SCRIPT_NAME../metric{"name":"ceilometer-high-rate","aggregation_methods":["mean","rate:mean"],"back_window":0,"definition":[{"granularity":"1
second","timespan":"1 hour"},{"granularity":"1 minute","timespan":"1
day"},{"granularity":"1 hour","timespan":"365 days"}]}<br>
--- SENT ---<br>
<p>
</p>
<p>
</p>
<p>
</p>
--- RECEIVED ---<br>
HTTP/1.1 409 Conflict<br>
Content-Length: 228<br>
Content-Type: text/html; charset=UTF-8<br>
Connection: close<br>
<br>
<html><br>
<head><br>
<title>409 Conflict</title><br>
</head><br>
<body><br>
<h1>409 Conflict</h1><br>
There was a conflict when trying to complete your request.<br
/><br /><br>
Archive policy ceilometer-high-rate already exists<br>
<br>
<br>
</body><br>
</html><br>
<p>--- RECEIVED ---</p>
<p>
</p>
<p>
</p>
<p>When though a content type of "application/json" was sent, the gnocchi-api returned "Content-Type: text/html; charset=UTF-8" and a "human readable" error instead of the expected JSON.
And that is why the exception is not properly casted and we end up in this archive_policy creation loop.</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>Why the response is html I was not yet able to find out.</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>Regards</p>
<p>
</p>
<p>Christian
</p>
</body>
</html>