Hey Molka, Rafael, all, On 08/03/2022 21:02, Molka Gharbaoui wrote:
I removed manually "ceilometer-low" and "ceilometer-high" from archive-policy list (both were creating a conflict) however nothing changes: If I execute ceilometer-upgrade/ceilometer-polling nothing "new" happens. When I execute ceilometer-agent-notification, ceilometer-low/ceilometer-high are readded to the archive-policy list and the error reappears. I guess I should not execute those commands manually but I cannot find a way to push the metrics to the database.
I just ran into this issue when playing with Ceilometer + Gnocchi on a devstack running stable/yoga and tried the same things. To went through the cascade of calls happening in the code. First there is ceilometer "ensuring" the archive policy exists https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3... This is done via python-gnocchiclient and the create method at https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f59622.... which then sends a POST to the create REST API endpoint of Gnocci at https://github.com/gnocchixyz/gnocchi/blob/f124cf76479720d3b9caf439241da12b2.... The mentioned "ensures_archives_policies" method then checks for the "ArchivePolicyAlreadyExists" exception (https://github.com/openstack/ceilometer/blob/bf263b11181f4e44850e991282766b3...) and apparently this is the issue here - this exception is NOT thrown, but simply a 409 Conflict error. Exceptions are created and then enriched via the https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f59622... method and in our case it should be https://github.com/gnocchixyz/python-gnocchiclient/blob/7355fb2d7d3311f59622... . So I first thought the string must have been changed somewhere somehow for it to not match anymore, but all that code is unchanged for years. But, when looking at a trace between the apache2 and the unix socket exposing the gnocci-api I saw something strange: --- SENT --- .7....proxy-sendchunked..1..APACHE_RUN_USER..ubuntu..APACHE_RUN_GROUP..ubuntu .PATH_INFO../v1/archive_policy/ .HTTP_HOST .192.168.0.100..HTTP_USER_AGENTW.ceilometer-agent-notification keystoneauth1/4.5.0 python-requests/2.27.1 CPython/3.8.10..HTTP_ACCEPT_ENCODING .gzip, deflate..HTTP_ACCEPT..application/json, */*..HTTP_CONNECTION .keep-alive..CONTENT_TYPE..application/json..HTTP_X_AUTH_TOKEN..gAAAAABiscoSYYjDWLVJzL8V12knrSmvI5aRYFTH4cZyQ0HKPBrheWqbh-6PS6Cf56kNTuTw5hwhpYa6UNEXkA_aIYj2xk6hTchgFaBOSrIHq36opuo46ktJCXcsEBQnus1_IZnWwuy8RBlzso6LGKeo37dzT_zloUYj_XhaMRDlgUifeAgUESQ..CONTENT_LENGTH..245..PATHF./usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin..SERVER_SIGNATUREJ.<address>Apache/2.4.41 (Ubuntu) Server at 192.168.0.100 Port 80</address> ..SERVER_SOFTWARE..Apache/2.4.41 (Ubuntu)..SERVER_NAME .192.168.0.100..SERVER_ADDR .192.168.0.100..SERVER_PORT..80..REMOTE_ADDR .192.168.0.100 .DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..REQUEST_SCHEME..http..CONTEXT_PREFIX....CONTEXT_DOCUMENT_ROOT../opt/stack/horizon/.blackhole/..SERVER_ADMIN..[no address given]..SCRIPT_FILENAME6.proxy:uwsgi://uwsgi-uds-gnocchi-api/v1/archive_policy/..REMOTE_PORT..42324..GATEWAY_INTERFACE..CGI/1.1..SERVER_PROTOCOL..HTTP/1.1..REQUEST_METHOD..POST..QUERY_STRING....REQUEST_URI../metric/v1/archive_policy/..SCRIPT_NAME../metric{"name":"ceilometer-high-rate","aggregation_methods":["mean","rate:mean"],"back_window":0,"definition":[{"granularity":"1 second","timespan":"1 hour"},{"granularity":"1 minute","timespan":"1 day"},{"granularity":"1 hour","timespan":"365 days"}]} --- SENT --- --- RECEIVED --- HTTP/1.1 409 Conflict Content-Length: 228 Content-Type: text/html; charset=UTF-8 Connection: close <html> <head> <title>409 Conflict</title> </head> <body> <h1>409 Conflict</h1> There was a conflict when trying to complete your request.<br /><br /> Archive policy ceilometer-high-rate already exists </body> </html> --- RECEIVED --- When though a content type of "application/json" was sent, the gnocchi-api returned "Content-Type: text/html; charset=UTF-8" and a "human readable" error instead of the expected JSON. And that is why the exception is not properly casted and we end up in this archive_policy creation loop. Why the response is html I was not yet able to find out. Regards Christian