[ceilometer][gnocchi][tacker] Internal Server Error in devstack for Zuul gate job
Hi telemetry team, I’m Ueha from Tacker team, The Zuul gate job of Tacker failed with the following error. Do you know the solution? The gate job has failed, so we would appreciate it if you could deal it with high priority. Thanks! for reference, the same error occurring in the ceilometer patch. (https://review.opendev.org/c/openstack/ceilometer/+/851338 ‘s telemetry-dsvm-integration-centos-9s job) ------------------ ++ /opt/stack/ceilometer/devstack/plugin.sh:start_ceilometer:322 : /usr/local/bin/ceilometer-upgrade 2022-07-29 05:20:42.125 61523 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /opt/stack/ceilometer/ceilometer/cmd/storage.py:42 2022-07-29 05:20:42.228 61523 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500) 2022-07-29 05:20:42.228 61523 ERROR ceilometer Traceback (most recent call last): 2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/bin/ceilometer-upgrade", line 10, in <module> 2022-07-29 05:20:42.228 61523 ERROR ceilometer sys.exit(upgrade()) 2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/opt/stack/ceilometer/ceilometer/cmd/storage.py", line 49, in upgrade 2022-07-29 05:20:42.228 61523 ERROR ceilometer tenacity.Retrying( ......... omit ......... 2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/lib/python3.8/dist-packages/gnocchiclient/client.py", line 52, in request 2022-07-29 05:20:42.228 61523 ERROR ceilometer raise exceptions.from_response(resp, method) 2022-07-29 05:20:42.228 61523 ERROR ceilometer gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500) 2022-07-29 05:20:42.228 61523 ERROR ceilometer ------------------ Full log: https://zuul.opendev.org/t/openstack/build/71262e66ecf34827a8a3435657aa9b3f Best Regards, Ueha
Hi Ueha, It seems gnocchi is failing and requires a regeneration of the protobuf client: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z... --- Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: File "/usr/local/lib/python3.8/dist-packages/google/protobuf/descriptor.py", line 755, in __new__ Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: _message.Message._CheckCalledFromGeneratedFile() Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: TypeError: Descriptors cannot not be created directly. Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. --- Kind regards, Radek -yoctozepto On Tue, 2 Aug 2022 at 11:53, ueha.ayumu@fujitsu.com <ueha.ayumu@fujitsu.com> wrote:
Hi telemetry team,
I’m Ueha from Tacker team,
The Zuul gate job of Tacker failed with the following error. Do you know the solution?
The gate job has failed, so we would appreciate it if you could deal it with high priority.
Thanks!
for reference, the same error occurring in the ceilometer patch.
(https://review.opendev.org/c/openstack/ceilometer/+/851338 ‘s telemetry-dsvm-integration-centos-9s job)
------------------
++ /opt/stack/ceilometer/devstack/plugin.sh:start_ceilometer:322 : /usr/local/bin/ceilometer-upgrade
2022-07-29 05:20:42.125 61523 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /opt/stack/ceilometer/ceilometer/cmd/storage.py:42
2022-07-29 05:20:42.228 61523 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500)
2022-07-29 05:20:42.228 61523 ERROR ceilometer Traceback (most recent call last):
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/bin/ceilometer-upgrade", line 10, in <module>
2022-07-29 05:20:42.228 61523 ERROR ceilometer sys.exit(upgrade())
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/opt/stack/ceilometer/ceilometer/cmd/storage.py", line 49, in upgrade
2022-07-29 05:20:42.228 61523 ERROR ceilometer tenacity.Retrying(
......... omit .........
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/lib/python3.8/dist-packages/gnocchiclient/client.py", line 52, in request
2022-07-29 05:20:42.228 61523 ERROR ceilometer raise exceptions.from_response(resp, method)
2022-07-29 05:20:42.228 61523 ERROR ceilometer gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500)
2022-07-29 05:20:42.228 61523 ERROR ceilometer
------------------
Full log: https://zuul.opendev.org/t/openstack/build/71262e66ecf34827a8a3435657aa9b3f
Best Regards,
Ueha
Hi Radek Thanks for your information! Umm.. When I checked the log, the following workarounds was suggested. ----------- If you cannot immediately regenerate your protos, some other possible workarounds are: 1. Downgrade the protobuf package to 3.20.x or lower. 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower). More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-u... ----------- The trigger seems to be that requirements upgraded the version of protobuf from 3.20.1 to 4.21.X. https://opendev.org/openstack/requirements/commit/0f0b7024ece8a47316e4d9775f... Anyway, we will try workaround "2" as a temporary fix and wait for the telemetry team to solve this problem. Thank you for your help! Best Regards, Ueha -----Original Message----- From: Radosław Piliszek <radoslaw.piliszek@gmail.com> Sent: Tuesday, August 2, 2022 7:03 PM To: Ueha, Ayumu/植波 歩 <ueha.ayumu@fujitsu.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer][gnocchi][tacker] Internal Server Error in devstack for Zuul gate job Hi Ueha, It seems gnocchi is failing and requires a regeneration of the protobuf client: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z... --- Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: File "/usr/local/lib/python3.8/dist-packages/google/protobuf/descriptor.py", line 755, in __new__ Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: _message.Message._CheckCalledFromGeneratedFile() Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: TypeError: Descriptors cannot not be created directly. Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. --- Kind regards, Radek -yoctozepto On Tue, 2 Aug 2022 at 11:53, ueha.ayumu@fujitsu.com <ueha.ayumu@fujitsu.com> wrote:
Hi telemetry team,
I’m Ueha from Tacker team,
The Zuul gate job of Tacker failed with the following error. Do you know the solution?
The gate job has failed, so we would appreciate it if you could deal it with high priority.
Thanks!
for reference, the same error occurring in the ceilometer patch.
(https://review.opendev.org/c/openstack/ceilometer/+/851338 ‘s telemetry-dsvm-integration-centos-9s job)
------------------
++ /opt/stack/ceilometer/devstack/plugin.sh:start_ceilometer:322 : /usr/local/bin/ceilometer-upgrade
2022-07-29 05:20:42.125 61523 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /opt/stack/ceilometer/ceilometer/cmd/storage.py:42
2022-07-29 05:20:42.228 61523 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500)
2022-07-29 05:20:42.228 61523 ERROR ceilometer Traceback (most recent call last):
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/bin/ceilometer-upgrade", line 10, in <module>
2022-07-29 05:20:42.228 61523 ERROR ceilometer sys.exit(upgrade())
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/opt/stack/ceilometer/ceilometer/cmd/storage.py", line 49, in upgrade
2022-07-29 05:20:42.228 61523 ERROR ceilometer tenacity.Retrying(
......... omit .........
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/lib/python3.8/dist-packages/gnocchiclient/client.py", line 52, in request
2022-07-29 05:20:42.228 61523 ERROR ceilometer raise exceptions.from_response(resp, method)
2022-07-29 05:20:42.228 61523 ERROR ceilometer gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500)
2022-07-29 05:20:42.228 61523 ERROR ceilometer
------------------
Full log: https://zuul.opendev.org/t/openstack/build/71262e66ecf34827a8a3435657a a9b3f
Best Regards,
Ueha
Hi, For a couple of years, Gnocchi is not anymore part of OpenStack, it is independent[1]. Especially, it is not part of OpenStack Telemetry. As you see, Gnocchi can use some help. Matthias [1] https://julien.danjou.info/gnocchi-independence/
Am 02.08.2022 um 14:44 schrieb ueha.ayumu@fujitsu.com:
Hi Radek
Thanks for your information! Umm.. When I checked the log, the following workarounds was suggested. ----------- If you cannot immediately regenerate your protos, some other possible workarounds are: 1. Downgrade the protobuf package to 3.20.x or lower. 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower). More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-u... -----------
The trigger seems to be that requirements upgraded the version of protobuf from 3.20.1 to 4.21.X. https://opendev.org/openstack/requirements/commit/0f0b7024ece8a47316e4d9775f...
Anyway, we will try workaround "2" as a temporary fix and wait for the telemetry team to solve this problem. Thank you for your help!
Best Regards, Ueha
-----Original Message----- From: Radosław Piliszek <radoslaw.piliszek@gmail.com> Sent: Tuesday, August 2, 2022 7:03 PM To: Ueha, Ayumu/植波 歩 <ueha.ayumu@fujitsu.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer][gnocchi][tacker] Internal Server Error in devstack for Zuul gate job
Hi Ueha,
It seems gnocchi is failing and requires a regeneration of the protobuf client:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...
---
Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: File "/usr/local/lib/python3.8/dist-packages/google/protobuf/descriptor.py", line 755, in __new__ Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: _message.Message._CheckCalledFromGeneratedFile() Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: TypeError: Descriptors cannot not be created directly. Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625 devstack@gnocchi-api.service[61040]: If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
---
Kind regards, Radek -yoctozepto
On Tue, 2 Aug 2022 at 11:53, ueha.ayumu@fujitsu.com <ueha.ayumu@fujitsu.com> wrote:
Hi telemetry team,
I’m Ueha from Tacker team,
The Zuul gate job of Tacker failed with the following error. Do you know the solution?
The gate job has failed, so we would appreciate it if you could deal it with high priority.
Thanks!
for reference, the same error occurring in the ceilometer patch.
(https://review.opendev.org/c/openstack/ceilometer/+/851338 ‘s telemetry-dsvm-integration-centos-9s job)
------------------
++ /opt/stack/ceilometer/devstack/plugin.sh:start_ceilometer:322 : /usr/local/bin/ceilometer-upgrade
2022-07-29 05:20:42.125 61523 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /opt/stack/ceilometer/ceilometer/cmd/storage.py:42
2022-07-29 05:20:42.228 61523 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500)
2022-07-29 05:20:42.228 61523 ERROR ceilometer Traceback (most recent call last):
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/bin/ceilometer-upgrade", line 10, in <module>
2022-07-29 05:20:42.228 61523 ERROR ceilometer sys.exit(upgrade())
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/opt/stack/ceilometer/ceilometer/cmd/storage.py", line 49, in upgrade
2022-07-29 05:20:42.228 61523 ERROR ceilometer tenacity.Retrying(
......... omit .........
2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/lib/python3.8/dist-packages/gnocchiclient/client.py", line 52, in request
2022-07-29 05:20:42.228 61523 ERROR ceilometer raise exceptions.from_response(resp, method)
2022-07-29 05:20:42.228 61523 ERROR ceilometer gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 500)
2022-07-29 05:20:42.228 61523 ERROR ceilometer
------------------
Full log: https://zuul.opendev.org/t/openstack/build/71262e66ecf34827a8a3435657a a9b3f
Best Regards,
Ueha
On 2022-08-02 19:49:13 +0200 (+0200), Matthias Runge wrote:
For a couple of years, Gnocchi is not anymore part of OpenStack, it is independent[1]. Especially, it is not part of OpenStack Telemetry. As you see, Gnocchi can use some help. [...]
In particular, Gnocchi left because its maintainers realized that not being part of OpenStack would help them get more contributors, wider usage, and better long-term maintenance. -- Jeremy Stanley
participants (4)
-
Jeremy Stanley
-
Matthias Runge
-
Radosław Piliszek
-
ueha.ayumu＠fujitsu.com