[ceilometer][gnocchi][tacker] Internal Server Error in devstack for Zuul gate job

Matthias Runge mrunge at matthias-runge.de
Tue Aug 2 17:49:13 UTC 2022


Hi,

For a couple of years, Gnocchi is not anymore part of OpenStack, it is independent[1]. Especially, it is not part of OpenStack Telemetry.
As you see, Gnocchi can use some help.

Matthias

[1] https://julien.danjou.info/gnocchi-independence/

> Am 02.08.2022 um 14:44 schrieb ueha.ayumu at fujitsu.com:
> 
> Hi Radek
> 
> Thanks for your information!
> Umm.. When I checked the log, the following workarounds was suggested.
> -----------
> If you cannot immediately regenerate your protos, some other possible workarounds are:
> 1. Downgrade the protobuf package to 3.20.x or lower.
> 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
> More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
> -----------
> 
> The trigger seems to be that requirements upgraded the version of protobuf from 3.20.1 to 4.21.X.
> https://opendev.org/openstack/requirements/commit/0f0b7024ece8a47316e4d9775f09f0e8d53b4edb
> 
> Anyway, we will try workaround "2" as a temporary fix and wait for the telemetry team to solve this problem.
> Thank you for your help!
> 
> Best Regards,
> Ueha
> 
> -----Original Message-----
> From: Radosław Piliszek <radoslaw.piliszek at gmail.com> 
> Sent: Tuesday, August 2, 2022 7:03 PM
> To: Ueha, Ayumu/植波 歩 <ueha.ayumu at fujitsu.com>
> Cc: openstack-discuss at lists.openstack.org
> Subject: Re: [ceilometer][gnocchi][tacker] Internal Server Error in devstack for Zuul gate job
> 
> Hi Ueha,
> 
> It seems gnocchi is failing and requires a regeneration of the protobuf client:
> 
> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_712/851478/2/check/tacker-functional-devstack-multinode-sol/71262e6/controller-tacker/logs/screen-gnocchi-api.txt
> 
> ---
> 
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]:   File
> "/usr/local/lib/python3.8/dist-packages/google/protobuf/descriptor.py",
> line 755, in __new__
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]:
> _message.Message._CheckCalledFromGeneratedFile()
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]: TypeError: Descriptors cannot not be created directly.
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]: If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
> 
> ---
> 
> Kind regards,
> Radek
> -yoctozepto
> 
> On Tue, 2 Aug 2022 at 11:53, ueha.ayumu at fujitsu.com <ueha.ayumu at fujitsu.com> wrote:
>> 
>> Hi telemetry team,
>> 
>> 
>> 
>> I’m Ueha from Tacker team,
>> 
>> The Zuul gate job of Tacker failed with the following error. Do you know the solution?
>> 
>> The gate job has failed, so we would appreciate it if you could deal it with high priority.
>> 
>> Thanks!
>> 
>> 
>> 
>> for reference, the same error occurring in the ceilometer patch.
>> 
>> (https://review.opendev.org/c/openstack/ceilometer/+/851338 ‘s 
>> telemetry-dsvm-integration-centos-9s job)
>> 
>> 
>> 
>> ------------------
>> 
>>  ++ /opt/stack/ceilometer/devstack/plugin.sh:start_ceilometer:322 :   /usr/local/bin/ceilometer-upgrade
>> 
>>  2022-07-29 05:20:42.125 61523 DEBUG ceilometer.cmd.storage [-] 
>> Upgrading Gnocchi resource types upgrade 
>> /opt/stack/ceilometer/ceilometer/cmd/storage.py:42
>> 
>>  2022-07-29 05:20:42.228 61523 CRITICAL ceilometer [-] Unhandled 
>> error: gnocchiclient.exceptions.ClientException: Internal Server Error 
>> (HTTP 500)
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer Traceback (most recent call last):
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer   File "/usr/local/bin/ceilometer-upgrade", line 10, in <module>
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer     sys.exit(upgrade())
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer   File "/opt/stack/ceilometer/ceilometer/cmd/storage.py", line 49, in upgrade
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer     tenacity.Retrying(
>> 
>>  ......... omit .........
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer   File "/usr/local/lib/python3.8/dist-packages/gnocchiclient/client.py", line 52, in request
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer     raise exceptions.from_response(resp, method)
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer 
>> gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP 
>> 500)
>> 
>>  2022-07-29 05:20:42.228 61523 ERROR ceilometer
>> 
>> ------------------
>> 
>> Full log: 
>> https://zuul.opendev.org/t/openstack/build/71262e66ecf34827a8a3435657a
>> a9b3f
>> 
>> 
>> 
>> Best Regards,
>> 
>> Ueha
>> 
>> 




More information about the openstack-discuss mailing list