[ceilometer][gnocchi][tacker] Internal Server Error in devstack for Zuul gate job
Matthias Runge
mrunge at matthias-runge.de
Tue Aug 2 17:49:13 UTC 2022
Hi,
For a couple of years, Gnocchi is not anymore part of OpenStack, it is independent[1]. Especially, it is not part of OpenStack Telemetry.
As you see, Gnocchi can use some help.
Matthias
[1] https://julien.danjou.info/gnocchi-independence/
> Am 02.08.2022 um 14:44 schrieb ueha.ayumu at fujitsu.com:
>
> Hi Radek
>
> Thanks for your information!
> Umm.. When I checked the log, the following workarounds was suggested.
> -----------
> If you cannot immediately regenerate your protos, some other possible workarounds are:
> 1. Downgrade the protobuf package to 3.20.x or lower.
> 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
> More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
> -----------
>
> The trigger seems to be that requirements upgraded the version of protobuf from 3.20.1 to 4.21.X.
> https://opendev.org/openstack/requirements/commit/0f0b7024ece8a47316e4d9775f09f0e8d53b4edb
>
> Anyway, we will try workaround "2" as a temporary fix and wait for the telemetry team to solve this problem.
> Thank you for your help!
>
> Best Regards,
> Ueha
>
> -----Original Message-----
> From: Radosław Piliszek <radoslaw.piliszek at gmail.com>
> Sent: Tuesday, August 2, 2022 7:03 PM
> To: Ueha, Ayumu/植波 歩 <ueha.ayumu at fujitsu.com>
> Cc: openstack-discuss at lists.openstack.org
> Subject: Re: [ceilometer][gnocchi][tacker] Internal Server Error in devstack for Zuul gate job
>
> Hi Ueha,
>
> It seems gnocchi is failing and requires a regeneration of the protobuf client:
>
> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_712/851478/2/check/tacker-functional-devstack-multinode-sol/71262e6/controller-tacker/logs/screen-gnocchi-api.txt
>
> ---
>
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]: File
> "/usr/local/lib/python3.8/dist-packages/google/protobuf/descriptor.py",
> line 755, in __new__
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]:
> _message.Message._CheckCalledFromGeneratedFile()
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]: TypeError: Descriptors cannot not be created directly.
> Jul 29 05:20:41.105688 ubuntu-focal-ovh-bhs1-0030564625
> devstack at gnocchi-api.service[61040]: If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
>
> ---
>
> Kind regards,
> Radek
> -yoctozepto
>
> On Tue, 2 Aug 2022 at 11:53, ueha.ayumu at fujitsu.com <ueha.ayumu at fujitsu.com> wrote:
>>
>> Hi telemetry team,
>>
>>
>>
>> I’m Ueha from Tacker team,
>>
>> The Zuul gate job of Tacker failed with the following error. Do you know the solution?
>>
>> The gate job has failed, so we would appreciate it if you could deal it with high priority.
>>
>> Thanks!
>>
>>
>>
>> for reference, the same error occurring in the ceilometer patch.
>>
>> (https://review.opendev.org/c/openstack/ceilometer/+/851338 ‘s
>> telemetry-dsvm-integration-centos-9s job)
>>
>>
>>
>> ------------------
>>
>> ++ /opt/stack/ceilometer/devstack/plugin.sh:start_ceilometer:322 : /usr/local/bin/ceilometer-upgrade
>>
>> 2022-07-29 05:20:42.125 61523 DEBUG ceilometer.cmd.storage [-]
>> Upgrading Gnocchi resource types upgrade
>> /opt/stack/ceilometer/ceilometer/cmd/storage.py:42
>>
>> 2022-07-29 05:20:42.228 61523 CRITICAL ceilometer [-] Unhandled
>> error: gnocchiclient.exceptions.ClientException: Internal Server Error
>> (HTTP 500)
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer Traceback (most recent call last):
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/bin/ceilometer-upgrade", line 10, in <module>
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer sys.exit(upgrade())
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/opt/stack/ceilometer/ceilometer/cmd/storage.py", line 49, in upgrade
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer tenacity.Retrying(
>>
>> ......... omit .........
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer File "/usr/local/lib/python3.8/dist-packages/gnocchiclient/client.py", line 52, in request
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer raise exceptions.from_response(resp, method)
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer
>> gnocchiclient.exceptions.ClientException: Internal Server Error (HTTP
>> 500)
>>
>> 2022-07-29 05:20:42.228 61523 ERROR ceilometer
>>
>> ------------------
>>
>> Full log:
>> https://zuul.opendev.org/t/openstack/build/71262e66ecf34827a8a3435657a
>> a9b3f
>>
>>
>>
>> Best Regards,
>>
>> Ueha
>>
>>
More information about the openstack-discuss
mailing list