[kolla] ujson issue affected to few containers.
Hi everyone, I'm not sure it should be bug report or not. So I email out about this issue. In these days, I found the Rocky source deployment always failed at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade. So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson. https://pastebin.com/nGqsM0uf Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346 But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi. I think we have to patch it, but not sure about the workaround. Does anyone have good idea? Many thanks, Eddie.
Hi Eddie, the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1] You said only Rocky source - but is this ubuntu or centos? Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths. The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky). [1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos= -yoctozepto czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about this issue.
In these days, I found the Rocky source deployment always failed at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade. So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
Hi Radosław, Sorry about lost distro information, the distro we're using is Ubuntu. We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue. BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python. Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about this
issue.
In these days, I found the Rocky source deployment always failed at
Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade.
So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
Hi Eddie, ujson is one of the few packages that has no wheel, so it is being recompiled each time. It might be the case that some change to gcc and/or python-dev parts caused that. The issue looks like an edge case of c99 spec non-conformity so valid compiles may break on it (or work - just that there is no guarantee). -yoctozepto czw., 30 sty 2020 o 09:06 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi Radosław,
Sorry about lost distro information, the distro we're using is Ubuntu.
We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue.
BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python.
Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about this issue.
In these days, I found the Rocky source deployment always failed at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade. So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
valid compilers* czw., 30 sty 2020 o 09:20 Radosław Piliszek <radoslaw.piliszek@gmail.com> napisał(a):
Hi Eddie,
ujson is one of the few packages that has no wheel, so it is being recompiled each time. It might be the case that some change to gcc and/or python-dev parts caused that. The issue looks like an edge case of c99 spec non-conformity so valid compiles may break on it (or work - just that there is no guarantee).
-yoctozepto
czw., 30 sty 2020 o 09:06 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi Radosław,
Sorry about lost distro information, the distro we're using is Ubuntu.
We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue.
BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python.
Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about this issue.
In these days, I found the Rocky source deployment always failed at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade. So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
Seeing this issue when messing around with Gnocchi on Ubuntu 18.04 as well. Temp solved it by installing ujson from master as suggested in [1] instead of pypi. [1] https://github.com/esnme/ultrajson/issues/346 On 1/30/20 9:10 AM, Eddie Yen wrote:
Hi Radosław,
Sorry about lost distro information, the distro we're using is Ubuntu.
We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue.
BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python.
Radosław Piliszek <radoslaw.piliszek@gmail.com <mailto:radoslaw.piliszek@gmail.com>> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history <https://pypi.org/project/ujson/#history> [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com <mailto:missile0407@gmail.com>> napisał(a): > > Hi everyone, > > I'm not sure it should be bug report or not. So I email out about this issue. > > In these days, I found the Rocky source deployment always failed at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade. > So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson. > > https://pastebin.com/nGqsM0uf > > Then I googled it and found this issue is already happened and released fixes. > https://github.com/esnme/ultrajson/issues/346 <https://github.com/esnme/ultrajson/issues/346> > > But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). > And this not only affected to Ceilometer, but may also Gnocchi. > > I think we have to patch it, but not sure about the workaround. > Does anyone have good idea? > > Many thanks, > Eddie.
In summary, looks like we have to wait the project release the fixed code on PyPI or compile the source code from its git project. Otherwise these containers may still affected this issue and can't deploy or working. We may going to try the Ubuntu binary deployment to see it also affect of not. Perhaps the user may going to deploy with binary on Ubuntu before the fix release to PyPI. - Eddie Tobias Urdin <tobias.urdin@binero.se> 於 2020年1月30日 週四 下午4:29寫道：
Seeing this issue when messing around with Gnocchi on Ubuntu 18.04 as well. Temp solved it by installing ujson from master as suggested in [1] instead of pypi.
[1] https://github.com/esnme/ultrajson/issues/346
On 1/30/20 9:10 AM, Eddie Yen wrote:
Hi Radosław,
Sorry about lost distro information, the distro we're using is Ubuntu.
We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue.
BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python.
Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about this
issue.
In these days, I found the Rocky source deployment always failed at
Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade.
So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
Well, the release does not look it's going to happen ever. Ubuntu binary rocky probably froze in time so it has a higher chance of working, though a potential rebuild will probably kill it as well. Let's start a general thread about ujson. -yoctozepto pt., 31 sty 2020 o 03:26 Eddie Yen <missile0407@gmail.com> napisał(a):
In summary, looks like we have to wait the project release the fixed code on PyPI or compile the source code from its git project. Otherwise these containers may still affected this issue and can't deploy or working.
We may going to try the Ubuntu binary deployment to see it also affect of not. Perhaps the user may going to deploy with binary on Ubuntu before the fix release to PyPI.
- Eddie
Tobias Urdin <tobias.urdin@binero.se> 於 2020年1月30日 週四 下午4:29寫道：
Seeing this issue when messing around with Gnocchi on Ubuntu 18.04 as well. Temp solved it by installing ujson from master as suggested in [1] instead of pypi.
[1] https://github.com/esnme/ultrajson/issues/346
On 1/30/20 9:10 AM, Eddie Yen wrote:
Hi Radosław,
Sorry about lost distro information, the distro we're using is Ubuntu.
We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue.
BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python.
Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about this issue.
In these days, I found the Rocky source deployment always failed at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade. So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
I checked ceilometer and it seems they dropped ujson in queens, it's only the gnocchi client that still uses it, unfortunately. -yoctozepto pt., 31 sty 2020 o 09:01 Radosław Piliszek <radoslaw.piliszek@gmail.com> napisał(a):
Well, the release does not look it's going to happen ever. Ubuntu binary rocky probably froze in time so it has a higher chance of working, though a potential rebuild will probably kill it as well.
Let's start a general thread about ujson.
-yoctozepto
pt., 31 sty 2020 o 03:26 Eddie Yen <missile0407@gmail.com> napisał(a):
In summary, looks like we have to wait the project release the fixed code on PyPI or compile the source code from its git project. Otherwise these containers may still affected this issue and can't deploy or working.
We may going to try the Ubuntu binary deployment to see it also affect of not. Perhaps the user may going to deploy with binary on Ubuntu before the fix release to PyPI.
- Eddie
Tobias Urdin <tobias.urdin@binero.se> 於 2020年1月30日 週四 下午4:29寫道：
Seeing this issue when messing around with Gnocchi on Ubuntu 18.04 as well. Temp solved it by installing ujson from master as suggested in [1] instead of pypi.
[1] https://github.com/esnme/ultrajson/issues/346
On 1/30/20 9:10 AM, Eddie Yen wrote:
Hi Radosław,
Sorry about lost distro information, the distro we're using is Ubuntu.
We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue.
BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python.
Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected, but monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about this issue.
In these days, I found the Rocky source deployment always failed at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade. So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
We tested the Rocky deployment based on Ubuntu binary, and confirm that the binary one was not affect the issue. So it means that it will need to use binary type on Rocky if the user going to use Gnocchi + Ceilometer. - Eddie Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月31日 週五 下午4:18寫道：
I checked ceilometer and it seems they dropped ujson in queens, it's only the gnocchi client that still uses it, unfortunately.
-yoctozepto
pt., 31 sty 2020 o 09:01 Radosław Piliszek <radoslaw.piliszek@gmail.com> napisał(a):
Well, the release does not look it's going to happen ever. Ubuntu binary rocky probably froze in time so it has a higher chance of working, though a potential rebuild will probably kill it as well.
Let's start a general thread about ujson.
-yoctozepto
pt., 31 sty 2020 o 03:26 Eddie Yen <missile0407@gmail.com> napisał(a):
In summary, looks like we have to wait the project release the fixed
Otherwise these containers may still affected this issue and can't deploy or working.
We may going to try the Ubuntu binary deployment to see it also affect of not. Perhaps the user may going to deploy with binary on Ubuntu before
- Eddie
Tobias Urdin <tobias.urdin@binero.se> 於 2020年1月30日 週四 下午4:29寫道：
Seeing this issue when messing around with Gnocchi on Ubuntu 18.04 as
well.
Temp solved it by installing ujson from master as suggested in [1] instead of pypi.
[1] https://github.com/esnme/ultrajson/issues/346
On 1/30/20 9:10 AM, Eddie Yen wrote:
Hi Radosław,
Sorry about lost distro information, the distro we're using is Ubuntu.
We have an old copy of ceilometer container image, the ujson.so version between old and latest are both 1.35 But only latest one affected this issue.
BTW, I read the last reply on issue page. Since he said the python 3 with newer GCC is OK, I think it may caused by python version issue or GCC compiler versioning. It may become a huge architect if it really caused by compiling issue, if Ubuntu updated GCC or python.
Radosław Piliszek <radoslaw.piliszek@gmail.com> 於 2020年1月30日 週四 下午3:48寫道：
Hi Eddie,
the issue is that the project did *not* do a release. The latest is still 1.35 from Jan 20, *2016*... [1]
You said only Rocky source - but is this ubuntu or centos?
Also, by the looks of [2] master ceilometer is no longer affected,
but
monasca and mistral might still be if they call affected paths.
The project looks dead so we are fried unless we override and start using its sources from git (hacky hacky).
[1] https://pypi.org/project/ujson/#history [2] http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=
-yoctozepto
czw., 30 sty 2020 o 03:31 Eddie Yen <missile0407@gmail.com> napisał(a):
Hi everyone,
I'm not sure it should be bug report or not. So I email out about
code on PyPI or compile the source code from its git project. the fix release to PyPI. this issue.
In these days, I found the Rocky source deployment always failed
at Ceilometer bootstrapping. Then I found it failed at ceilometer-upgrade.
So I tried to looking at ceilometer-upgrade.log and the error shows it failed to import ujson.
Then I googled it and found this issue is already happened and released fixes. https://github.com/esnme/ultrajson/issues/346
But it seems like the container still using the questionable one, even today (Jan 30 UTC+8). And this not only affected to Ceilometer, but may also Gnocchi.
I think we have to patch it, but not sure about the workaround. Does anyone have good idea?
Many thanks, Eddie.
participants (3)
-
Eddie Yen
-
Radosław Piliszek
-
Tobias Urdin