[ironic] [stable] Bifrost stable/stein is broken (by eventlet?): help needed
Hi folks, I'm trying to revive the Bifrost stable/stein CI, and after fixing a bunch of issues in https://review.opendev.org/749014 I've hit a wall with what seems an eventlet problem: ironic-inspector fails to start with: Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:16> ignored I've managed to find similar issues, but they should have been resolved in the eventlet version in stein (0.24.1). Any ideas? If we cannot fix it, we'll have to EOL stein and earlier on bifrost. Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Thu, 3 Sep 2020 at 11:31, Dmitry Tantsur <dtantsur@redhat.com> wrote:
Hi folks,
I'm trying to revive the Bifrost stable/stein CI, and after fixing a bunch of issues in https://review.opendev.org/749014 I've hit a wall with what seems an eventlet problem: ironic-inspector fails to start with:
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:16> ignored
I've managed to find similar issues, but they should have been resolved in the eventlet version in stein (0.24.1). Any ideas?
If we cannot fix it, we'll have to EOL stein and earlier on bifrost.
Strange. Do you know why this affects only bifrost and not ironic inspector CI?
Dmitry
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Fri, Sep 4, 2020 at 9:26 AM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 3 Sep 2020 at 11:31, Dmitry Tantsur <dtantsur@redhat.com> wrote:
Hi folks,
I'm trying to revive the Bifrost stable/stein CI, and after fixing a
bunch of issues in https://review.opendev.org/749014 I've hit a wall with what seems an eventlet problem: ironic-inspector fails to start with:
Exception AttributeError: "'_SocketDuckForFd' object has no attribute
'_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:16> ignored
I've managed to find similar issues, but they should have been resolved
in the eventlet version in stein (0.24.1). Any ideas?
If we cannot fix it, we'll have to EOL stein and earlier on bifrost.
Strange. Do you know why this affects only bifrost and not ironic inspector CI?
I'm totally lost, but at the very least the normal CI is devstack, so a lot of things can be different. Dmitry
Dmitry
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
O'Neill
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On 9/3/20 3:30 AM, Dmitry Tantsur wrote:
*External email: Use caution opening links or attachments*
Hi folks,
I'm trying to revive the Bifrost stable/stein CI, and after fixing a bunch of issues in https://review.opendev.org/749014 I've hit a wall with what seems an eventlet problem: ironic-inspector fails to start with:
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:16> ignored
I've managed to find similar issues, but they should have been resolved in the eventlet version in stein (0.24.1). Any ideas?
If we cannot fix it, we'll have to EOL stein and earlier on bifrost.
Dmitry
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
The "ignored" makes me think that it shouldn't actually be a problem -- are we assuming that's the error because of logs like https://c972f4bb262ae2d5c5d6-598e1d61c0aab85aa3b67b337ca2c556.ssl.cf2.rackcd... ? Digging down to https://c972f4bb262ae2d5c5d6-598e1d61c0aab85aa3b67b337ca2c556.ssl.cf2.rackcd... shows tracebacks like File ".../eventlet/hubs/__init__.py", line 39, in get_default_hub import eventlet.hubs.epolls File ".../eventlet/hubs/epolls.py", line 13, in <module> from eventlet.hubs.hub import BaseHub File ".../eventlet/hubs/hub.py", line 24, in <module> import monotonic File ".../monotonic.py", line 169, in <module> raise RuntimeError('no suitable implementation for this system: ' + repr(e)) RuntimeError: no suitable implementation for this system: AttributeError("'module' object has no attribute 'epolls'",) Maybe it's worth looking at why monotonic can't find a suitable implementation? Tim
On Fri, Sep 4, 2020 at 11:12 PM Tim Burke <tburke@nvidia.com> wrote:
On 9/3/20 3:30 AM, Dmitry Tantsur wrote:
*External email: Use caution opening links or attachments*
Hi folks,
I'm trying to revive the Bifrost stable/stein CI, and after fixing a bunch of issues in https://review.opendev.org/749014 I've hit a wall with what seems an eventlet problem: ironic-inspector fails to start with:
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:16> ignored
I've managed to find similar issues, but they should have been resolved in the eventlet version in stein (0.24.1). Any ideas?
If we cannot fix it, we'll have to EOL stein and earlier on bifrost.
Dmitry
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
The "ignored" makes me think that it shouldn't actually be a problem -- are we assuming that's the error because of logs like
https://c972f4bb262ae2d5c5d6-598e1d61c0aab85aa3b67b337ca2c556.ssl.cf2.rackcd... ?
Digging down to
https://c972f4bb262ae2d5c5d6-598e1d61c0aab85aa3b67b337ca2c556.ssl.cf2.rackcd... shows tracebacks like
File ".../eventlet/hubs/__init__.py", line 39, in get_default_hub import eventlet.hubs.epolls File ".../eventlet/hubs/epolls.py", line 13, in <module> from eventlet.hubs.hub import BaseHub File ".../eventlet/hubs/hub.py", line 24, in <module> import monotonic File ".../monotonic.py", line 169, in <module> raise RuntimeError('no suitable implementation for this system: ' + repr(e)) RuntimeError: no suitable implementation for this system: AttributeError("'module' object has no attribute 'epolls'",)
Maybe it's worth looking at why monotonic can't find a suitable implementation?
It used to be an issue in eventlet, but we seem to be using a new enough version to avoid it (the initial problem was IIRC around the pike timeframe). Dmitry
Tim
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
participants (3)
-
Dmitry Tantsur
-
Mark Goddard
-
Tim Burke