On Tue, May 17, 2022 at 8:24 PM Clark Boylan <cboylan@sapwetik.org> wrote:

On Tue, May 17, 2022, at 10:56 AM, Dmitry Tantsur wrote:

...

Hi all,

It is happening again, the Bifrost CI is broken because libvirt-python cannot be built from source, this time on Stream 9.

Missing type converters: int *:1 ERROR: failed virDomainQemuMonitorCommandWithFiles

I created a gist with a reproducer: https://gist.github.com/dtantsur/835303c6a68ed77157016f5955183115.

I cannot count how many times we had to deal with similar errors. I assume, libvirt-python has to be newer than the installed Python (8.2.0 in CS9, 8.0.0 in constraints). Should we stop constraining libvirt-python? Any other ideas?

Your libvirt-python version needs to be at least as new as your libvirt version. New libvirt-python versions are expected to continue to work with old libvirt versions as well (though it may need to be built against the specific libvirt?). In this case it looks like CentOS Stream 9 libvirt is newer than what was in constraints.

Generally constraints should update quickly. Looking at master upper-constraints libvirt-python was updated to 8.3.0 on May 4 and according to pypi the package updated on May 2 which seems reasonable. The problem here appears to be that you want this to work on a stable branch (yoga) and stable branches do not update constraints.

My suggestion is that we use stable platforms for testing stable releases. The CentOS Stream releases seem to get updates that break stable software expectations far more than our other platforms. When working against master and trying to chase the latest and greatest this is probably a feature, but is problematic when you want rate of change to fall to near zero. I would consider not using Stream on stable branches if these problems persist.

What would you suggest to use to test Red Hat systems then? Dmitry

...

Dmitry

-- Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 14, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

-- Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 14, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

On Wed, May 18, 2022, at 12:46 AM, Radosław Piliszek wrote:

On Wed, 18 May 2022 at 08:27, Dmitry Tantsur <dtantsur@redhat.com> wrote:

...

On Tue, May 17, 2022 at 8:24 PM Clark Boylan <cboylan@sapwetik.org> wrote:

...

On Tue, May 17, 2022, at 10:56 AM, Dmitry Tantsur wrote:

...

Hi all,

It is happening again, the Bifrost CI is broken because libvirt-python cannot be built from source, this time on Stream 9.

Missing type converters: int *:1 ERROR: failed virDomainQemuMonitorCommandWithFiles

I created a gist with a reproducer: https://gist.github.com/dtantsur/835303c6a68ed77157016f5955183115.

I cannot count how many times we had to deal with similar errors. I assume, libvirt-python has to be newer than the installed Python (8.2.0 in CS9, 8.0.0 in constraints). Should we stop constraining libvirt-python? Any other ideas?

Your libvirt-python version needs to be at least as new as your libvirt version. New libvirt-python versions are expected to continue to work with old libvirt versions as well (though it may need to be built against the specific libvirt?). In this case it looks like CentOS Stream 9 libvirt is newer than what was in constraints.

Generally constraints should update quickly. Looking at master upper-constraints libvirt-python was updated to 8.3.0 on May 4 and according to pypi the package updated on May 2 which seems reasonable. The problem here appears to be that you want this to work on a stable branch (yoga) and stable branches do not update constraints.

My suggestion is that we use stable platforms for testing stable releases. The CentOS Stream releases seem to get updates that break stable software expectations far more than our other platforms. When working against master and trying to chase the latest and greatest this is probably a feature, but is problematic when you want rate of change to fall to near zero. I would consider not using Stream on stable branches if these problems persist.

What would you suggest to use to test Red Hat systems then?

We have both Rocky and OpenEuler images available. I believe both attempt to be RHEL downstreams similar to what CentOS once provided.

Eh, come on. Stream breaks but so will the next minor RHEL release. The real issue is that libvirt python relies on the correct C bindings too heavily. Long-term I would love to see pure Python bindings but short-term I suggest Ironic follows the steps of Nova, Masakari, Kolla and DevStack.

I disagree with several aspects of this. While old CentOS point releases did occasionally break things, the pain was nowhere near what stream has been like. I've had to debug a number of stream problems that are assumed to be infra problems but it is just the upstream moving quickly and breaking but not also fixing quickly. Ping was broken for about a month for example. Fixing Rocky for example, once a year or whatever the update cadence is, would be nothing like the Stream treadmill we have been running since the beginning of the year. As far as libvirt python bindings go the issue that caused you to update devstack was because CentOS stream was using CentOS !stream wheels. That issue is completely separate to whether or not you should consume libvirt-python from properly built for that distro wheels and we hadn't had problems doing that for many years that I know of. We fixed the mixup in wheel sources for CentOS Stream and you could've continued to consume the wheels from pypi as far as I know. However, it is possible that recent updates that have broken bifrost would've also forced us to rebuild those wheels, but we don't know because we basically stopped doing that in devstack.

-yoctozepto