Hi all,

The problem is resolved at zun side. I have created a few patches to re-enable the zun role in kolla-ansible:
https://review.opendev.org/c/openstack/kolla-ansible/+/904093
https://review.opendev.org/c/openstack/kolla-ansible/+/904164
https://review.opendev.org/c/openstack/kolla-ansible/+/904094

For people who want to know the technical details, zun pinned docker 20 because the latest version of docker dropped support for the --cluster-store option [1]. In before, zun used kuryr-libnetwork (a docker network plugin) with "global" capability scope which relied on the --cluster-store option. Right now, zun has switched to the "local" capability scope so the problem has been worked around. The related patches:
https://review.opendev.org/c/openstack/zun/+/903884
https://review.opendev.org/c/openstack/zun/+/903872
https://review.opendev.org/c/openstack/kuryr-libnetwork/+/903925
https://review.opendev.org/c/openstack/kuryr-libnetwork/+/877180

With above fixes, zun no longer required docker 20 and etcd. The problem from kolla side should be resolved.

Best regards,
Hongbin


[1] https://github.com/moby/moby/issues/40383

On Fri, Sep 15, 2023 at 12:04 AM Hongbin Lu <hongbin034@gmail.com> wrote:
Hi,

I am from Zun team. The goal is to fix it in "C" release. I will keep the status updated in this ticket: https://bugs.launchpad.net/zun/+bug/2007142 . Sorry for all the inconvenience.

Best regards,
Hongbin

On Thu, Sep 14, 2023 at 6:56 PM Jan Gutter <openstack@jangutter.com> wrote:
Hi folks,

In yesterday's Kolla IRC meeting we discussed the path forward for Zun
[1] and came to the conclusion that it might be better for all
involved to drop support for Zun for the 2023.1 release until it can
be updated to support the necessary dependencies. More specific
details at the end of the message.

With this in mind, the plan is to start this process in a week from
now, unless a better solution is presented. There is still some time
for discussion, please don't hesitate!

There are two paths forward after this:

If Zun can be updated to support the dependencies during the
maintenance timeframe of 2023.1, then support would be restored.

If no support is available at the time Caracal releases, however, it
will be considered for removal in kolla and kolla-ansible.

What this means:

Kolla-ansible operators currently running Zun should hold off
migration to 2023.1. Any development help would of course be
appreciated. Please coordinate with the Zun project to help with
development.

Details on the dependency problems:

* Currently, Zun has an external dependency on Docker 20.x, and uses a
feature that has been removed in later versions of Docker.
* This Docker feature also relies on an older version of etcd (3.3)
* The mitigation [2] is to pin to the old versions of Docker and etcd,
but this is not possible with Debian Bookworm hosts.

Why this is a problem in Kolla-ansible (and Kolla):

* This can't easily be solved by vendoring in an older version of
etcd: the migration path it introduces to operators would incur a
significant amount of risk.
* Host level involvement is also particularly painful, the host
version of Docker needs to connect to the etcd service running in the
containers.
* Even though the containers can be kept buildable and the
configuration and inventory settings could be maintained, a working
setup is not testable in CI.

Thanks very much for everyone's patience, contributions and time!

[1]: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-13-13.00.log.html
[2]: https://bugs.launchpad.net/zun/+bug/2007142