[kolla-ansible][zun] Intention to drop support for Zun for the 2023.1 and path forward
Hi folks,
In yesterday's Kolla IRC meeting we discussed the path forward for Zun [1] and came to the conclusion that it might be better for all involved to drop support for Zun for the 2023.1 release until it can be updated to support the necessary dependencies. More specific details at the end of the message.
With this in mind, the plan is to start this process in a week from now, unless a better solution is presented. There is still some time for discussion, please don't hesitate!
There are two paths forward after this:
If Zun can be updated to support the dependencies during the maintenance timeframe of 2023.1, then support would be restored.
If no support is available at the time Caracal releases, however, it will be considered for removal in kolla and kolla-ansible.
What this means:
Kolla-ansible operators currently running Zun should hold off migration to 2023.1. Any development help would of course be appreciated. Please coordinate with the Zun project to help with development.
Details on the dependency problems:
* Currently, Zun has an external dependency on Docker 20.x, and uses a feature that has been removed in later versions of Docker. * This Docker feature also relies on an older version of etcd (3.3) * The mitigation [2] is to pin to the old versions of Docker and etcd, but this is not possible with Debian Bookworm hosts.
Why this is a problem in Kolla-ansible (and Kolla):
* This can't easily be solved by vendoring in an older version of etcd: the migration path it introduces to operators would incur a significant amount of risk. * Host level involvement is also particularly painful, the host version of Docker needs to connect to the etcd service running in the containers. * Even though the containers can be kept buildable and the configuration and inventory settings could be maintained, a working setup is not testable in CI.
Thanks very much for everyone's patience, contributions and time!
[1]: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-13-13.00.log.... [2]: https://bugs.launchpad.net/zun/+bug/2007142
Hi,
I am from Zun team. The goal is to fix it in "C" release. I will keep the status updated in this ticket: https://bugs.launchpad.net/zun/+bug/2007142 . Sorry for all the inconvenience.
Best regards, Hongbin
On Thu, Sep 14, 2023 at 6:56 PM Jan Gutter openstack@jangutter.com wrote:
Hi folks,
In yesterday's Kolla IRC meeting we discussed the path forward for Zun [1] and came to the conclusion that it might be better for all involved to drop support for Zun for the 2023.1 release until it can be updated to support the necessary dependencies. More specific details at the end of the message.
With this in mind, the plan is to start this process in a week from now, unless a better solution is presented. There is still some time for discussion, please don't hesitate!
There are two paths forward after this:
If Zun can be updated to support the dependencies during the maintenance timeframe of 2023.1, then support would be restored.
If no support is available at the time Caracal releases, however, it will be considered for removal in kolla and kolla-ansible.
What this means:
Kolla-ansible operators currently running Zun should hold off migration to 2023.1. Any development help would of course be appreciated. Please coordinate with the Zun project to help with development.
Details on the dependency problems:
- Currently, Zun has an external dependency on Docker 20.x, and uses a
feature that has been removed in later versions of Docker.
- This Docker feature also relies on an older version of etcd (3.3)
- The mitigation [2] is to pin to the old versions of Docker and etcd,
but this is not possible with Debian Bookworm hosts.
Why this is a problem in Kolla-ansible (and Kolla):
- This can't easily be solved by vendoring in an older version of
etcd: the migration path it introduces to operators would incur a significant amount of risk.
- Host level involvement is also particularly painful, the host
version of Docker needs to connect to the etcd service running in the containers.
- Even though the containers can be kept buildable and the
configuration and inventory settings could be maintained, a working setup is not testable in CI.
Thanks very much for everyone's patience, contributions and time!
Hi all,
The problem is resolved at zun side. I have created a few patches to re-enable the zun role in kolla-ansible: https://review.opendev.org/c/openstack/kolla-ansible/+/904093 https://review.opendev.org/c/openstack/kolla-ansible/+/904164 https://review.opendev.org/c/openstack/kolla-ansible/+/904094
For people who want to know the technical details, zun pinned docker 20 because the latest version of docker dropped support for the --cluster-store option [1]. In before, zun used kuryr-libnetwork (a docker network plugin) with "global" capability scope which relied on the --cluster-store option. Right now, zun has switched to the "local" capability scope so the problem has been worked around. The related patches: https://review.opendev.org/c/openstack/zun/+/903884 https://review.opendev.org/c/openstack/zun/+/903872 https://review.opendev.org/c/openstack/kuryr-libnetwork/+/903925 https://review.opendev.org/c/openstack/kuryr-libnetwork/+/877180
With above fixes, zun no longer required docker 20 and etcd. The problem from kolla side should be resolved.
Best regards, Hongbin
[1] https://github.com/moby/moby/issues/40383
On Fri, Sep 15, 2023 at 12:04 AM Hongbin Lu hongbin034@gmail.com wrote:
Hi,
I am from Zun team. The goal is to fix it in "C" release. I will keep the status updated in this ticket: https://bugs.launchpad.net/zun/+bug/2007142 . Sorry for all the inconvenience.
Best regards, Hongbin
On Thu, Sep 14, 2023 at 6:56 PM Jan Gutter openstack@jangutter.com wrote:
Hi folks,
In yesterday's Kolla IRC meeting we discussed the path forward for Zun [1] and came to the conclusion that it might be better for all involved to drop support for Zun for the 2023.1 release until it can be updated to support the necessary dependencies. More specific details at the end of the message.
With this in mind, the plan is to start this process in a week from now, unless a better solution is presented. There is still some time for discussion, please don't hesitate!
There are two paths forward after this:
If Zun can be updated to support the dependencies during the maintenance timeframe of 2023.1, then support would be restored.
If no support is available at the time Caracal releases, however, it will be considered for removal in kolla and kolla-ansible.
What this means:
Kolla-ansible operators currently running Zun should hold off migration to 2023.1. Any development help would of course be appreciated. Please coordinate with the Zun project to help with development.
Details on the dependency problems:
- Currently, Zun has an external dependency on Docker 20.x, and uses a
feature that has been removed in later versions of Docker.
- This Docker feature also relies on an older version of etcd (3.3)
- The mitigation [2] is to pin to the old versions of Docker and etcd,
but this is not possible with Debian Bookworm hosts.
Why this is a problem in Kolla-ansible (and Kolla):
- This can't easily be solved by vendoring in an older version of
etcd: the migration path it introduces to operators would incur a significant amount of risk.
- Host level involvement is also particularly painful, the host
version of Docker needs to connect to the etcd service running in the containers.
- Even though the containers can be kept buildable and the
configuration and inventory settings could be maintained, a working setup is not testable in CI.
Thanks very much for everyone's patience, contributions and time!
participants (2)
-
Hongbin Lu
-
Jan Gutter