November 2023 - openstack-discuss

[openstack][kolla-ansible] kolla-ansible stucked when reconfigure
by Nguyễn Hữu Khôi 22 Dec '23

22 Dec '23

Hello guys. I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause. Would you share some experiences on it because my solution causes instances on host to timeout some seconds. Thank you. Nguyen Huu Khoi

2 7

[kolla-ansible][zun] Intention to drop support for Zun for the 2023.1 and path forward
by Jan Gutter 21 Dec '23

21 Dec '23

Hi folks, In yesterday's Kolla IRC meeting we discussed the path forward for Zun [1] and came to the conclusion that it might be better for all involved to drop support for Zun for the 2023.1 release until it can be updated to support the necessary dependencies. More specific details at the end of the message. With this in mind, the plan is to start this process in a week from now, unless a better solution is presented. There is still some time for discussion, please don't hesitate! There are two paths forward after this: If Zun can be updated to support the dependencies during the maintenance timeframe of 2023.1, then support would be restored. If no support is available at the time Caracal releases, however, it will be considered for removal in kolla and kolla-ansible. What this means: Kolla-ansible operators currently running Zun should hold off migration to 2023.1. Any development help would of course be appreciated. Please coordinate with the Zun project to help with development. Details on the dependency problems: * Currently, Zun has an external dependency on Docker 20.x, and uses a feature that has been removed in later versions of Docker. * This Docker feature also relies on an older version of etcd (3.3) * The mitigation [2] is to pin to the old versions of Docker and etcd, but this is not possible with Debian Bookworm hosts. Why this is a problem in Kolla-ansible (and Kolla): * This can't easily be solved by vendoring in an older version of etcd: the migration path it introduces to operators would incur a significant amount of risk. * Host level involvement is also particularly painful, the host version of Docker needs to connect to the etcd service running in the containers. * Even though the containers can be kept buildable and the configuration and inventory settings could be maintained, a working setup is not testable in CI. Thanks very much for everyone's patience, contributions and time! [1]: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-13-13.00.log… [2]: https://bugs.launchpad.net/zun/+bug/2007142

2 2

[all][tc][ptls] Eventlet: Python 3.12, and other concerning discoveries
by Jay Faulkner 20 Dec '23

20 Dec '23

Hi all, For those who aren't aware, eventlet (https://github.com/eventlet/eventlet) is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading. A couple of weeks ago it was brought to my attention, in the #openstack-tc IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12 support for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today. First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817 and a PR up here: https://github.com/eventlet/eventlet/pull/820 to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638 there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12. Furthermore, activity on the eventlet project is extremely slim right now. There have been no commits to the primary branch in nearly eight months and it's been a single maintainer project for an extremely long time. There are also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it. Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing unit tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic database migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed: https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3… . I don't have a specific proposal for a solution. I do know that right now, OpenStack services are built "all-in" on eventlet, with no obvious or easy migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this email to raise an alarm about this situation, and to hope that community members can work together to solve the problem. Thanks for reading and your help, Jay Faulkner OpenStack TC Chair

8 15

[zaqar][tc] master and 2023.2 unit tests are broken
by Takashi Kajinami 18 Dec '23

18 Dec '23

Hello, Today I happened to notice that unit tests in zaqar is broken because some tests requires base test class from oslo.db which was removed in 13.0.0 release[1]. The 13.0.0 release of oslo.db was made during 2023.2 cucle so these tests are also broken in stable/2023.2 branch. In addition, py311 job was made voting in master but is broken and we probably need to fix bindep.txt so that it works in Debian 12. I'm sending this mail so that people maintaining the project is aware of the situation. I'll add [tc] tag because if the CI is not fixed then zaqar may become a new candidate for inactive projects. Thank you, Takashi [1] https://review.opendev.org/c/openstack/oslo.db/+/874239

2 1

Cyborg nova reports mdev-capable resource is not available
by Karl Kloppenborg 07 Dec '23

07 Dec '23

Hi Cyborg Team! Karl from Helm Team. When creating a VM with the correct flavor, the mdev gets created by cyborg agent and I can see it in the nodedev-list --cap mdev. However Nova then fails with: nova.virt.libvirt.driver [<removed>- - default default] Searching for available mdevs... _get_existing_mdevs_not_assigned /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py :8357 2023-09-21 14:34:47.808 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Available mdevs at: set(). 2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] No available mdevs where found. Creating an new one... _allocate_mdevs /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driv er.py:8496 2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] Attempting to create new mdev... _create_new_mediated_device /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py:8385 2023-09-21 14:34:48.455 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Failed to create mdev. No free space found among the following devices: ['pci_0000_4b_03_1', … <truncated list>]. 2023-09-21 14:34:48.456 1901814 ERROR nova.compute.manager [<removed> - - default default] [instance: 2026e2a2-b17a-43ab-adcb-62a907f58b51] Instance failed to spawn: nova.exception.ComputeResourcesUnavailable: Insufficient compute resources: mdev-capable resource is not available. Once this happened, ARQ removes the mdev and cleans up. I’ve got Cyborg 2023.2 running and have a device profile like so: karl@Karls-Air ~ % openstack accelerator device profile show e2b07e11-fe69-4f33-83fc-0f9e38adb7ae +-------------+---------------------------------------------------------------------------+ | Field | Value | +-------------+---------------------------------------------------------------------------+ | created_at | 2023-09-21 13:30:05+00:00 | | updated_at | None | | uuid | e2b07e11-fe69-4f33-83fc-0f9e38adb7ae | | name | VGPU_A40-Q48 | | groups | [{'resources:VGPU': '1', 'trait:CUSTOM_NVIDIA_2235_A40_48Q': 'required'}] | | description | None | +-------------+---------------------------------------------------------------------------+ karl@Karls-Air ~ % I can see the allocation candidate: karl@Karls-Air ~ % openstack allocation candidate list --resource VGPU=1 | grep A40 | 41 | VGPU=1 | 229bf15f-5689-3d2c-b37b-5c8439ea6a71 | VGPU=0/1 | OWNER_CYBORG,CUSTOM_NVIDIA_2235_A40_48Q | karl@Karls-Air ~ % Am I missing something critical here? Because I cannot seem to figure this out… have I got a PCI address wrong, or something? Any help from the Cyborg or Nova teams would be fantastic. Thanks, Karl.

4 9

[openstack-ansible] overriding host vars for compute_hosts
by pi3.14＠tuta.io 06 Dec '23

06 Dec '23

Hey folks, Unfortunately, I am not able to specify host specific variables for the compute hosts. I am trying to specify the `nova_user_ssl_*` variables for each compute node. When these variables are specified in `user_variables.yml` and then specified in `openstack_user_config.yml`, they don't actually end up overriding the variables. But they are recognised correctly in `openstack_inventory.json`. When I remove the variables from `user_variables.yml`, everything works correctly. Is it not possible to specify general variables in `user_variables.yml` and then overwrite them in `openstack_user_config.yml`? Example for `openstack_user_config.yml` (same Problem when using `host_vars` instead of `container_vars`): ``` compute_hosts: compute1: ip: 192.168.1.2 container_vars: nova_user_ssl_cert: "/path/to/cert1" nova_user_ssl_key: "/path/to/key1" nova_user_ssl_ca_cert: "/path/to/ca1" ``` thanks Roby

2 1

[cinder] 2024.1 (Caracal) R-17 virtual mid cycle (6 Dec, 2023)
by Rajat Dhasmana 06 Dec '23

06 Dec '23

Hello Argonauts, We will be conducting the first midcycle of 2024.1 (Caracal) on 6th December, 2023. Following are the details: Date: 6th December, 2023 Time: 1400-1600 UTC Meeting link: https://meet.google.com/auq-jiem-zxy Etherpad: https://etherpad.opendev.org/p/cinder-caracal-midcycles We have 2 weeks so please add your topics timely. Thanks Rajat Dhasmana

1 1

[cinder][nova][glance][ironic] restricting image metadata values to 255 chars
by Jorge Merlino 05 Dec '23

05 Dec '23

Hello stackers, It has come to our attention recently that nova, cinder, and glance treat image properties differently. This came up in a discussion on a patch to extend the allowable length of image metadata values in cinder [0]. Briefly: - glance allows image property values to be unrestricted in the schema, but effectively limited to 65535 bytes by the database. - cinder restricts the length of an image property value to 255 chars (by request schema) if you try to update the volume_image_properties on a volume, but if you create a volume from an image, cinder happily copies over all the image properties from glance, giving them an effective limit of 65535 bytes. - nova mostly truncates image properties values to 255 chars (it's a bit more complicated, but we can consider that to be nova's behavior for the purposes of this discussion). (We're not sure what ironic does, but they're in the subject line as a consumer of images and volumes.) Note that we're talking about metadata *values* here; the metadata keys are already restricted by charset to basically US-ASCII and 255 length. Values are allowed to be unicode. The point of this email is that it would be good for the services to be consistent about image metadata values. Thus, we propose that all services should restrict image metadata values to 255 unicode characters. (Which is sort of happening already, but not consistently.) The further point of this email is to find out what the impact of this restriction would be on current users. So please respond if you have a use case that would be affected by this change, and could not be addressed by, for example, instead of having one enormous value for a single key, the metadata was broken into a number of key/value pairs that would satisfy the 255 char restriction. Thanks for your attention to this! [0] https://review.opendev.org/c/openstack/cinder/+/868485

2 2

A few questions about OpenStack projects archival strategy
by arabat50＠gmail.com 04 Dec '23

04 Dec '23

Dear OpenStack team, I hope this email finds you well. I am Ali Arabat, a PhD student in Software Engineering at École de Technologie Supérieure in Montreal, University of Quebec, Canada, conducting research studies on the evolution of multi-component systems, among which OpenStack plays a significant role. Through this email, we would like to ask a few questions related to our study on OpenStack. Our study revealed a significant increase in the number of cross-component changes over time. However, Such cross-component changes have decreased after archiving a large number of projects beyond 500. Therefore, we would like to have clarification and answers to a few questions as follows: What are the main factors in archiving OpenStack projects? Are there any policies established by OpenStack to archive such projects i.e., end of life? Who generally decides to abandon a given project? We highly value your input and we genuinely appreciate your time. Thank you for considering our request, and we look forward to your valuable input. Best regards, --------------------------------------- Ali Arabat Ph.D. student in Software Engineering Department of Software Engineering and IT École de Technologie Supérieure (ÉTS) | University of Quebec 1100, rue Notre-Dame Ouest, Montréal, QC H3C 1K3, Canada

2 1

[ironic] multi-conductor architecture, standalone and DHCP
by Dmitry Tantsur 01 Dec '23

01 Dec '23

Hi folks, This is detailed context on the discussions we've had recently about multi-conductor in metal3. I'm sending it to potentially interested people and cc'ing the openstack ML. The metal3 side is https://github.com/dtantsur/ironic-operator/issues/3 (although that covers a few more different issues). In metal3 we have Ironic without Neutron (or really any other OpenStack components). So far, we have only supported a single Ironic instance in a cluster, but we want to move towards a multi-conductor setup where every Kubernetes control plane node will host an Ironic (so, 3 Ironics in a normal deployment). The biggest problem stems from DHCP for the provisioning network. With Neutron, Ironic can configure it to direct a node to boot from a specific iPXE server. With Metal3, we have a static DHCP configuration. If we keep it the way it is, it will always direct a node to boot from the iPXE server on the same machine as dnsmasq (regardless of how many dnsmasq replicas we run). This has to be changed. I see several options, none of which are straightforward. (1) iPXE-level fix: boot config API This is what https://bugs.launchpad.net/ironic/+bug/2044561 proposes. If we stop using static iPXE scripts and start going through the Ironic API instead, Ironic can reach out to the correct conductor internally, always serving the right script. (2) dnsmasq DHCP implementation Ironic now has a non-neutron managed DHCP implementation through DHCP host and options files: https://opendev.org/openstack/ironic/src/branch/master/ironic/dhcp/dnsmasq.…. This could be a half of the solution. Since Metal3 does not support new node auto-discovery, we can run 3 copies of dnsmasq with dhcp-ignore=!known and rely on this DHCP provider to configure only 1 dnsmasq for each node. The other 2 will ignore the requests. The problem is in the DHCP range. As far as I understand, the 3 dnsmasqs must be configured with disjoint ranges to prevent leases from clashing. This seems non-trivial with Kubernetes which expects to create identical pods. Adam has suggested using annotations from inside ironic-operator, which could work if we find a way to read them inside the pod after it has started (which seems to pose a chicken-and-egg problem). Other workarounds are possible if we go down this path. E.g. ironic-operator could split the DHCP range and creating a mapping hostIP:subrange,hostIP2:subrange2 and pass it to pods. Then each pod will pick its subrange based on its IP address. Huge downside: every time a Kubernetes control plane node is replaced or changes its IP, the whole Ironic installation will get restarted. (2.1) Pre-created leases Another Metal3 contributor has suggested requiring an IPAM implementation (metal3 actually has one) that will create leases centrally, allowing us to use the same DHCP range. The only problem: how to pass them to dnsmasq in runtime without restarting all pods? We could probably have a small sidecar container next to dnsmasq that watches the Kubernetes API for new leases in IPAM. Or... (2.2) Dynamic DHCP and Kea The same contributor is working on Kea support for their deployment: https://hackmd.io/@7RNIJtmvSIeNpSOMTW9oqA/SkWhyr7Hp. There seem to be limitations in the free version, which caused them to write a tiny downstream plugin: https://gitlab.com/Orange-OpenSource/kanod/kanod-kea/-/blob/controlled-leas…. Changing the DHCP server is a dramatic step, I'm not sure if our small community can afford that. It may also pose problems for downstreams like OpenShift, requiring Metal3 to support both servers in the long run. Any thoughts are welcome. Dmitry

3 3