- openstack-discuss - lists.openstack.org

[openstack-helm] health-probes timeout for neutron agents (l3 and dhcp) and nova compute
by Alberto Molina Coballes 27 Mar '25

27 Mar '25

Hi, We’re facing an issue when deploying OpenStack 2024.1 with OpenStack Helm and OVS, the health probes fail for neutron-l3-agent, neutron-dhcp-agent and nova-compute while the internal RPC communication between the nova and neutron components look fine: Neutron: openstack network agent list +--------------------------------------+--------------------+--------------------------------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+--------------------------------+-------------------+-------+-------+---------------------------+ | 433c2b07-5c82-4545-991d-6d0de6044ae9 | DHCP agent | fig-virt-intdev-alberto-node-0 | nova | :-) | UP | neutron-dhcp-agent | | 56c050fa-e58e-46a6-8191-0d0d481de246 | Open vSwitch agent | fig-virt-intdev-alberto-node-0 | None | :-) | UP | neutron-openvswitch-agent | | 5ea28388-3fac-4ebb-bce4-6b210f8a3766 | Metadata agent | fig-virt-intdev-alberto-node-0 | None | :-) | UP | neutron-metadata-agent | | 6a4506ca-6fe4-489a-b370-6e426db40d50 | DHCP agent | fig-virt-intdev-alberto-node-1 | nova | :-) | UP | neutron-dhcp-agent | | 772a1d3a-b432-4ac3-8dfe-d3affc6f226d | Metadata agent | fig-virt-intdev-alberto-node-1 | None | :-) | UP | neutron-metadata-agent | | 7bd8360c-468a-4473-a470-abfb1ab435da | Open vSwitch agent | fig-virt-intdev-alberto-node-1 | None | :-) | UP | neutron-openvswitch-agent | | 9d362a63-5740-44ed-ac77-4548eee59205 | L3 agent | fig-virt-intdev-alberto-node-0 | nova | :-) | UP | neutron-l3-agent | | a2cc9dc4-1600-4c5d-a250-2e43cf062cf3 | L3 agent | fig-virt-intdev-alberto-node-1 | nova | :-) | UP | neutron-l3-agent | +--------------------------------------+--------------------+--------------------------------+-------------------+-------+-------+---------------------------+ The neutron-l3 and neutron-dhcp pods are not ready: k -n openstack get pods -l application=neutron |grep 'l3\|dhcp' neutron-dhcp-agent-default-fw5rm 0/1 Running 0 127m neutron-dhcp-agent-default-vh7c4 0/1 Running 0 127m neutron-l3-agent-default-4h5tp 0/1 Running 0 24m neutron-l3-agent-default-bdngv 0/1 Running 0 24m Because the health probes fail: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 4m38s (x119 over 127m) kubelet Readiness probe failed: Health probe timed out. Agent is down or response timed out When trying to execute the health probe manually from the container, the probe fails with a timeout after 60 seconds (RPC_PROBE_TIMEOUT=60) neutron@fig-virt-intdev-alberto-node-1:/$ python /tmp/health-probe.py --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --agent-queue-name dhcp_agent --use-fqdn Health probe timed out. Agent is down or response timed out The situation for nova is quite similar: openstack compute service list +--------------------------------------+----------------+---------------------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +--------------------------------------+----------------+---------------------------------+----------+---------+-------+----------------------------+ | eaf4684c-88a4-47dd-b340-b7dbebf148ba | nova-conductor | nova-conductor-679d57f977-zbq92 | internal | enabled | up | 2025-03-27T13:11:33.000000 | | 9164d63f-608c-44cc-8a92-2f2bac07c622 | nova-scheduler | nova-scheduler-845f87b9b5-vq9p5 | internal | enabled | up | 2025-03-27T13:11:36.000000 | | 464cd394-8b26-49ad-8c82-f014ca00ec7e | nova-compute | fig-virt-intdev-alberto-node-1 | nova | enabled | up | 2025-03-27T13:11:33.000000 | | aa7e5f6d-2533-4953-bb8d-9f9bfb7fc6bd | nova-compute | fig-virt-intdev-alberto-node-0 | nova | enabled | up | 2025-03-27T13:11:33.000000 | +--------------------------------------+----------------+---------------------------------+----------+---------+-------+----------------------------+ And the nova-compute pods are not ready because the health probes fail: k -n openstack get pods -l component=compute NAME READY STATUS RESTARTS AGE nova-compute-default-t8w7x 1/2 Running 1 (70m ago) 3h31m nova-compute-default-vz7nf 1/2 Running 1 (70m ago) 3h31m To be sure that the OpenStack RPC is working fine, the health probes have been deleted for nova-compute, then nova-cell-setup is executed and the hypervisors are ready. Some VMs have been launched in this environment successfully (nova-compute), getting IP from the DHCP agent or adding floating IPs (l3-agent) and the OpenStack behaviour is normal. So I’m confused about the health-probes failing. Any idea on what can be wrong or what to check is welcome. Thanks! Alberto Note: This is a Kubernetes cluster running on two VMs (OpenStack). This virtual environment has been redeployed several times with some modifications and the issue is fully reproducible: the probes for nova-compute, neutron-l3-agent and neutron-dhcp-agent always fail.

1 0

[Glance]Cancelling Weekly Meeting (March 27)
by Mridula Joshi 27 Mar '25

27 Mar '25

Hello, Cancelling today's weekly meeting, As most of the members are not around. Apologies for the last min update. Thanks, Mridula Joshi

1 0

[designate] multiple zone catalogs with multiple zones
by Benedikt Trefzer 26 Mar '25

26 Mar '25

Hi List I'm trying to create an additional pool with a separate catalog zone. For the default pool, I can do an AXFR to the catalog domain, this works (either without tsig key or with a tsig). For the catalog of the second pool cannot be queried (Log sais Zone not found). In the DB I can see the zone. Questions: - Is it possible to have multiple pools with separate catalogs, does anyone have a running setup ? - Reading this [1] line in the code, I assume It's not possible having multiple zones without using tsig, correct ? Thanks a lot. Benedikt [1] https://opendev.org/openstack/designate/src/branch/master/designate/mdns/ha…

1 1

Nova features
by engineer2024 26 Mar '25

26 Mar '25

Where can I find the nova feature requests which are yet to be developed ? I am thinking of contributing to adding new features.

3 3

[largescale-sig] Next meeting: March 26, 16utc
by Thierry Carrez 26 Mar '25

26 Mar '25

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 16UTC to facilitate America timezones participation. In these troubled DST times, you should doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20250326T16 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

2 1

#eventlet-removal Flamingo PTG
by Herve Beraud 26 Mar '25

26 Mar '25

Hello folks, We’re meeting again on the Project Teams Gathering in April (7-11th). The slot for the Eventlet Removal Initiative PTG is: Tuesday (Apr 8) 1400-1600 UTC The etherpad is: https://etherpad.opendev.org/p/apr2025-ptg-eventlet Please feel free to insert your topic proposals. Additional details can be found at https://ptg.opendev.org/ Regards, -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[release][ptl][tc] OpenInfra Live for OpenStack 2025.1 ‘Epoxy’
by helena＠openinfra.dev 26 Mar '25

26 Mar '25

Hi everyone, Following the release of OpenStack 2025.1 ‘Epoxy’ next Wednesday, we'd like to go ahead and schedule the OpenInfra Live for Thursday, April 3 at 1400 UTC. If you would like to represent your project team’s updates in the Epoxy release as well as their plans for Flamingo, please let me know. If you are not familiar, you can find reference examples like 2024.2 Dalmation [1] and 2024.1 Caracal [2]. [1] https://www.youtube.com/watch?v=6igJNIJ9yFE [2] https://www.youtube.com/watch?v=3BgAXbFaVqw&t=1707s Thanks! Helena helena(a)openinfra.dev Community Programs Manager The OpenInfra Foundation [openinfraLogo-sm.png]

3 2

[neutron][ops][upgrade] l3-agent failure during upgrade to Zed
by Eugen Block 25 Mar '25

25 Mar '25

Hi *, maybe I missed some announcement or something, but usually, I read the release notes [0] before upgrading our OpenStack cloud. I didn't notice anything regarding DB schema upgrades. And after the upgrade from Yoga to Zed in a test environment went well, I tried the same in our production today. Note that I didn't have a router in my test cloud, so that's probably why I didn't notice anything. Unfortunately, there has been a schema change, that's why the l3-agent failed to start properly with this error: 2025-03-21 12:29:14.527 846393 CRITICAL neutron [None req-e225ff0a-82e1-473b-9eba-9a11caa7ace7 - - - - - -] Unhandled error: oslo_messaging.rpc.client.RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1054, "Unknown column 'portforwardings.external_port' in 'SELECT'") Indeed, the upgraded control node didn't have "external_port" anymore in /usr/lib/python3/dist-packages/neutron/db/models/port_forwarding.py, while the not yet upgraded control node did. So the situation could only be resolved by proceeding with the upgrade. But that meant an interruption for our virtual routers, causing floating IPs to be unreachable for a couple of minutes. Note that we're using highly-available routers. I thought about setting "no-ha" for each router, but that can only be done for disabled routers, which is not an option, of course. And it doesn't really fit into the "rolling upgrade" concept, which has worked great so far. Since we moved to Ubuntu last September (while still on Victoria), we've been able to upgrade to Yoga without any issues. And while the interruption today was not too critical, I was still surprised that such an important change didn't even make it into the Zed release notes. Was that a mistake or did I miss something? Are there other places I need to check before attempting an upgrade? Thanks, Eugen [0] https://docs.openstack.org/releasenotes/neutron/zed.html

3 9

[dev] New contributing organization help
by Dominik Danelski 25 Mar '25

25 Mar '25

Hello community, I work at a silver member of OpenInfra and will make code contributions on behalf of my employer. I am aware about the requirement to sign Contributor License Agreement listing me as the responsible person. Other than that, are there any additional requirements on identifying me as a corporate developer? For example contributor guide lists creating INDIVIDUAL Member account as one of the setup steps and it got me a bit worried. Kind regards Dominik Danelski @ CloudFerro

4 3

[kolla] Flamingo PTG
by Michał Nasiadka 25 Mar '25

25 Mar '25

Hello Koalas, We’re meeting again on the Project Teams Gathering in April (7-11th). The slots for Kolla/Kolla-Ansible/Kayobe PTG are: Tuesday - 07:00-08:00 UTC (general, Kolla and Kolla-Ansible) Wednesday - 07:00-08:00 UTC (general, Kolla and Kolla-Ansible) Friday - 14:00-15:00 UTC (Kayobe) If there’s a need of an additional afternoon (UTC) slot - please let us know. The ether pad is at the usual location: https://etherpad.opendev.org/p/kolla-flamingo-ptg Please insert your topic proposals and see you on the PTG. Additional details can be found at https://ptg.opendev.org/ Cheers, Michal

1 0