December 2023 - openstack-discuss

[kolla] [2023.1] [masakari] Orphaned instances on node shutdown
by Rob Jefferson 11 Dec '23

11 Dec '23

Folks, I'm running into an issue with instances appearing to be running, even though they shouldn't be. The environment in this case is a virtualized OpenStack Kolla (2023.1) cluster, with three compute nodes and three control plane nodes. I am trying to see what happens when I create and terminate instances, in an attempt to figure out failover behavior. The three computes are in a single segment. I created four instances: +--------------------+--------+---------------------------+ | Name | Status | Host | +--------------------+--------+---------------------------+ | rj_testinstance_04 | ACTIVE | devtest-libvirt1-muon1005 | | rj_testinstance_03 | ACTIVE | devtest-libvirt1-muon1004 | | rj_testinstance_02 | ACTIVE | devtest-libvirt1-muon1006 | | rj_testinstance | ACTIVE | devtest-libvirt1-muon1005 | +--------------------+--------+---------------------------+ I decided to see what would happen if I went in and shutdown devtest-libvirt1-muon-1005 (which has two instances running). So I ssh'd into the machine and entered (as root): `echo c > /proc/sysrq-trigger` to force a crash, which duly occurred. I then made sure the host was seen as down: +---------------------------+-------+ | Hypervisor Hostname | State | +---------------------------+-------+ | devtest-libvirt1-muon1005 | down | | devtest-libvirt1-muon1004 | up | | devtest-libvirt1-muon1006 | up | +---------------------------+-------+ But even after this, the instances seem to still be up, including on the now crashed machine: +--------------------+--------+---------------------------+ | Name | Status | Host | +--------------------+--------+---------------------------+ | rj_testinstance_04 | ACTIVE | devtest-libvirt1-muon1005 | | rj_testinstance_03 | ACTIVE | devtest-libvirt1-muon1004 | | rj_testinstance_02 | ACTIVE | devtest-libvirt1-muon1006 | | rj_testinstance | ACTIVE | devtest-libvirt1-muon1005 | +--------------------+--------+---------------------------+ I would have expected the instances to have been regenerated on other compute hosts, or some sort of error. The nodes are on the same segment, so Masakari should have at least attempted to roll over the instances to other computes in the segment. But checking the instance details suggests that nothing has happened. The instances are just... lost. The Masakari host monitor log file reflects reality: 2023-12-11 19:21:49.953 8 INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'devtest-libvirt1-muon1005' is 'offline' (current: 'offline'). Needless to say, I'm very confused here and am sure what's going on. Any advice would help. Thanks, Rob

1 1

[neutron][ovn] Outbound SNAT behavior
by James Denton 11 Dec '23

11 Dec '23

Hi everyone, While looking at an OVS->OVN migration, we came across a use-case that doesn’t provide 1:1 compatibility and I wanted to check with the community to see if there were any suggestions or ideas on how to proceed. In an OVS-based environment, we have users with a typical layout: <external provider network> | <neutron router> | | <tenantnet1> <tenantnet2> The users have implemented static routes on the Neutron router using the extraroutes API, which works great, and the default SNAT behavior implemented within the qrouter namespace handles any network traffic egressing the router, not just directly-connected tenant networks (see iptables entries below): -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat -A neutron-l3-agent-snat -o qg-700f84b0-6b -j SNAT --to-source 168.47.160.223 --random-fully -A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 168.47.160.223 --random-fully -A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat OVN, on the other hand, appears to implement snat rules in the logical router based on directly-connected tenant networks, which means that traffic sourcing from other networks (not managed by Neutron but routable within a Neutron router, including to a VM or appliance) is not source NAT’d and egresses the router as its real IP. This behavior results in the traffic being effectively blackholed, as there’s no return route and no real desire to have exposed those IPs anyway. Here’s an example of the nat list which includes the directly connected tenant network: root@lab-controller01:~# ovn-nbctl lr-nat-list neutron-edfd7115-5a81-46b7-9692-fe227d395b0b TYPE GATEWAY_PORT EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT snat 192.168.3.33 10.254.254.0/24 The extraroute in the router is: destination: 172.16.0.0/24, nexthop: 10.254.254.6 I can add an entry manually, which allows the snat for the secondary network to work as expected, but results in some other issues where the snat(s) don’t get added back on a disable/enable of the router: root@lab-controller01:~# ovn-nbctl lr-nat-add neutron-edfd7115-5a81-46b7-9692-fe227d395b0b snat 192.168.3.33 172.16.0.0/24 root@lab-controller01:~# ovn-nbctl lr-nat-list neutron-edfd7115-5a81-46b7-9692-fe227d395b0b TYPE GATEWAY_PORT EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT snat 192.168.3.33 10.254.254.0/29 snat 192.168.3.33 172.16.0.0/24 I understand this is probably not a supported usecase, but one that worked nonetheless, so any suggestions on how to mirror this functionality in a supported way would be greatly appreciated. Thanks, James

1 0

[tc] Technical Committee Meeting Tomorrow, Dec 12
by Jay Faulkner 11 Dec '23

11 Dec '23

Hi all, The weekly meeting of the Technical Committee will be held Tuesday, Dec 12 at 1800 UTC in #openstack-tc on OFTC. See below for the proposed agenda: - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-12-05-18.00.html - Gmann - Create TC resolution expressing approval for OIF bylaws change - Create TC charter change in response to OIF bylaws changes - JayF - Communicate deadline for approval of above OIF bylaws related governance changes - Gate health check - Leaderless projects (gmann) - Implementation of Unmaintained branch statuses - 2024.1 TC Tracker - https://etherpad.opendev.org/p/tc-2024.1-tracker - Open Discussion and Reviews Thanks all, Jay Faulkner TC Chair

1 0

[kolla-ansible] rocky spice containers missing
by Michael Knox 11 Dec '23

11 Dec '23

Hi all. I was looking to use SPICE instead of VNC, but noticed that there are no rocky containers for it on quay.io. I found a ticket for CentOS 8 source, is this still the same reason? https://bugs.launchpad.net/kolla-ansible/+bug/1955549 Cheers Michael

3 3

[neutron] Bug Deputy Report December 4 - December 10
by Lajos Katona 11 Dec '23

11 Dec '23

Hi Neutrinos, I was the Neutron bug deputy last week, and I send the report of last week's bugs. - High prio bugs - Unassigned: https://bugs.launchpad.net/neutron/+bug/2045549 (OVS jobs randomly fails as Guest VMs not(or delayed) configured with DHCP) - https://bugs.launchpad.net/neutron/+bug/2045725 (scenario jobs randomly fails downloading ubuntu image) - related devstack patch: https://review.opendev.org/c/openstack/devstack/+/901699 - Medium prio bugs: - https://bugs.launchpad.net/neutron/+bug/2045811 (neutron-ovn-db-sync-util can fail with KeyError) - https://review.opendev.org/c/openstack/neutron/+/902841 - https://bugs.launchpad.net/neutron/+bug/2045889 ([OVN] ML2/OVN mech driver does not set the OVS bridge name in the port VIF details) - related mail thread: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - https://bugs.launchpad.net/neutron/+bug/2045950 (Security group performance issue for iptables driver due to "stateless feature") - performance issue with stateless security-groups. Best wishes Lajos Katona (lajoskatona)

1 0

[all] Ussuri and Train EOL
by Elõd Illés 11 Dec '23

11 Dec '23

Hi, With the new TC resolution '2023-07-24 Unmaintained status replaces Extended Maintenance' [1] the last 3 EM branches (Xena, Wallaby, Victoria) will transition to Unmaintained, but older branches (Ussuri, Train) should move to End of Life unless someone steps up as maintainer of this branch for this project. (In that case the branch of that project will transition to unmaintained/{ussuri,train}) To signal this, please add a -1 for the given generated EOL patch [2][3] if you or the team want to keep this branch open for further maintenance. If there is no response from anyone within a reasonable time frame, then PTLs/release liaisons will be asked to confirm the EOL transition with their +1. Thanks, Előd Illés irc: elodilles @ #openstack-stable / #openstack-release [1] https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… [2] https://review.opendev.org/q/topic:train-eol [3] https://review.opendev.org/q/topic:ussuri-eol

1 0

[docs]OpenStack compatible list
by steven.wang＠gigacomputing.com 11 Dec '23

11 Dec '23

Hi, For OpenStack compatible check, how should we know the connection with base OS? Example; If the machine support RHEL9 and OpenStack Wallaby, can we say it is compatible to the newest OpenStack than Wallaby? So it should be no problem with Xena, Yoga, Zed, Antelope and Bobcat? Or please kindly let me know where can find the similar document? Thanks. Steven

2 2

System configuration in OpenStack
by lostghost1 10 Dec '23

10 Dec '23

Hi all, Sorry if this has been asked already, but is it feasible to use OpenStack to configure running systems - including virtual instances, containers, and the physical nodes themselves. I know that conceptually OpenStack takes a hands-off approach - it deals with images in abstract and further configuration is delegated to tools such as Ansible. Same for bare metal nodes - you provision them with Ironic, flash an image to them, and that's where configuaration ends for OpenStack. At least this is how I understand it, may be wrong. I was thinking about introducing systems configuration as a first-class concept into OpenStack - that means integration with the package manager, writing of config files from templates, filesystem management in general. The most well thought-out "solution" that there is for systems configuration is Nix in my opinion. I could intergrate it into OpenStack, I could write my own package manager. I haven't decided yet, these are just ideas for a pet project. So if I were to do such a thing, how would I go about achieving it? Obviously I would need to write a Horizon console module for the web UI. But beyond that I'd also need integration with other components. What do you all think?

1 0

Redundancy and high availability of basic services...
by Jorge Visentini 09 Dec '23

09 Dec '23

Hello everybody. I've been studying for some time now, reading official and third-party documentation about deploying and day-to-day maintenance of openstack. What I don't usually read about is the redundancy and high availability of basic services, such as the database, keystone, neutron... I wanted to better understand how it works in a small deployment scenario... I mean... 5 or 10 nodes and 3 controllers, for example... Here we use block storage fiber channel, so we won't use ceph... Of course, I won't implement it in production overnight, but I would like to hahaha but I would like to better understand this part of service redundancy. Thank you all for the emails sent every day. I learn a lot from them. -- Att, Jorge Visentini +55 55 98432-9868

1 0

[all] [ironic] Heads-up: another eventlet-related issue on Stream 8
by Dmitry Tantsur 08 Dec '23

08 Dec '23

Hi stackers, I've noticed this some time ago, but somehow decided that it wouldn't affect OpenStack upstream. Early sign of burn-out? Anyway... Red Hat has fixed CVE 2023-40217 in its Python 3.6 packages (used by default on CentOS Stream 8). Unfortunately, it was done in a way that breaks TLS in eventlet (more details in https://issues.redhat.com/browse/OCPBUGS-20486) It manifests in the following traceback: Traceback (most recent call last): File "/usr/lib64/python3.6/ssl.py", line 754, in __init__ self.getpeername() OSError: [Errno 107] Transport endpoint is not connected During handling of the above exception, another exception occurred: Traceback (most recent call last): <... SNIP ...> File "/usr/lib/python3.6/site-packages/oslo_service/wsgi.py", line 173, in start self.dup_socket = sslutils.wrap(self.conf, self.dup_socket) File "/usr/lib/python3.6/site-packages/oslo_service/sslutils.py", line 104, in wrap return ssl.wrap_socket(sock, **ssl_kwargs) # nosec File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 422, in wrap_socket return GreenSSLSocket(sock, *a, **kw) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 117, in __init__ ca_certs, do_handshake_on_connect and six.PY2, *args, **kw) File "/usr/lib64/python3.6/ssl.py", line 759, in __init__ blocking = (self.gettimeout() != 0) File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 145, in gettimeout return self._timeout AttributeError: 'GreenSSLSocket' object has no attribute '_timeout' The fix should be rather simple IMO: just remove the condition on Python 2 from here: https://github.com/eventlet/eventlet/blob/master/eventlet/green/ssl.py#L108. But I'm not sure, given the state of eventlet community, we can land anything these. The issue affects, for example. ironic-python-agent on stable/yoga. We're considering our options, the most likely is to use a non-default Python, which is 3.9. Xena and older did not support 3.9, so they're probably out in the cold... Thoughts, ideas? Dmitry

3 5