August 2023 - openstack-discuss - lists.openstack.org

[kolla-ansible][tooz][devstack] Upgrade path to etcd v3.4
by Jan Gutter 15 Aug '23

15 Aug '23

Hi folks, etcd maintains two stable branches [1], at the moment these appear to be the v3.4 and v3.5 series. etcd does not support skip-version updates without dataloss[2] out of the box, operators are encouraged to do online updates. I took a quick sample of some etcd versions: * devstack currently uses etcd v3.3.12 (at time of writing) [3] * kolla uses etcd v3.3.27 [4] * tripleo uses rdo etcd: it appears they're already on 3.4.14 [5] * upstream etcd is at 3.4.27 and 3.5.9 [6] Now, the bad news: The client side endpoints of etcd is changing from `v3alpha` to `v3`, with a `v3beta` step added to ensure maximum confusion [7] tooz, for example, currently defaults to `v3alpha` for the recommended etcd3gw backend [8], but it's configurable in the client side by passing an extra option to the backend url. So just updating etcd from 3.3 to 3.4 usually breaks things until a couple of orchestration updates are made. It seems providing a smooth upgrade path would require coordination between orchestration, services that depend on etcd as a backend, and a couple of middleware libraries. The lack of a skip-version upgrade support means it's probably also better to do this sooner rather than later. 3.4 is already late in its release cycle and due to go out of maintenance soon. Options: * upgrade to 3.4 and use `v3` endpoint everywhere - fix forward? * try to detect endpoints in middleware (etcd3gw) somehow? * ask ChatGPT how to farm goats Links (not permalinks!): [1]: https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/b… [2]: https://etcd.io/docs/v3.3/upgrades/upgrade_3_4/ [3]: https://opendev.org/openstack/devstack/src/branch/master/stackrc#L723 [4]: https://opendev.org/openstack/kolla/src/branch/master/docker/etcd/Dockerfil… [5]: https://review.rdoproject.org/r/plugins/gitiles/rdoinfo/+/master/buildsys-t… [6]: https://github.com/etcd-io/etcd/releases/ [7]: https://etcd.io/docs/v3.5/dev-guide/api_grpc_gateway/ [8]: https://opendev.org/openstack/tooz/src/branch/master/tooz/drivers/etcd3gw.p…

2 2

[tc] Technical Committee next weekly meeting Tomorrow on August 15, 2023
by Nikolla, Kristi 14 Aug '23

14 Aug '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, Aug 15, 2023 at 1800 UTC on #openstack-tc on OFTC IRC. Please propose items to the agenda by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting As of now, the agenda for tomorrow's meeting is • Roll call • Follow up on past action items • rosmaita to review guidelines patch and poke at automating it • Gate health check • Testing runtime for 2024.1 release • https://review.opendev.org/c/openstack/governance/+/891225 • User survey question updates by Aug 18 • https://lists.openstack.org/pipermail/openstack-discuss/2023-July/034589.ht… • Open Discussion and Reviews • https://review.opendev.org/q/projects:openstack/governance+is:open • https://review.opendev.org/c/openstack/project-team-guide/+/843457 • https://review.opendev.org/c/openstack/governance/+/888771 Thank you, Kristi Nikolla

1 0

LDAP failover fails
by Albert Braden 14 Aug '23

14 Aug '23

We are experiencing the LDAP failover issue described in [1]. Redhat’s solution is to not bother fixing the bug, and to tell customers to put the LDAP server behind a load-balancer. According to Redhat, that is not a good solution for FreeIPA, as explained in [2] and further elucidated in the blog post [3] that it references. I see that the community has a bug open for this [4] and the bug is being worked on here [5] but there has been no activity since 10/22. What is the status of this bugfix? Does it just need someone to review and merge it, or is there more work to be done? How are other FreeIPA users working around this problem? [1] https://bugzilla.redhat.com/show_bug.cgi?id=2024602#c3 [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… [3] http://ssimo.org/blog/id_019.html [4] https://bugs.launchpad.net/keystone/+bug/1953622 [5] https://review.opendev.org/c/openstack/keystone/+/821086

2 4

[openstack][neutron] it is good if we set fixed ip and set ip in instance netplan also
by Nguyễn Hữu Khôi 13 Aug '23

13 Aug '23

Hello guys. Sometimes I see that my instances won't get IP from Neutron DHCP Agent. Is it good if we set a fixed port for instance and set a static ip in the instance's netplan configuration also? Nguyen Huu Khoi

5 10

[all][elections][ptl][tc] Combined PTL/TC 2024.1 cycle Election Season
by Ian Y. Choi 11 Aug '23

11 Aug '23

Election details: https://governance.openstack.org/election/ The nomination period officially begins Aug 16, 2023 23:45 UTC. Please read the stipulations and timelines for candidates and electorate contained in these governance documents[1][2][3]. Due to circumstances of timing, PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. Similarly, there will only be a TC poll if there are more than 4 candidates stepping forward the available TC positionss. All members of the community are encouraged to confirm their email address as described at: https://governance.openstack.org/election/#electorate There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you which to discuss in private please email any of the election officials[4] so that we may address your concerns. Thank you, [1] https://governance.openstack.org/tc/reference/charter.html [2] https://governance.openstack.org/tc/reference/role-of-the-tc.html [3] https://docs.openstack.org/project-team-guide/ptl.html [4] https://governance.openstack.org/election/#election-officials

1 0

[release] Release countdown for week R-7, Aug 14-18
by Elõd Illés 11 Aug '23

11 Aug '23

Development Focus ----------------- We are entering the last weeks of the 2023.2 Bobcat development cycle. From now until the final release, we'll send a countdown email like this every week. It's probably a good time for teams to take stock of their library and client work that needs to be completed yet. The non-client library freeze is coming up, followed closely by the client lib freeze. Please plan accordingly to avoid any last minute rushes to get key functionality in. General Information ------------------- Next week is the Extra-ATC freeze, in preparation for elections. All contributions to OpenStack are valuable, but some are not expressed as Gerrit code changes. Please list active contributors to your project team who do not have a code contribution this cycle, and therefore won't automatically be considered an Active Technical Contributor and allowed to vote. This is done by adding extra-atcs to https://opendev.org/openstack/governance/src/branch/master/reference/projec… before the Extra-ATC freeze on August 17th, 2023. A quick reminder of the upcoming freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (August 24th, 2023). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (August 31st, 2023) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, August 31st, 2023. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (September 14th) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (September 28th) Finally, now is also a good time to start planning what highlights you want for your deliverables in the cycle highlights. The deadline to submit an initial version for those is set to Feature freeze (August 31st, 2023). Background on cycle-highlights: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125613.html Project Team Guide, Cycle-Highlights: https://docs.openstack.org/project-team-guide/release-management.html#cycle… knelson [at] openstack.org/diablo_rojo on IRC is available if you need help selecting or writing your highlights Upcoming Deadlines & Dates -------------------------- Extra-ATC freeze: August 17th, 2023 (R-7 week) Non-client library freeze: August 24th, 2023 (R-6 week) Client library freeze: August 31st, 2023 (R-5 week) Bobcat-3 milestone: August 31st, 2023 (R-5 week) Final 2023.2 Bobcat release: October 4th, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[OSA] CEPH libvirt secrets
by Murilo Morais 11 Aug '23

11 Aug '23

Good evening everyone! Guys, I'm having trouble adding a second CEPH cluster for Nova/Cincer to consume. I'm using the following configuration: cinder_backends: ceph1: volume_driver: cinder.volume.drivers.rbd.RBDDriver rbd_pool: ceph1_vol rbd_ceph_conf: /etc/ceph/ceph1.conf rbd_store_chunk_size: 8 volume_backend_name: ceph1 rbd_user: ceph1_vol rbd_secret_uuid: "{{ cinder_ceph_client_uuid }}" report_discard_supported: true ceph2: volume_driver: cinder.volume.drivers.rbd.RBDDriver rbd_pool: ceph2_vol rbd_ceph_conf: /etc/ceph/ceph2.conf rbd_store_chunk_size: 8 volume_backend_name: ceph2 rbd_user: ceph2_vol rbd_secret_uuid: "{{ cinder_ceph_client_uuid2 }}" report_discard_supported: true ceph_extra_confs: - src: /etc/openstack_deploy/ceph/ceph1.conf dest: /etc/ceph/ceph1.conf client_name: ceph1_vol keyring_src: /etc/openstack_deploy/ceph/ceph1_vol.keyring keyring_dest: /etc/ceph/ceph1.client.ceph1_vol.keyring secret_uuid: '{{ cinder_ceph_client_uuid }}' - src: /etc/openstack_deploy/ceph/ceph2.conf dest: /etc/ceph/ceph2.conf client_name: ceph2_vol keyring_src: /etc/openstack_deploy/ceph/ceph2_vol.keyring keyring_dest: /etc/ceph/ceph2.client.ceph2_vol.keyring secret_uuid: '{{ cinder_ceph_client_uuid2 }}' But when executing the `virsh secret-list` command it only shows the UUID of "cinder_ceph_client_uuid". Both "cinder_ceph_client_uuid" and "cinder_ceph_client_uuid2" are defined in "user_secrets.yml". I have a slight impression that I didn't configure something, but I don't know what, because I didn't find anything else to be done, according to the documentation [1], or it went unnoticed by me. [1] https://docs.openstack.org/openstack-ansible-ceph_client/latest/configure-c… Thanks in advance!

2 5

[nova] VM gets stuck in BUILD status after SQL DBConnectionError error in nova-conductor
by Pavlo Bychikhin 10 Aug '23

10 Aug '23

Hi all, Sometimes when I create a bunch of VMs (40) at once, one of them gets stuck in BUILD status. From logs I found that this happens due to SQL error in nova-conductor: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query'). There is an attempt to schedule the VM: .... nova-scheduler"sending reply msg_id: 593a990393d5439580870d8412ab49f3 reply queue: reply_a0992fe3f9924cb1b45bfa8b27057580 time elapsed: 1.49198414385s" And right after that the SQL error happens: nova-conductor "Exception during message handling" "DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT instance_id_mappings.created_at AS instance_id_mappings_created ......" And after that the VM stays in BUILD status forever. My setup is Galera cluster behind Haproxy load balancer. I checked various DB server timeouts - they are totally fine. Ones in MySQL correspond ones in nova.conf and haproxy.cfg. In the MySQL logs there are messages like: [Warning] Aborted connection 17798924 to db: 'nova' user: 'nova' host: 'xxx.xxx.xxx.xxx' (Got an error reading communication packets) In the Haproxy logs there are messages like: <134>2023-08-05T08:44:34.325338+00:00 dbs01 haproxy[1275]: xxx.xxx.xxx.xxx:44880 [05/Aug/2023:08:42:44.441] mysql_cluster_nova_conn mysql_cluster_nova_conn/dbs02 1/0/109884 11438 CD 4125/1884/1884/1884/0 0/0 "CD" means "The client unexpectedly aborted during data transfer" So the Haproxy believes the connection is closed at client side. I checked the network interface counters - they are also fine (no errors, drops etc). I found some discussions in the Internet. People say, when the DB is behind load balancer - such 'Lost connection' events are pretty normal and not harmful to the Openstack components. I would agree with them. I think, Nova should be able to recover from accidental DB connection lost. Or to set the ERROR status to the VM if that's impossible. My Nova version is 17.0.13 (Queens). May be someone encountered such issue and knows how to fix? Or that's a bug in Nova and I need to upgrade it to a more recent version? Thanks in advance ---- Best regards, Pavlo Bychikhin

1 0

Kayobe VLAN & bond configuration
by Stefan Pinter 10 Aug '23

10 Aug '23

Hi! I am trying to set up Openstack with Kayobe and right now I’m very stuck at VLAN configuration with LACP bonds on my two network nodes. Last Friday the network even came to a hold when activating Openvswitch because it started a broadcast storm or something in the whole company even though I have 2 VLANs just for this project. So I am trying to do it like this: #bond configuration dummy_net_interface: bond0 dummy_net_bootproto: static dummy_net_bond_slaves: - eno12399np0 - eno12409np1 dummy_net_bond_mode: 802.3ad dummy_net_bond_miimon: 100 dummy_net_bond_lacp_rate: fast #configure mgmt interface mgmt_net_interface: brmgmt mgmt_net_bridge_ports: - bond0 #external net work interface and bridge configuration #ext_net_bridge_ports: ext_net_interface: "brbond{{ ext_net_vlan }}" ext_net_bridge_ports: - "bond0.{{ ext_net_vlan }}" Part of my networks.yml … external_net_names: - ext_net … ext_net_cidr: 172.31.246.0/24 ext_net_no_ip: true ext_net_vlan: 705 Last time this worked, when I ran a “kayobe overcloud host configure” (the node had network connection) but then “kayobe overcloud service deploy” killed the network when it started the Openvswitch. Now (today) this configuration doesn’t even allow to reach the network/(to be)neutron nodes (they lose network connection) when the playbook restarts system-networkd. It already creates OVS PHY bridges and lots of stuff even though I ran “overcloud service destroy” before that. Should it do that at this stage? Are there any examples to set up a network node with bonds and vlans and external neutron networks out there? Any suggestions? Thank you 😊 Kind regards Stefan ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 0

Kayobe VLAN & bond configuration
by BPt-FM BBS IaaS 10 Aug '23

10 Aug '23

Hi! I am trying to set up Openstack with Kayobe and right now I’m very stuck at VLAN configuration with LACP bonds on my two network nodes. Last Friday the network even came to a hold when activating Openvswitch because it started a broadcast storm or something in the whole company even though I have 2 VLANs just for this project. So I am trying to do it like this: #bond configuration dummy_net_interface: bond0 dummy_net_bootproto: static dummy_net_bond_slaves: - eno12399np0 - eno12409np1 dummy_net_bond_mode: 802.3ad dummy_net_bond_miimon: 100 dummy_net_bond_lacp_rate: fast #configure mgmt interface mgmt_net_interface: brmgmt mgmt_net_bridge_ports: - bond0 #external net work interface and bridge configuration #ext_net_bridge_ports: ext_net_interface: "brbond{{ ext_net_vlan }}" ext_net_bridge_ports: - "bond0.{{ ext_net_vlan }}" Part of my networks.yml … external_net_names: - ext_net … ext_net_cidr: 172.31.246.0/24 ext_net_no_ip: true ext_net_vlan: 705 Last time this worked, when I ran a “kayobe overcloud host configure” (the node had network connection) but then “kayobe overcloud service deploy” killed the network when it started the Openvswitch. Now (today) this configuration doesn’t even allow to reach the network/(to be)neutron nodes (they lose network connection) when the playbook restarts system-networkd. It already creates OVS PHY bridges and lots of stuff even though I ran “overcloud service destroy” before that. Should it do that at this stage? Are there any examples to set up a network node with bonds and vlans and external neutron networks out there? Any suggestions? Thank you 😊 The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 0