August 2023 - openstack-discuss - lists.openstack.org

[magnum] Correct labels / versions to use Kubernetes v1.24 or later versions of v1.23 with Magnum
by Jay Rhine 15 Aug '23

15 Aug '23

Hello! With the latest Antelope release of OpenStack, the release states that it is now compatible with v1.24. I do see there are changes in the underlying templates, etc. that now allow a later v1.24 tag to successfully install. However, just changing the “kube_tag” label is not really enough to fully run a good v1.24. The the tags for the other images (etcd, cinder csi related, etc.) are still at the same versions of the v1.23.3 release that has been around for a while. I see discussions in the comments on https://storyboard.openstack.org/#!/story/2010028 about various labels that were used to get v1.24 to work well in one case. Is there a document anywhere or table, etc. that would have a documented set of labels that would be recommended? Or any other information on running different Kubernetes releases via label updates with Magnum? Thanks for any advice, Jay -- Jay Rhine Rumble Cloud Architect jay.rhine(a)rumble.com

2 4

[neutron] Bug deputy Aug 7 - Aug 13
by Rodolfo Alonso Hernandez 15 Aug '23

15 Aug '23

Hello Neutrinos: This is the bug report from the week Aug 7 - Aug 13. High: * https://bugs.launchpad.net/neutron/+bug/2030718. "Trunk port can not be deleted after removing trunk network". Unassigned. * https://bugs.launchpad.net/neutron/+bug/2030903. "Test failure with SQLAlchemy master branch using TableClause.insert()". Assigned to Miro. * https://bugs.launchpad.net/neutron/+bug/2031184. "OVN Metadata agent tearsdown all namespaces on restart". Assigned to Miro. Medium: * https://bugs.launchpad.net/neutron/+bug/2030747. "Port creation on shared network fails with port_security defined". Unassigned. * https://bugs.launchpad.net/neutron/+bug/2030773. "OVN DB Sync always logs warning messages about updating all router ports". Assigned to Lucas. * https://bugs.launchpad.net/neutron/+bug/2030804. " Missing ip rule causes FIP removal to fail". Assigned to Brian. * https://bugs.launchpad.net/neutron/+bug/2031087. "ICMPv6 Neighbor Advertisement packets from VM's link-local address dropped by OVS". Unassigned. * https://bugs.launchpad.net/neutron/+bug/2031097. "Router causes temporary ARP poisioning/hijack of DVR FIP when migrating/resizing VM". Unassigned. Low: * https://bugs.launchpad.net/neutron/+bug/2030520. "Add support for iPXE over IPv6 in the ML2/OVN DHCP implementation". Assigned to Slawek. * https://bugs.launchpad.net/neutron/+bug/2030735. "HA router state may be set incorrectly in the Neutron DB in some cases". Unassigned. * https://bugs.launchpad.net/neutron/+bug/2030753. "Network components in Neutron". Assigned to Brian. Untriagged: * https://bugs.launchpad.net/neutron/+bug/2030741. "[OVN] Lack of AZs awareness in L3 port scheduler". Unassigned. * https://bugs.launchpad.net/neutron/+bug/2031085. "error in callback mechanism". Unassigned. Regards.

1 0

[kolla-ansible][tooz][devstack] Upgrade path to etcd v3.4
by Jan Gutter 15 Aug '23

15 Aug '23

Hi folks, etcd maintains two stable branches [1], at the moment these appear to be the v3.4 and v3.5 series. etcd does not support skip-version updates without dataloss[2] out of the box, operators are encouraged to do online updates. I took a quick sample of some etcd versions: * devstack currently uses etcd v3.3.12 (at time of writing) [3] * kolla uses etcd v3.3.27 [4] * tripleo uses rdo etcd: it appears they're already on 3.4.14 [5] * upstream etcd is at 3.4.27 and 3.5.9 [6] Now, the bad news: The client side endpoints of etcd is changing from `v3alpha` to `v3`, with a `v3beta` step added to ensure maximum confusion [7] tooz, for example, currently defaults to `v3alpha` for the recommended etcd3gw backend [8], but it's configurable in the client side by passing an extra option to the backend url. So just updating etcd from 3.3 to 3.4 usually breaks things until a couple of orchestration updates are made. It seems providing a smooth upgrade path would require coordination between orchestration, services that depend on etcd as a backend, and a couple of middleware libraries. The lack of a skip-version upgrade support means it's probably also better to do this sooner rather than later. 3.4 is already late in its release cycle and due to go out of maintenance soon. Options: * upgrade to 3.4 and use `v3` endpoint everywhere - fix forward? * try to detect endpoints in middleware (etcd3gw) somehow? * ask ChatGPT how to farm goats Links (not permalinks!): [1]: https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/b… [2]: https://etcd.io/docs/v3.3/upgrades/upgrade_3_4/ [3]: https://opendev.org/openstack/devstack/src/branch/master/stackrc#L723 [4]: https://opendev.org/openstack/kolla/src/branch/master/docker/etcd/Dockerfil… [5]: https://review.rdoproject.org/r/plugins/gitiles/rdoinfo/+/master/buildsys-t… [6]: https://github.com/etcd-io/etcd/releases/ [7]: https://etcd.io/docs/v3.5/dev-guide/api_grpc_gateway/ [8]: https://opendev.org/openstack/tooz/src/branch/master/tooz/drivers/etcd3gw.p…

2 2

[tc] Technical Committee next weekly meeting Tomorrow on August 15, 2023
by Nikolla, Kristi 15 Aug '23

15 Aug '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, Aug 15, 2023 at 1800 UTC on #openstack-tc on OFTC IRC. Please propose items to the agenda by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting As of now, the agenda for tomorrow's meeting is • Roll call • Follow up on past action items • rosmaita to review guidelines patch and poke at automating it • Gate health check • Testing runtime for 2024.1 release • https://review.opendev.org/c/openstack/governance/+/891225 • User survey question updates by Aug 18 • https://lists.openstack.org/pipermail/openstack-discuss/2023-July/034589.ht… • Open Discussion and Reviews • https://review.opendev.org/q/projects:openstack/governance+is:open • https://review.opendev.org/c/openstack/project-team-guide/+/843457 • https://review.opendev.org/c/openstack/governance/+/888771 Thank you, Kristi Nikolla

1 0

LDAP failover fails
by Albert Braden 14 Aug '23

14 Aug '23

We are experiencing the LDAP failover issue described in [1]. Redhat’s solution is to not bother fixing the bug, and to tell customers to put the LDAP server behind a load-balancer. According to Redhat, that is not a good solution for FreeIPA, as explained in [2] and further elucidated in the blog post [3] that it references. I see that the community has a bug open for this [4] and the bug is being worked on here [5] but there has been no activity since 10/22. What is the status of this bugfix? Does it just need someone to review and merge it, or is there more work to be done? How are other FreeIPA users working around this problem? [1] https://bugzilla.redhat.com/show_bug.cgi?id=2024602#c3 [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… [3] http://ssimo.org/blog/id_019.html [4] https://bugs.launchpad.net/keystone/+bug/1953622 [5] https://review.opendev.org/c/openstack/keystone/+/821086

2 4

[openstack][neutron] it is good if we set fixed ip and set ip in instance netplan also
by Nguyễn Hữu Khôi 14 Aug '23

14 Aug '23

Hello guys. Sometimes I see that my instances won't get IP from Neutron DHCP Agent. Is it good if we set a fixed port for instance and set a static ip in the instance's netplan configuration also? Nguyen Huu Khoi

5 10

[all][elections][ptl][tc] Combined PTL/TC 2024.1 cycle Election Season
by Ian Y. Choi 12 Aug '23

12 Aug '23

Election details: https://governance.openstack.org/election/ The nomination period officially begins Aug 16, 2023 23:45 UTC. Please read the stipulations and timelines for candidates and electorate contained in these governance documents[1][2][3]. Due to circumstances of timing, PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. Similarly, there will only be a TC poll if there are more than 4 candidates stepping forward the available TC positionss. All members of the community are encouraged to confirm their email address as described at: https://governance.openstack.org/election/#electorate There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you which to discuss in private please email any of the election officials[4] so that we may address your concerns. Thank you, [1] https://governance.openstack.org/tc/reference/charter.html [2] https://governance.openstack.org/tc/reference/role-of-the-tc.html [3] https://docs.openstack.org/project-team-guide/ptl.html [4] https://governance.openstack.org/election/#election-officials

1 0

[release] Release countdown for week R-7, Aug 14-18
by Elõd Illés 11 Aug '23

11 Aug '23

Development Focus ----------------- We are entering the last weeks of the 2023.2 Bobcat development cycle. From now until the final release, we'll send a countdown email like this every week. It's probably a good time for teams to take stock of their library and client work that needs to be completed yet. The non-client library freeze is coming up, followed closely by the client lib freeze. Please plan accordingly to avoid any last minute rushes to get key functionality in. General Information ------------------- Next week is the Extra-ATC freeze, in preparation for elections. All contributions to OpenStack are valuable, but some are not expressed as Gerrit code changes. Please list active contributors to your project team who do not have a code contribution this cycle, and therefore won't automatically be considered an Active Technical Contributor and allowed to vote. This is done by adding extra-atcs to https://opendev.org/openstack/governance/src/branch/master/reference/projec… before the Extra-ATC freeze on August 17th, 2023. A quick reminder of the upcoming freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (August 24th, 2023). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (August 31st, 2023) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, August 31st, 2023. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (September 14th) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (September 28th) Finally, now is also a good time to start planning what highlights you want for your deliverables in the cycle highlights. The deadline to submit an initial version for those is set to Feature freeze (August 31st, 2023). Background on cycle-highlights: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125613.html Project Team Guide, Cycle-Highlights: https://docs.openstack.org/project-team-guide/release-management.html#cycle… knelson [at] openstack.org/diablo_rojo on IRC is available if you need help selecting or writing your highlights Upcoming Deadlines & Dates -------------------------- Extra-ATC freeze: August 17th, 2023 (R-7 week) Non-client library freeze: August 24th, 2023 (R-6 week) Client library freeze: August 31st, 2023 (R-5 week) Bobcat-3 milestone: August 31st, 2023 (R-5 week) Final 2023.2 Bobcat release: October 4th, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[OSA] CEPH libvirt secrets
by Murilo Morais 11 Aug '23

11 Aug '23

Good evening everyone! Guys, I'm having trouble adding a second CEPH cluster for Nova/Cincer to consume. I'm using the following configuration: cinder_backends: ceph1: volume_driver: cinder.volume.drivers.rbd.RBDDriver rbd_pool: ceph1_vol rbd_ceph_conf: /etc/ceph/ceph1.conf rbd_store_chunk_size: 8 volume_backend_name: ceph1 rbd_user: ceph1_vol rbd_secret_uuid: "{{ cinder_ceph_client_uuid }}" report_discard_supported: true ceph2: volume_driver: cinder.volume.drivers.rbd.RBDDriver rbd_pool: ceph2_vol rbd_ceph_conf: /etc/ceph/ceph2.conf rbd_store_chunk_size: 8 volume_backend_name: ceph2 rbd_user: ceph2_vol rbd_secret_uuid: "{{ cinder_ceph_client_uuid2 }}" report_discard_supported: true ceph_extra_confs: - src: /etc/openstack_deploy/ceph/ceph1.conf dest: /etc/ceph/ceph1.conf client_name: ceph1_vol keyring_src: /etc/openstack_deploy/ceph/ceph1_vol.keyring keyring_dest: /etc/ceph/ceph1.client.ceph1_vol.keyring secret_uuid: '{{ cinder_ceph_client_uuid }}' - src: /etc/openstack_deploy/ceph/ceph2.conf dest: /etc/ceph/ceph2.conf client_name: ceph2_vol keyring_src: /etc/openstack_deploy/ceph/ceph2_vol.keyring keyring_dest: /etc/ceph/ceph2.client.ceph2_vol.keyring secret_uuid: '{{ cinder_ceph_client_uuid2 }}' But when executing the `virsh secret-list` command it only shows the UUID of "cinder_ceph_client_uuid". Both "cinder_ceph_client_uuid" and "cinder_ceph_client_uuid2" are defined in "user_secrets.yml". I have a slight impression that I didn't configure something, but I don't know what, because I didn't find anything else to be done, according to the documentation [1], or it went unnoticed by me. [1] https://docs.openstack.org/openstack-ansible-ceph_client/latest/configure-c… Thanks in advance!

2 5

[nova] VM gets stuck in BUILD status after SQL DBConnectionError error in nova-conductor
by Pavlo Bychikhin 11 Aug '23

11 Aug '23

Hi all, Sometimes when I create a bunch of VMs (40) at once, one of them gets stuck in BUILD status. From logs I found that this happens due to SQL error in nova-conductor: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query'). There is an attempt to schedule the VM: .... nova-scheduler"sending reply msg_id: 593a990393d5439580870d8412ab49f3 reply queue: reply_a0992fe3f9924cb1b45bfa8b27057580 time elapsed: 1.49198414385s" And right after that the SQL error happens: nova-conductor "Exception during message handling" "DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT instance_id_mappings.created_at AS instance_id_mappings_created ......" And after that the VM stays in BUILD status forever. My setup is Galera cluster behind Haproxy load balancer. I checked various DB server timeouts - they are totally fine. Ones in MySQL correspond ones in nova.conf and haproxy.cfg. In the MySQL logs there are messages like: [Warning] Aborted connection 17798924 to db: 'nova' user: 'nova' host: 'xxx.xxx.xxx.xxx' (Got an error reading communication packets) In the Haproxy logs there are messages like: <134>2023-08-05T08:44:34.325338+00:00 dbs01 haproxy[1275]: xxx.xxx.xxx.xxx:44880 [05/Aug/2023:08:42:44.441] mysql_cluster_nova_conn mysql_cluster_nova_conn/dbs02 1/0/109884 11438 CD 4125/1884/1884/1884/0 0/0 "CD" means "The client unexpectedly aborted during data transfer" So the Haproxy believes the connection is closed at client side. I checked the network interface counters - they are also fine (no errors, drops etc). I found some discussions in the Internet. People say, when the DB is behind load balancer - such 'Lost connection' events are pretty normal and not harmful to the Openstack components. I would agree with them. I think, Nova should be able to recover from accidental DB connection lost. Or to set the ERROR status to the VM if that's impossible. My Nova version is 17.0.13 (Queens). May be someone encountered such issue and knows how to fix? Or that's a bug in Nova and I need to upgrade it to a more recent version? Thanks in advance ---- Best regards, Pavlo Bychikhin

1 0