June 2024 - openstack-discuss - lists.openstack.org

[designate][rabbitmq] Worker and Central fail to connect to broker even after authentication
by Francesco Di Nucci 11 Jun '24

11 Jun '24

Hi, I'm having a strange issue with Designate. Setup: Central, mdns, producer and worker services up&running; WSGI configured; api disabled in favor of WSGI Issue: Central and Worker state that connection to RabbitMQ failed as refused, but there is nothing in Rabbit logs. Same time and setup, Producer logs correctly reports RPC responses. If I restart services, in RabbitMQ logs designate-producer, designate-central designate-worker and mod_wsgi are reported as authenticated as user 'designate' and given access to the same vhost, but then Central and Worker start failing again Has anyone encountered the same issue? Regards Francesco Di Nucci

1 0

[qa][devstack][ceph][cinder][manila][nova][glance] Cleaning up the old package based ceph installer
by Goutham Pacha Ravi 11 Jun '24

11 Jun '24

Hello, For a long time we've had multiple ways of deploying ceph for devstack environments in single and over multiple nodes to test integration with nova, cinder, manila, glance and rgw. Of late, we've preferred using the ceph community's preferred installation method: using the ceph orchestrator via its cephadm API. For OpenStack services, it really shouldn't matter _how_ ceph is deployed; and so I wanted to clean up the devstack plugin project to just have one installation method: cephadm. This installation method has been stable since the 2024.1 release cycle; so if there are no objections, I'd love to delete a bunch of code to deploy ceph via packages (apt/rpm) and with docker/podman containers. No new devstack options are _required_ unless you want to customize things. There's an integrated job, several cinder, manila and nova jobs that use both all-in-one and multi-node test jobs with cephadm. Any objections? Thanks, Goutham

2 2

[publiccloud-sig] Cancelling this weeks meeting
by Felix Kronlage-Dammers 11 Jun '24

11 Jun '24

Hi, neither Tobias nor me are available tomorrow (June 12th) for the Public-Cloud SIG Call. Feel free to meet anyways, but neither Tobias nor me will be around to facilitate the call. Next regular call will then be on June 26th. felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-16)
by Goutham Pacha Ravi 10 Jun '24

10 Jun '24

Hello Stackers, We're three weeks away from Milestone-2 of the 2024.2 (Dalmatian) release cycle [1]. Teams are pursuing mid-cycle meetings and design specification freezes in the next couple of weeks. I'd like to begin this week's recap with a shoutout to the TaCT SIG and especially Dr. Jens Harbott (frickler) for helping us with our PyPi cleanup efforts [2]. Apart from this; we had very few governance changes in the past week. I want to take the opportunity to remind you about our gate hygiene efforts. As a contributor, if the continuous integration system (Zuul) is upset with your change, please avoid re-testing with a simple "recheck" comment (or even a comment like "recheck, unrelated failure"). We're trying to isolate unstable CI tests, and if you flag specific failing tests or infrastructure issues, we might be able to address them better. === Weekly Meeting === Last week's meeting was short, and was held simultaneously on Video [3] and IRC [4]. We discussed adding a second maintainer account on all OpenStack packages on PyPi that would be able to push package updates and make project metadata changes (such as adding/removing maintainers). We'll be incorporating this new account soon. We also spoke about graduating Skyline from the list of "emerging projects" [5]. The project team guide has a lot of criteria to create and maintain projects in the OpenStack way; these criteria however, are applied subjectively and are flexible to accommodate specific situations, such as not requiring weekly meetings or formal presence at the PTG. These activities are encouraged. Skyline's contributors are small in number, but have welcomed new contributors. Our recent user surveys have shown that deployments have started adopting Skyline; and installer/configuration management services are seeing feature requests to include Skyline. So the TC feels we need to take this project off of the "emerging" projects list. We volunteered Dr. Jens (frickler) to assess the situation and coordinate this. Our next weekly meeting is tomorrow (June 11th 2024) at 1800 UTC, exclusively on OFTC's #openstack-tc channel. Please see the agenda at [6]. Remember, you can always propose topics to this agenda; and if you're trying to bring up something on short notice, I'll make the attempt to leave some time aside for Open Discussion. === Governance Proposals === ==== Merged ==== Appoint ChunYang Wu as PTL of Trove | https://review.opendev.org/c/openstack/governance/+/920462 ==== Open for review/discussion ==== Remove Eventlet From Openstack | https://review.opendev.org/c/openstack/governance/+/902585 Add TC liaison in DPL model implementation | https://review.opendev.org/c/openstack/governance/+/917516 Update criteria for the inactive projects to become active again | https://review.opendev.org/c/openstack/governance/+/921500 Update to include docs and miscellaneuos repos for AC status | https://review.opendev.org/c/openstack/governance/+/915021 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] https://releases.openstack.org/dalmatian/schedule.html [2] https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L52 [3] https://www.youtube.com/watch?v=34w1Aoai1EA [4] https://meetings.opendev.org/meetings/tc/2024/tc.2024-06-04-18.00.log.html [5] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [6] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[manila] stuck at connecting ceph backend with manila
by Shubham Kumar Yadav 10 Jun '24

10 Jun '24

hello, I'm trying to install manila in my openstack cloud but having problem with "*Authorizing the driver to communicate with Ceph*" *1 -> *i have created a configmap with these confs and mounted it on manila-share pod [client.manila]client mount uid = 0client mount gid = 0log file = /opt/stack/logs/ceph-client.manila.logadmin socket = /opt/stack/status/stack/ceph-$name.$pid.asokkeyring = /etc/ceph/manila.keyring *2 -> *then i have created a secret with client.manila keyring and attached it to the manila-share pod at keyring = /etc/ceph/manila.keyring *3 -> *inside manila-share pod i see the logs 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 *4 -> *My ceph cluster is healthy 5 -> My manila confs look like (what could be wrong here)? conf: manila: DEFAULT: debug: true enabled_share_protocols: NFS,CIFS,CEPHFS enabled_share_backends: cephfsnative1 cephfsnative1: driver_handles_share_servers: False share_backend_name: CEPHFSNATIVE1 share_driver: manila.share.drivers.cephfs.driver.CephFSDriver cephfs_conf_path: /etc/ceph/ceph.conf cephfs_protocol_helper_type: CEPHFS cephfs_auth_id: manila cephfs_cluster_name: rook-ceph ( my rook-ceph cluster name) cephfs_filesystem_name: myfs (i created this fs manually) *6 -> P*ermissions for the user manila ceph auth get-or-create client.manila -o manila.keyring \ mgr 'allow rw' \ mon 'allow r' Hope to hear from you soon with a solution Thank you!

2 1

[Neutron] Bug deputy report for week of June 3rd
by Jakub Libosvar 10 Jun '24

10 Jun '24

Hi, here is the bug deputy report. There are two bugs that require attention: - One seems to be an old issue with conntrack sync, I put it to the wishlist but maybe can be closed as a duplicate. - Another one doesn’t contain enough information to be triaged, please see below. Kuba Needs attention --------------- * Packets dropped in br-int bridge - https://bugs.launchpad.net/neutron/+bug/2067851 - It seems like report needs help troubleshooting the problem or use some accelerated datapath like DPDK. High ---- * Issue associating floating IP with OVN load balancer - https://bugs.launchpad.net/neutron/+bug/2068644 - Patch proposed by Will Szumski: https://review.opendev.org/c/openstack/neutron/+/921663 * SFC: networking-sfc-tempest failures - https://bugs.launchpad.net/neutron/+bug/2068727 - Patch proposed by Lajos: https://review.opendev.org/c/openstack/networking-sfc/+/921514 * Using "joined" relation on tags can cause slow query times in certain situations - https://bugs.launchpad.net/neutron/+bug/2068761 - Revert proposed by Miro: https://review.opendev.org/c/openstack/neutron/+/921566 Low --- * Neutron should have better regex matching in zuul jobs - https://bugs.launchpad.net/neutron/+bug/2068523 - Assigned to Brian Wishlist -------- * [ovn] Dose L3 router ha support sync conntrack states - https://bugs.launchpad.net/neutron/+bug/2068704 - This was discussed at the last PTG in April and probably can be closed as a duplicate of https://bugs.launchpad.net/neutron/+bug/1365438 Invalid ------- * No support on using domain name for ovn connection string in ml2_conf.ini - https://bugs.launchpad.net/neutron/+bug/2067871 - It’s been pointed out on the bug that the issue is fixed in OVS

1 0

[Neutron] Is there a way to work around "External network not reachable from subnet" by setting a static route?
by Florian Haas 10 Jun '24

10 Jun '24

Hi everyone, here's a question I've been chewing on for some time. Please bear with me while I illustrate the background. 1. Suppose you have a regular (internal) Neutron network, subnet, and router. You now plug the subnet into the router and set an external gateway. You then create a port on the subnet. You associate a floating IP with that port. All is well. 2. Suppose instead you do plug the subnet into the router, but you don't set an external gateway on it. You try to associate a floating IP with that port. This creates a BadRequest because: "External network <uuid> is not reachable from subnet <uuid>". 3. Suppose instead you do set an external gateway on the router, but you don't connect the subnet to it. Again, you try to associate a floating IP with that port. This also creates a BadRequest because: "External network <uuid> is not reachable from subnet <uuid>". So far, so good. Up to this point, everything is behaving according to intuitive expectations. Now suppose we have * the external network, * an "outer" internal network, * a Neutron router connecting the two (that is, the router has the external network as its gateway, and has an interface to the "outer" subnet), * an "inner" internal network, * a Nova server (let's call that the "gateway"), which connects the "inner" and "outer" networks, a * another Nova server (the "worker") that sits on the "inner" network. (Suppose, for the sake of discussion, that the "gateway" runs some kind of web application firewall or other virtual security appliance, and the "worker" is a general-purpose Linux instance.) Now I can give the Neutron router a static route so that traffic to the "inner" network uses the "gateway" as its nexthop. Provided the "gateway" does its forwarding correctly, this means that the "worker" has internet connectivity via the "gateway" (which, in turn, achieves this via the router). This, too, works fine for outbound connections originating with the "worker". If I now want to assign a floating IP to the "worker", then routing-wise there really should be no problem: the router should take care of SNAT/DNAT, it would translate the floating IP into a private IP on the "inner" network, would consult its routing table, and throw packets for that "inner" network to the "gateway" to handle, per the static route that's been set. But when I try to associate a floating IP to the "worker" in this scenario, I *still* get the "External network not reachable from subnet" error, because the associated internal check[1] does not take the existing static route into account. This appears to have been an issue in other contexts as well: the defunct Tricircle project appears to have worked around this limitation via a service plugin that just omitted that check.[2] Now my questions are these: 1. Shouldn't this be fixed in core, instead? It would seem to me like the default check is too simplistic, but I might be missing something. 2. Is there a way to make this check take static routes into account, other than providing a service plugin like Tricircle used to do? Thanks! Cheers, Florian References: [1] get_router_for_floatingip(): https://opendev.org/openstack/neutron/src/commit/ebed620aafdff75a05fb5d42ad… [2] get_router_for_floatingip(): https://opendev.org/openstack/tricircle/src/commit/8db8fb30f5757c46a953962d…

2 2

[ironic] Notes from Baremetal SIG, June 5 2024
by Jay Faulkner 10 Jun '24

10 Jun '24

Hi all, The Bare Metal SIG met up Wednesday, June 5 2024 at CERN – one day before the large OpenInfra Days event. There were over 2 dozen Ironic operators and contributors present. Riccardo gave a presentation overviewing Metal3. JayF gave a presentation covering the basics of most Ironic features and highlighted several features which were deprecated. While the talks were recorded, I don't believe the recordings are available yet – I do plan on recreating an updated version of my talk for Youtube in the near future for anyone interested. After the presentations and lunch, participants returned to discuss their use cases and ask questions relating to the earlier presentations. The discussion, as well as specific action items/follow-ups, were documented here: https://etherpad.opendev.org/p/baremetal-sig-cern-june-5-2024 – I will be documenting these action items and RFEs as appropriate over the next week. Add in a short afternoon break to tour the CERN data center, seeing the thousands of machines provisioned by Ironic for CERN, and it was a pleasant day discussing and building rapport with the Ironic community. In my over ten years of working in Ironic, this was the most productive operator/contributor feedback session I've had the pleasure of attending – the attendees were engaged and I'm grateful for their participation and attendance. A special thanks to: - Dmitry Tantsur and Julia Kreger connected up remotely to the session to help field questions. Their expertise was a great help! - Arne Wieback and his team at CERN were gracious hosts and provided us a great VIP tour of the datacenter space. If you were at this event, and have feedback on how we can make future ones better, feel free to reach out here or privately with your ideas. Thanks, Jay Faulkner

1 0

[designate] [neutron] DNS and SSL error 524297
by Francesco Di Nucci 10 Jun '24

10 Jun '24

Hi all, I am using Designate with WSGI and switched to a SSL setup after tests. Somehow now there are floating IPs with a DNS name/entry that do not show in DNS zones, and when I try to release them, the neutron WSGI on network node reports this error: /SSL exception connecting to https://my-controller.example.com:9001/zones?name=zone.cloud.example.com.: HTTPSConnectionPool(host='my-controller.example.com', port=9001): Max retries exceeded with url: /zones?name=zone.cloud.example.com. (Caused by SSLError(SSLError(524297, '[SSL] PEM lib (_ssl.c:4065)')))/ I thought of an SSL config error, but the setup is identical to other working endpoints, and when testing with "/openssl s_client -connect my-controller.example.com:9001"/, everything just works Has anyone encountered this error? Thanks in advance Francesco Di Nucci

1 0

[blazar] Next IRC meeting cancelled
by Pierre Riteau 08 Jun '24

08 Jun '24

Hello, Due to unavailability of key participants, the next IRC meeting scheduled for June 13 is cancelled. We will meet again on June 27. Pierre Riteau (priteau)

1 0