June 2024 - openstack-discuss - lists.openstack.org

[Neutron] Is there a way to work around "External network not reachable from subnet" by setting a static route?
by Florian Haas 08 Jul '24

08 Jul '24

Hi everyone, here's a question I've been chewing on for some time. Please bear with me while I illustrate the background. 1. Suppose you have a regular (internal) Neutron network, subnet, and router. You now plug the subnet into the router and set an external gateway. You then create a port on the subnet. You associate a floating IP with that port. All is well. 2. Suppose instead you do plug the subnet into the router, but you don't set an external gateway on it. You try to associate a floating IP with that port. This creates a BadRequest because: "External network <uuid> is not reachable from subnet <uuid>". 3. Suppose instead you do set an external gateway on the router, but you don't connect the subnet to it. Again, you try to associate a floating IP with that port. This also creates a BadRequest because: "External network <uuid> is not reachable from subnet <uuid>". So far, so good. Up to this point, everything is behaving according to intuitive expectations. Now suppose we have * the external network, * an "outer" internal network, * a Neutron router connecting the two (that is, the router has the external network as its gateway, and has an interface to the "outer" subnet), * an "inner" internal network, * a Nova server (let's call that the "gateway"), which connects the "inner" and "outer" networks, a * another Nova server (the "worker") that sits on the "inner" network. (Suppose, for the sake of discussion, that the "gateway" runs some kind of web application firewall or other virtual security appliance, and the "worker" is a general-purpose Linux instance.) Now I can give the Neutron router a static route so that traffic to the "inner" network uses the "gateway" as its nexthop. Provided the "gateway" does its forwarding correctly, this means that the "worker" has internet connectivity via the "gateway" (which, in turn, achieves this via the router). This, too, works fine for outbound connections originating with the "worker". If I now want to assign a floating IP to the "worker", then routing-wise there really should be no problem: the router should take care of SNAT/DNAT, it would translate the floating IP into a private IP on the "inner" network, would consult its routing table, and throw packets for that "inner" network to the "gateway" to handle, per the static route that's been set. But when I try to associate a floating IP to the "worker" in this scenario, I *still* get the "External network not reachable from subnet" error, because the associated internal check[1] does not take the existing static route into account. This appears to have been an issue in other contexts as well: the defunct Tricircle project appears to have worked around this limitation via a service plugin that just omitted that check.[2] Now my questions are these: 1. Shouldn't this be fixed in core, instead? It would seem to me like the default check is too simplistic, but I might be missing something. 2. Is there a way to make this check take static routes into account, other than providing a service plugin like Tricircle used to do? Thanks! Cheers, Florian References: [1] get_router_for_floatingip(): https://opendev.org/openstack/neutron/src/commit/ebed620aafdff75a05fb5d42ad… [2] get_router_for_floatingip(): https://opendev.org/openstack/tricircle/src/commit/8db8fb30f5757c46a953962d…

2 4

Kolla-Ansible Issues
by Alexandra Rong 06 Jul '24

06 Jul '24

Hello OpenStack-Ansible team, We are students in Germany studying Business Informatics. Currently, we have a project where we need to build an OpenStack environment using Kolla-Ansible. Our goal is to use Gnocchi and CloudKitty to gather metrics on the usage of the VMs in OpenStack. We tried following the official Kolla-Ansible deployment guide but encountered issues starting Kolla-Ansible, as shown in the attached screenshots. Could you possibly help us? Best regards, Felix and Alexandra Ansible.cfg: Error: All-in-one:

5 4

[CloudKitty] query regarding ElasticSearch usage
by Rafael Weingärtner 04 Jul '24

04 Jul '24

Hello OpenStack community and users of CloudKitty! There have been some discussions in our bi-weekly meetings and vPTG meetings regarding the deprecation and removal of ElasticSearch support as a storage backend. Therefore, this e-mail is to consult/query users regarding ElasticSearch users. If there are no people using it anymore with CloudKitty, we are considering the alternative of removing it and maintaining only the OpenSearch support. We would like to hear back from you guys. Let me know if you are using ElasticSearch, and if you intend to stick with it. Also, do you guys feel the need to improve something in its support in CloudKitty. Thanks in advance for your feedback! -- Rafael Weingärtner

5 5

[nova] volume information was duplicated when Live Migration failed
by Junya Noguchi (Fujitsu) 04 Jul '24

04 Jul '24

To whom it may concern Hi. I encountered an issue with OpenStack where volume information was duplicated when Live Migration failed. Could you please provide a solution to this issue? [Issue] After a Live Migration failure, when I checked the volume where the error occurred using the 'openstack volume list' command, the following was displayed: +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda | I expect that the 'Attached to' display should only appear once as there is one volume connected, but it is duplicated for each failed attempt. The versions of OpenStack and nova are as follows: OpenStack Zed $ openstack --version openstack 6.0.0 $ nova --version nova CLI is deprecated and will be a removed in a future release 18.1.0 [Details] The volume display is duplicated under the following conditions: - When Live Migration of an instance fails If the communication between the ports required for Live Migration is closed, Live Migration will fail when executed. $ sudo netstat -tuln | grep 16509 $ The output of nova-compute.log at the time of Live Migration failure is as follows: --- 2024-02-28 17:04:32.119 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Live Migration failure: unable to connect to server at 'XXXXX.com:16509': Connection refused: libvirt.libvirtError: unable to connect to server at 'XXXXX.com:16509': Connection refused 2024-02-28 17:04:32.119 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation thread notification thread_finished /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10516 2024-02-28 17:04:32.607 2785 DEBUG nova.virt.libvirt.migration [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] VM running on src, migration failed _log /usr/lib/python3.9/site-packages/nova/virt/libvirt/migration.py:432 2024-02-28 17:04:32.608 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10330 2024-02-28 17:04:32.608 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation has aborted --- After Live Migration, when I check the volume where the error occurred using the 'openstack volume list' command, the following is displayed: +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda | '73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda' is displayed as many times as Live Migration failed. The duplicate display did not disappear even after Live Migration was successful. Thanks,

2 2

[kolla-ansible][letsencrypt] containers are running but not getting certificates.
by Jonathan Proulx 03 Jul '24

03 Jul '24

Hi All, I'm trying to do a test multinode deploy using 2023.2 I have letsencrypt_webserver and letsencrypt_lego contsainers running and I'm seeing random traffic in the /var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly confident they're plumbed through to the public internet properly, but I don't seem to be getting certificates. how can I trigger a renewal attempt so I can maybe see what I've screwed up? Thanks, -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL

3 6

Warehouse failure
by ゞPositive î 01 Jul '24

01 Jul '24

The Centos system has stopped updating now, and we hope that the official website manual Centos can be changed to Rocky Linux. Now we are using Rocky Linux to install OpenStack using the Centos system installation method. Among them, the installation source link for yum install Centos release openstack ussuri and yum install Centos release openstack xena is invalid. We hope to receive your solution. ゞPositive ? 451526204(a)qq.com  

2 1

2025.1 E OpenStack Release Naming Poll
by Allison Price 29 Jun '24

29 Jun '24

Hi everyone, It’s that time again! Please join me in voting to name the 31st release of OpenStack, starting with the letter ‘E’! https://civs1.civs.us/cgi-bin/vote.pl?id=E_d8469898c60545f2&akey=a16aa8cad5… The deadline to vote is Friday, June 28 at 7 UTC. Cheers, Allison

2 1

Openstack-ansible upgrade
by Danish Khan 28 Jun '24

28 Jun '24

Dear Team, We have few openstack-ansible Train release in our environment. We want to upgrade these setup to atleast Yoga release. Currently we are doing upgrade step by step. like from Train to Ussuri then to Victoria then to wallaby and so on(in N+1 series). Can we skip few releases and directly go to Yoga or any other feasible release? Regards, Danish Khan

4 3

[release] Release countdown for week R-13, Jul 01-05
by Elõd Illés 28 Jun '24

28 Jun '24

Development Focus ----------------- The Dalmatian-2 milestone is next week, on July 4th, 2024! 2024.2 Dalmatian-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/dalmatian/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library that has not been otherwise released since milestone 1. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. Next week is also the deadline to freeze the contents of the final release. All new '2024.2 Dalmatian' deliverables need to have a deliverable file in https://opendev.org/openstack/releases/src/branch/master/deliverables and need to have done a release by milestone-2. Upcoming Deadlines & Dates -------------------------- Dalmatian-2 Milestone: July 4th, 2024 Előd Illés irc: elodilles @ #openstack-release

1 0

Re: [manila] stuck at connecting ceph backend with manila
by Goutham Pacha Ravi 28 Jun '24

28 Jun '24

Hey Shubham, Please keep the conversation on the list. On Thu, Jun 27, 2024 at 10:36 PM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: > > Hi, actually the keyring file contains the manila keyring which is mounted on manila-share pod > > 1 -> ls -l /etc/ceph (from manila-share pod) > total 12 > -rw-r--r-- 1 root root 279 Jun 26 08:02 ceph.conf > -rw-r--r-- 1 root root 62 Jun 26 08:02 keyring > -rw-r--r-- 1 root root 92 Mar 4 20:27 rbdmap > > 2 -> i have already generated manila keyring Do you mean the content of the keyring is in the "keyring" file? In your earlier email you showed the contents of your ceph.conf, it referred to the file name of the manila keyring as "/etc/ceph/manila.keyring". I don't see that file in the output above. > > On Fri, 28 Jun 2024 at 00:46, Goutham Pacha Ravi <gouthampravi(a)gmail.com> wrote: >> >> Hi, >> >> On Thu, Jun 27, 2024 at 12:26 AM Shubham Kumar Yadav >> <shubham.kumar.yadav369(a)gmail.com> wrote: >> > >> > Hi, these are the outputs you asked for >> > >> > 1 -> ls -l /etc/ceph >> > total 12 >> > -rw-r--r-- 1 root root 279 Jun 26 08:02 ceph.conf >> > -rw-r--r-- 1 root root 62 Jun 26 08:02 keyring >> > -rw-r--r-- 1 root root 92 Mar 4 20:27 rbdmap >> >> The manila keyring is missing here; Can you add it here and retry? >> >> >> > >> > >> > 2 -> ceph --verbose fs ls -n client.manila >> > parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name='client.manila', cluster=None, cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['fs', 'ls'] >> > [errno 1] RADOS permission error (error connecting to the cluster) >> > >> > 3 -> i created keyring using >> > >> > ceph auth get-or-create client.manila -o manila.keyring \ >> > mgr 'allow rw' \ >> > mon 'allow r' >> > >> > >> > Hope to hear from you soon >> > Thank you! >> > >> > >> > >> > On Thu, 27 Jun 2024 at 05:25, Goutham Pacha Ravi <gouthampravi(a)gmail.com> wrote: >> >> >> >> The client capabilities look correct; Can you paste the output of "ls >> >> -l /etc/ceph" from the manila-share pod? >> >> Also attempt to use "--verbose" with the Ceph CLI; maybe there'll be a >> >> helpful log: "ceph --verbose fs ls -n client.manila" >> >> >> >> I'm unsure if you're creating the keyring file by hand.. but if you >> >> are, you can use this command to generate a file instead: >> >> >> >> ceph auth get-or-create client.manila -o /etc/ceph/ceph.client.manila.keyring >> >> You can use this ^ command to generate it and then make sure to copy >> it into the /etc/ceph folder that's mounted into the manila-share pod. >> >> >> >> >> >> >> >> >> On Wed, Jun 26, 2024 at 12:02 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> >> > >> >> > Mmm wait, I believe I'm mistaken, I didn't read the entire config file. >> >> > Please omit my comment. >> >> > >> >> > On Wed, Jun 26, 2024, 12:54 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> >> >> >> >> >> Hi again, >> >> >> It seems you are missing osd permission on manila user authorization: >> >> >> >> >> >> Allow manila user to read/write on osd's >> >> >> >> >> >> Cheers. >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Jun 26, 2024, 7:36 AM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: >> >> >>> >> >> >>> hello, >> >> >>> I'm trying to install manila in my openstack cloud but having problem with "Authorizing the driver to communicate with Ceph" >> >> >>> >> >> >>> 1 -> i have created a configmap with these confs and mounted it on manila-share pod >> >> >>> >> >> >>> [client.manila] >> >> >>> client mount uid = 0 >> >> >>> client mount gid = 0 >> >> >>> log file = /opt/stack/logs/ceph-client.manila.log >> >> >>> admin socket = /opt/stack/status/stack/ceph-$name.$pid.asok >> >> >>> keyring = /etc/ceph/manila.keyring >> >> >>> >> >> >>> >> >> >>> 2 -> then i have created a secret with client.manila keyring and attached it to the manila-share pod at >> >> >>> >> >> >>> keyring = /etc/ceph/manila.keyring >> >> >>> >> >> >>> 3 -> inside manila-share pod i see the logs >> >> >>> >> >> >>> 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 >> >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 >> >> >>> >> >> >>> >> >> >>> 4 -> My ceph cluster is healthy >> >> >>> >> >> >>> 5 -> My manila confs look like (what could be wrong here)? >> >> >>> >> >> >>> conf: >> >> >>> manila: >> >> >>> DEFAULT: >> >> >>> debug: true >> >> >>> enabled_share_protocols: NFS,CIFS,CEPHFS >> >> >>> enabled_share_backends: cephfsnative1 >> >> >>> cephfsnative1: >> >> >>> driver_handles_share_servers: False >> >> >>> share_backend_name: CEPHFSNATIVE1 >> >> >>> share_driver: manila.share.drivers.cephfs.driver.CephFSDriver >> >> >>> cephfs_conf_path: /etc/ceph/ceph.conf >> >> >>> cephfs_protocol_helper_type: CEPHFS >> >> >>> cephfs_auth_id: manila >> >> >>> cephfs_cluster_name: rook-ceph ( my rook-ceph cluster name) >> >> >>> cephfs_filesystem_name: myfs (i created this fs manually) >> >> >>> >> >> >>> 6 -> Permissions for the user manila >> >> >>> >> >> >>> ceph auth get-or-create client.manila -o manila.keyring \ >> >> >>> mgr 'allow rw' \ >> >> >>> mon 'allow r' >> >> >>> >> >> >>> >> >> >>> 7 -> ceph --version >> >> >>> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> >> >>> >> >> >>> openstackhelm Manila Chart >> >> >>> name: manila >> >> >>> version: 0.1.7 >> >> >>> >> >> >>> 8 -> ceph -n client.manila fs ls >> >> >>> [errno 1] RADOS permission error (error connecting to the cluster) >> >> >>> (while it works with admin) >> >> >>> >> >> >>> Hope to hear from you soon with a solution >> >> >>> Thank you!

1 0

[manila] stuck at connecting ceph backend with manila
by Goutham Pacha Ravi 27 Jun '24

27 Jun '24

Hi, On Thu, Jun 27, 2024 at 12:26 AM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: > > Hi, these are the outputs you asked for > > 1 -> ls -l /etc/ceph > total 12 > -rw-r--r-- 1 root root 279 Jun 26 08:02 ceph.conf > -rw-r--r-- 1 root root 62 Jun 26 08:02 keyring > -rw-r--r-- 1 root root 92 Mar 4 20:27 rbdmap The manila keyring is missing here; Can you add it here and retry? > > > 2 -> ceph --verbose fs ls -n client.manila > parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name='client.manila', cluster=None, cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['fs', 'ls'] > [errno 1] RADOS permission error (error connecting to the cluster) > > 3 -> i created keyring using > > ceph auth get-or-create client.manila -o manila.keyring \ > mgr 'allow rw' \ > mon 'allow r' > > > Hope to hear from you soon > Thank you! > > > > On Thu, 27 Jun 2024 at 05:25, Goutham Pacha Ravi <gouthampravi(a)gmail.com> wrote: >> >> The client capabilities look correct; Can you paste the output of "ls >> -l /etc/ceph" from the manila-share pod? >> Also attempt to use "--verbose" with the Ceph CLI; maybe there'll be a >> helpful log: "ceph --verbose fs ls -n client.manila" >> >> I'm unsure if you're creating the keyring file by hand.. but if you >> are, you can use this command to generate a file instead: >> >> ceph auth get-or-create client.manila -o /etc/ceph/ceph.client.manila.keyring You can use this ^ command to generate it and then make sure to copy it into the /etc/ceph folder that's mounted into the manila-share pod. >> >> >> >> On Wed, Jun 26, 2024 at 12:02 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> > >> > Mmm wait, I believe I'm mistaken, I didn't read the entire config file. >> > Please omit my comment. >> > >> > On Wed, Jun 26, 2024, 12:54 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> >> >> >> Hi again, >> >> It seems you are missing osd permission on manila user authorization: >> >> >> >> Allow manila user to read/write on osd's >> >> >> >> Cheers. >> >> >> >> >> >> >> >> On Wed, Jun 26, 2024, 7:36 AM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: >> >>> >> >>> hello, >> >>> I'm trying to install manila in my openstack cloud but having problem with "Authorizing the driver to communicate with Ceph" >> >>> >> >>> 1 -> i have created a configmap with these confs and mounted it on manila-share pod >> >>> >> >>> [client.manila] >> >>> client mount uid = 0 >> >>> client mount gid = 0 >> >>> log file = /opt/stack/logs/ceph-client.manila.log >> >>> admin socket = /opt/stack/status/stack/ceph-$name.$pid.asok >> >>> keyring = /etc/ceph/manila.keyring >> >>> >> >>> >> >>> 2 -> then i have created a secret with client.manila keyring and attached it to the manila-share pod at >> >>> >> >>> keyring = /etc/ceph/manila.keyring >> >>> >> >>> 3 -> inside manila-share pod i see the logs >> >>> >> >>> 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 >> >>> >> >>> >> >>> 4 -> My ceph cluster is healthy >> >>> >> >>> 5 -> My manila confs look like (what could be wrong here)? >> >>> >> >>> conf: >> >>> manila: >> >>> DEFAULT: >> >>> debug: true >> >>> enabled_share_protocols: NFS,CIFS,CEPHFS >> >>> enabled_share_backends: cephfsnative1 >> >>> cephfsnative1: >> >>> driver_handles_share_servers: False >> >>> share_backend_name: CEPHFSNATIVE1 >> >>> share_driver: manila.share.drivers.cephfs.driver.CephFSDriver >> >>> cephfs_conf_path: /etc/ceph/ceph.conf >> >>> cephfs_protocol_helper_type: CEPHFS >> >>> cephfs_auth_id: manila >> >>> cephfs_cluster_name: rook-ceph ( my rook-ceph cluster name) >> >>> cephfs_filesystem_name: myfs (i created this fs manually) >> >>> >> >>> 6 -> Permissions for the user manila >> >>> >> >>> ceph auth get-or-create client.manila -o manila.keyring \ >> >>> mgr 'allow rw' \ >> >>> mon 'allow r' >> >>> >> >>> >> >>> 7 -> ceph --version >> >>> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> >>> >> >>> openstackhelm Manila Chart >> >>> name: manila >> >>> version: 0.1.7 >> >>> >> >>> 8 -> ceph -n client.manila fs ls >> >>> [errno 1] RADOS permission error (error connecting to the cluster) >> >>> (while it works with admin) >> >>> >> >>> Hope to hear from you soon with a solution >> >>> Thank you!

1 0

Antelope: strange failure in VM migration from a Yoga HV to an Antelope one
by Michel Jouvin 27 Jun '24

27 Jun '24

Hi, In our Yoga -> Antelope upgrade, we have been able to live migrate active VMs from most Yoga HV to an Antelope one. In one case, the migration starts well but at the end of the disk migration, there is an "unexpected error" and the migration is canceled. I have not been able to find any more detailed information about the failure. In the source HV, nova-compute log says: ------------------- 2024-06-13 13:41:09.544 4108750 INFO nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Migration running for 2804 secs, memory 100% remaining (bytes processed=0, remaining=0, total=0); disk 1% remaining (bytes processed=106548297728, remaining=863371264, total=107411668992). 2024-06-13 13:41:20.436 4108750 INFO nova.compute.resource_tracker [req-db52ee9a-b0e4-4ac1-83f6-f57600ccc275 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Updating resource usage from migration 35e1a5e8-0f7d-4ff9-9075-c8da30649b4a 2024-06-13 13:41:45.678 4108750 INFO nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Migration running for 2840 secs, memory 4% remaining (bytes processed=13900749654, remaining=691298304, total=16782270464); disk 0% remaining (bytes processed=107429101568, remaining=262144, total=107429363712). 2024-06-13 13:41:51.153 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] VM Paused (Lifecycle Event) 2024-06-13 13:41:51.212 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] During sync_power_state the instance has a pending task (migrating). Skip. 2024-06-13 13:41:51.390 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] VM Resumed (Lifecycle Event) 2024-06-13 13:41:51.426 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] During sync_power_state the instance has a pending task (migrating). Skip. 2024-06-13 13:41:51.772 4108750 ERROR nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Migration operation has aborted 2024-06-13 13:41:51.774 4108750 ERROR nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Live Migration failure: internal error: QEMU unexpectedly closed the monitor (vm='instance-00042fe6'): 2024-06-13T10:54:25.319005Z qemu-kvm: warning: Machine type 'pc-i440fx-rhel7.6.0' is deprecated: machine types for previous major releases are deprecated 2024-06-13 13:41:51.806 4108750 INFO nova.compute.manager [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Swapping old allocation on dict_keys(['0a7be64c-cff9-4fa7-93ac-ddd0a965e93a']) held by migration 35e1a5e8-0f7d-4ff9-9075-c8da30649b4a for instance 2024-06-13 13:41:52.882 4108750 WARNING nova.compute.manager [req-8c267aa3-29dc-43ec-9a1b-9c4ada34dd3f f0d2557b870a44fa90c98b507629407e 5cf79c1092c44348addf9bee098f138e - default default] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Received unexpected event network-vif-unplugged-7446e7d3-a85d-4bd3-a609-decf5e7dd591 for instance with vm_state active and task_state None. 2024-06-13 13:41:55.758 4108750 WARNING nova.compute.manager [req-59c5fb1f-4223-4e14-90cf-c97bb0b9130a f0d2557b870a44fa90c98b507629407e 5cf79c1092c44348addf9bee098f138e - default default] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Received unexpected event network-vif-plugged-7446e7d3-a85d-4bd3-a609-decf5e7dd591 for instance with vm_state active and task_state None. ----------------- It happens for all the VMs hosted by the source HV that have no link between them (not the same project) and whatever is the destination HV (same HW model) so I guess it has to do with the source HV. And there were no such problems when migrating VMs between the other HVs with the same HW model. Any idea where to look for details that may help to identify the problem? Best regards, Michel

1 2

答复: [largescale-sig]scaling story
by Alex Song (宋文平) 27 Jun '24

27 Jun '24

0 *H÷ 010 +0 *H÷ $Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Hi Mohammed, The max number of conns for RabbitMQ is estimated through cacluating the conns of nova and cinder components. We have 3000 compute nodes, 3 control nodes and 15 cinder-volume nodes, We deploy rabbitmq on control nodes and use master-slave mode, the totally conns of RabbitMQ is almost 2w, so we set 2w for RabbitMQ configuration. Before deploying the SB relay, each ovn SB process needed to manage an average of 600+connections, the CPU usage is often 100% and the requests is slow processing. After adding the relay, each relay process handles about 60 connections, the total relay process is set to 10. After testing, each relay process has a low CPU usage and can process information quickly. -----é®ä»¶åä»¶----- åä»¶äºº: Thierry Carrez [mailto:thierry@openinfra.dev] åéæ¶éŽ: 2024å¹Ž6æ19æ¥ 17:29 æ¶ä»¶äºº: Alex Song (å®æå¹³) <songwenping(a)inspur.com> äž»é¢: Re: [largescale-sig]scaling story Hi Alex, Thanks for the great write up! I would love to see more of this. How did you adjust the max number of conns for RabbitMQ and for the relay I assume you used https://docs.ovn.org/en/latest/tutorials/ovn-ovsdb-relay.html <https://docs.ovn.org/en/latest/tutorials/ovn-ovsdb-relay.html> ? Thanks Mohammed *From: *Alex Song (å®æå¹³) <songwenping(a)inspur.com> *Date: *Wednesday, June 12, 2024 at 4:43â¯AM *To: *openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> *Subject: *[largescale-sig]scaling story Hi everyone, we want to share our scaling story. Please refer the link: https://etherpad.opendev.org/p/large-scale-inspur <https://etherpad.opendev.org/p/large-scale-inspur> Thanks, Alex Song 0É0± xðáwIÊèë`{0 *H÷ 0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA0 170109092830Z 340511122004Z0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA0"0 *H÷ 0 «ä5ïc$Œ©æ'µ¯Þ6>úUKÛdÔ²Áe9Î~{BîÒLgD÷*wvVÊŠ/DýUj_xá\m/ óž=kÐ·éGÙœQ€ýx~Wùgk ÛÜøãÔ7É6NçÏ*?n°Ê²mhùè{Ã¯ôÌÆ 7üF-Î<@ÃÓÍ¬WçÅyåLZrF 6~føÈ×T~$0d¡ýL|zšøW=ötÚ%ýq,¥Ã~Ÿ"ÀýŸÑö2T,QÕÔ,dºÂÅ^§ÈôïåJ)ëVvp Ó£00 +7CA0U0Uÿ0ÿ0U^YŠŽLX`Nöµ¥9Š2Á5j0 +70# +7&a°$öz(¶o§K0 *H÷ daòYÙ~×ì×NÑ3ŽËlP±i!¿ýÅòsÝLºP;I4hžØþ¡L¶äŠJêà·Õ:1àEÕs®9UÛ:8Mh{Ä²ìºŒ£·ÚŒó×÷C_ŸhoÏúÆËYM&UØ³?9êèr·ìgP_÷mÇ#YÖpîw0º¹«0$p³jºnº¥¯/Ôä¬÷Ú ŠÖ5uþQÝ¶h(y¶ènnã%E,øú&Zâ"dKØ³ãi5VúP9±œ/ñ õkØŸsá/Xðï»¯€µ³O÷Í-ó¿Mè³3Óž{G(ªŸßŒG0C0+ ~+mçbí++0 *H÷ 0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA0 191128064731Z 241126064731Z010 &ò,dcom10 &ò,dlangchao10 &ò,dhome10Uæµªæœ®ä¿¡æ¯10U å®æå¹³1%0# *H÷ songwenping@inspur.com0"0 *H÷ 0 ã4ÒEd$i÷18#,åH~èyµþJRáLêg¡$@6vÔ<sâelPO4ýooXZç1¢iéÁ IòKjÚO§h;Rfùê7Ÿp¬ŒæäÝPòüÿ~æ E}é{có» (ù'+zÞ s ,!K£;û¥¶DEu1+ðÏ»õØÛähJjŸj']a¥Î·7¡õòhõßHŒ8QzÆ{ËÍ73>, ªí|gœ0y€Xv'ÚÊ ¬ÀO#)³°øÕM^œ¢VñóŸ¯ìFñ;µ5HŽù£Ã0¿0= +700.&+7ò©×z©=÷Ø\Jý&§Md\0)U%"0 ++ +7 0U 05 +7 (0&0 +0 +0 +7 0D *H÷ 7050*H÷ 0*H÷ 0+0 *H÷ 0UW 0²^qûìÇUã¡b[]s0U#0^YŠŽLX`Nöµ¥9Š2Á5j0U00ÿ ü ùºldap:///CN=INSPUR-CA,CN=JTCA2012,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=home,DC=langchao,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint:http://JTCA2012.home.langchao.com/CertEnroll/INSPUR-CA.crl0)+00±+0€ldap:///CN=INSPUR-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=home,DC=langchao,DC=com?cACertificate?base?objectClass=certificationAuthority0a+0Uhttp://JTCA2012.home.langchao.com/CertEnroll/JTCA2012.home.langchao.com_INSPUR-CA.crt0IUB0@ & +7 songwenping@inspur.comsongwenping@inspur.com0 *H÷ §öãý+·>³â¿Xíe-÷¯[ÂÎ©É\ÃAæMà.U'ù||Wø$Šn' `9Ýµ¯éÉM^mz0Äð -AÒ¯×Tï+£§¡RbfzÆX#ùkY<ØDþxólõA%ÑøN¬×grÛÚ·ãF!GþóüVœÍ±Ð[ÏL®GÅ3€$ /DžÃiú_h|cI}DôþyªÉñ Ç}³W$_Äáy]ò±DÄ1(tr)6gêäÑæÄLd$!t{x±Z±·þÏeqmkþÒÒ÷r³ßxõ%lUÉÒ4d100p0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA~+mçbí++0 + ø0 *H÷ 1 *H÷ 0 *H÷ 1 240627081404Z0# *H÷ 1ŠÒ5KCuâ|!:¶!~ÏÚ/Ý0 +71r0p0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA~+mçbí++0*H÷ 1r p0Y10 &ò,dcom10 &ò,dlangchao10 &ò,dhome10U INSPUR-CA~+mçbí++0 *H÷ 100 `He*0 `He0 *H÷ 0 `He0*H÷ 0 *H÷ @0+0 `He0 `He0 `He0 *H÷ Jû¡ûäž«$]ŽíijO yªLMÑ¢ÅúŸC±_U¶[1Jnl~/Aò¢\E gTŽ ENºÁ óª už6ÄµÕ/Ô/j/,ltµ~O¥Æù]Î0È9§DAªªC'8µG*9ÄíËüÛâèÔðHmá3ŽYËù*Ý'Ÿ¯ù§Ý,â9iâÙ2±×Á·ÞÌøtüoÙ wýÂ!7e&ß ëFÃ®T¢é3.k-îgøvc«nŠ*(G> óóFK°"óx aÙ Æ&?ûæxNJº~'Øü¯L

1 0

[Glance] Extending Specs Freeze Deadline by 1 week
by Pranali Deore 27 Jun '24

27 Jun '24

Hello All, As discussed in the last glance meeting [1] we are extending the glance specs freeze deadline by 1 week since we have 2 new specs up recently. Current Spec Freeze date (for Glance) : 5th July 2024 New Spec Freeze date (for Glance) : 12th July 2024 Please make sure your specs are updated before the new deadline. [1]: https://meetings.opendev.org/irclogs/%23openstack-meeting/%23openstack-meet… Thanks, Pranali Deore

1 0

[manila] stuck at connecting ceph backend with manila
by Shubham Kumar Yadav 27 Jun '24

27 Jun '24

hello, I'm trying to install manila in my openstack cloud but having problem with "*Authorizing the driver to communicate with Ceph*" *1 -> *i have created a configmap with these confs and mounted it on manila-share pod [client.manila]client mount uid = 0client mount gid = 0log file = /opt/stack/logs/ceph-client.manila.logadmin socket = /opt/stack/status/stack/ceph-$name.$pid.asokkeyring = /etc/ceph/manila.keyring *2 -> *then i have created a secret with client.manila keyring and attached it to the manila-share pod at keyring = /etc/ceph/manila.keyring *3 -> *inside manila-share pod i see the logs 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. 2024-06-03 09:00:53.713 6 ERROR manila.share.manager 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 *4 -> *My ceph cluster is healthy 5 -> My manila confs look like (what could be wrong here)? conf: manila: DEFAULT: debug: true enabled_share_protocols: NFS,CIFS,CEPHFS enabled_share_backends: cephfsnative1 cephfsnative1: driver_handles_share_servers: False share_backend_name: CEPHFSNATIVE1 share_driver: manila.share.drivers.cephfs.driver.CephFSDriver cephfs_conf_path: /etc/ceph/ceph.conf cephfs_protocol_helper_type: CEPHFS cephfs_auth_id: manila cephfs_cluster_name: rook-ceph ( my rook-ceph cluster name) cephfs_filesystem_name: myfs (i created this fs manually) *6 -> P*ermissions for the user manila ceph auth get-or-create client.manila -o manila.keyring \ mgr 'allow rw' \ mon 'allow r' *7 -> *ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) openstackhelm Manila Chart name: manila version: 0.1.7 8 -> *ceph -n client.manila fs ls* * [errno 1] RADOS permission error (error connecting to the cluster)* (while it works with admin) Hope to hear from you soon with a solution Thank you!

3 3

[dev][swift][account-reaper] Account with sharded bucket
by Thiago De Moraes Teixeira 27 Jun '24

27 Jun '24

Hi, I'm using swift-account-reaper for accounts with buckets in an unsharded state, and it looks fine, wiping the account. But when I try to wipe an account with sharded buckets, I encounter some problematic scenarios. First, when deleting the account, account-reaper lists objects from the sharded bucket, an empty list is returned from *direct_get_container* function, so the reaper interprets it as an empty bucket, and the delete fails with a 409 Conflict because objects exist inside the bucket. My second thought was to delete the account that holds shard buckets (i.e. .shards_AUTH_<project_id>) so the reaper will wipe this account. Then, with all buckets empty, they'll be deleted from the principal account (AUTH_<project_id>). But in this case, when the reaper tries to delete objects inside this account, all objects are not found because the object path used is "/.shards_AUTH_<project_id>/<bucket>-<hash>- <timestamp>-<shard_index>/<key>" instead of "/AUTH_<project_id>/ <bucket>/<key>". Am I doing something wrong in trying to delete this account? Or does account-reaper, in fact, not support sharded buckets? I'm exploring some patches to make it work, and I can volunteer to work on this fix if necessary. Thanks! -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*

2 5

[neutron] Neutron CI meeting
by Rodolfo Alonso Hernandez 26 Jun '24

26 Jun '24

Hello Neutrinos: This is a personal request. I no longer can attend the CI meeting on Tuesdays at 15:00UTC. Due to the usually low attendance to this meeting, I would like to propose a new slot (if possible) to have this meeting weekly and be able to participate, as I've been doing during the last years. I will also raise this question in the next team meeting (Tuesdays at 14:00UTC). Just out of curiosity, I've created a poll with 3 slots (including the current one), to know what are your preferences. Please, vote only if you usually attend this meeting [1]. Thank you in advance. [1]https://doodle.com/meeting/participate/id/e3KAwKRe

1 0

[nova] Next THREE nova meetings are at risk
by Sylvain Bauza 25 Jun '24

25 Jun '24

Hey folks, Sorry for the very short notice, but I just realized that I can't attend : - today's nova meeting on June 25th (because of some appointment at the same time) - next week's nova meeting on July 2nd (because on PTO for that only day) - other week's nova meeting on July 9th (because of a one-week PTO) As a result, unless someone is proposing to chair one or other meeting, we may need to cancel them. As a reminder, we'll have a spec review day on July 1st, so please prepare your specs in advance. Anyone from the community fancy to run those meetings ? Thanks, -Sylvain

2 1

[openstack-ansible] Centralized logging
by MARCO MOSCATELLI 25 Jun '24

25 Jun '24

HI everyone I'm looking for a system to have centralized logging of all servers and services of an openstack deployment installed via openstack-ansible, have you some suggestions? Elasticsearch or Opensearch? Have you some reference for the installation with openstack-ansible? thanks Best regards Marco

1 0

[neutron] Bug deputy report for week of June 17th
by Brian Haley 25 Jun '24

25 Jun '24

Hi, I was the stand-in Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2069689 - broken out-of-tree modules due to the movement of wsgi module - n-d-r and vpnaas patches have merged - https://review.opendev.org/c/openstack/neutron/+/922185 necessary? High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2069704 - rally jobs broken on unmaintantained/yoga and older branches - https://review.opendev.org/c/openstack/neutron/+/922230 * https://bugs.launchpad.net/neutron/+bug/2069831 - [eventlet] Deprecation of eventlet in Neutron - Rodolfo assigned to himself - All related patches will be tracked here Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/2069581 - [WSGI] Missing periodic workers when using WSGI server - https://review.opendev.org/c/openstack/neutron/+/922090 - Other patches still in-progress * https://bugs.launchpad.net/neutron/+bug/2069625 - [OVN] classless-static-route DHCP option is not sanitized and overrides default routes - https://review.opendev.org/c/openstack/neutron/+/922119 * https://bugs.launchpad.net/neutron/+bug/2069718 - [ovn] No connection to VM during live-migration - Seems valid, will discuss at weekly meeting * https://bugs.launchpad.net/neutron/+bug/2069744 - [FT] Run test_periodic_sync_routers_task tests serially - https://review.opendev.org/c/openstack/neutron/+/922218 Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/2069595 - Neutron processes not following the same naming convention - Rodolfo assigned to himself, patches in progress Misc bugs --------- * https://bugs.launchpad.net/neutron/+bug/2069753 - [question] best practices for using RDMA? - Responded with some answers and ask for ML thread Wishlist bugs ------------- * None

1 0

[manila] stuck at connecting ceph backend with manila
by Shubham Kumar Yadav 25 Jun '24

25 Jun '24

hello, I'm trying to install manila in my openstack cloud but having problem *1 -> ceph -n client.manila fs subvolume ls myfs [errno 1] RADOS permission error (error connecting to the cluster)* this is what i get after running that not sure what permissions any idea? i had shared my confs earlier Hope to hear from you soon Thank you!

1 0

Migratekit: near-live migrations from VMware to OpenStack
by Mohammed Naser 25 Jun '24

25 Jun '24

Hi everyone! Over the last few weeks, I’ve been trying to solve the issue of migrations from VMware to OpenStack since they are very downtime heavy at the moment with the existing toolset. In addition, there seems to be no readily available open source projects that can handle the entire process from end to end. After a ton of research and leveraging awesome other open source projects (libnbd, ndbkit, etc), I was able to build a project that allows users to migrate from VMware to OpenStack with almost no downtime. https://github.com/vexxhost/migratekit You can find the documentation on how to use it in the README. I am hoping this is useful for others and that we start to have other people start contributing to the project and testing it in their environments. In all transparency, this was built against an ESXi 8.0 environment (no vCenter) using an evaluation license. We welcome those who have environments that they can test against and hope to increase the coverage for versions. 😊 Please try and give feedback and we hope to that folks find this useful and contribute back to have a fully open sourced way of migrating from VMware. There’s a lot more work to do, but this is a solid start at this point. 😊 Thanks, Mohammed

4 3

[publiccloud-sig] Reminder - next meeting June 26th - 0700 UTC
by Felix Kronlage-Dammers 25 Jun '24

25 Jun '24

Hi everyone, This Wednesday (June 26th, that is tomorrow) the next meeting of the public cloud sig is going to happen. We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you on Wednesday! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com <mailto:fkr@osb-alliance.com>

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-14)
by Goutham Pacha Ravi 25 Jun '24

25 Jun '24

Hello Stackers, We're halfway through the 2024.2 release cycle! Next Thursday (2024-07-04) is milestone 2 [1] of this "Dalmatian" release cycle. This is an important milestone for OpenStack project teams to target bug fixes and it serves as a deadline for project-specific activities. The Technical Committee also uses this milestone to vet project activity to adjust the list of Emerging and Inactive OpenStack projects [2]. We'll likely see some proposals regarding this in the next week. During this past week, we merged governance proposals to nominate a PTL for the OpenStack Charms project and to nominate individual TC members as liaisons to projects with distributed project leadership. Looking ahead at the next release; there are a few important activities that need your attention: - The Virtual OpenInfra Project Teams Gathering is scheduled for October 21-25, 2024. Please read Kendall Nelson's (diablo_rojo) note here [3]. Project teams will be signing up over the next several weeks; I'd encourage you all to confirm your attendance by grabbing a ticket! [4] - Allison Price (aprice) sent out a poll to find a codename for the 2025.1 release. I encourage you all to participate - the deadline is June 28th, 2024! [5] I don't know if we have officially accepted that we have an animal theme going; shouldn't we? :) CentOS 8 Stream has reached end-of-life, and Clark Boylan (clarkb) informed OpenDev infrastructure consumers that centos-8-stream nodes will be removed from nodepool/Zuul CI [6]. Project teams are encouraged to check their use of centos-8-stream across all maintained/"unmaintained" branches in their repositories and modify/delete such jobs. Don't forget to address any documentation within your project teams! === Weekly Meeting === Last week's IRC meeting [7] began with a recap of the effort to secure our deliverables on PyPI. Our efforts to rein in "unowned" packages and packages with inconsistent metadata will continue. We considered the idea of adding a second maintainer account across all packages. While we agree on the benefits of a second maintainer account, given that there's no viable API to automate adding/adjusting maintainers, achieving this will be a laborious manual process. We have shelved the idea for now and are going to watch for updates on PyPI's community accounts or for a meaningful maintainer API. We then delved into persistent CI failures and gate stability. We took an action item to skip slow tests in integrated jobs using Ceph that are frequently timing out or finishing just around the permitted timeout. The next OpenStack PTL and TC Election season begins in approximately 50 days (Nomination period opens August 14th [1]). We'll attempt to get any election-related changes in place before the election season. Our next IRC weekly meeting will be held tomorrow (2024-06-25) at 18:00 UTC in OFTC's #openstack-tc channel. Please find the agenda at [8] and do join us! === Governance Proposals === ==== Merged ==== Fixing the validate-legacy script | https://review.opendev.org/c/openstack/governance/+/920848 Appoint James Page as OpenStack Charms PTL | https://review.opendev.org/c/openstack/governance/+/921731 Add TC liaison in DPL model implementation | https://review.opendev.org/c/openstack/governance/+/917516 ==== Open for review/discussion ==== Retire kuryr-kubernetes and kuryr-tempest-plugin | https://review.opendev.org/c/openstack/governance/+/922507 Update criteria for the inactive projects to become active again | https://review.opendev.org/c/openstack/governance/+/921500 Adding a rule to manage the affiliation diversity requirement in the TC | https://review.opendev.org/c/openstack/governance/+/922512 Remove Eventlet From Openstack | https://review.opendev.org/c/openstack/governance/+/902585 Show TC and SIG repos under reference | https://review.opendev.org/c/openstack/governance/+/918488 Adding a rule to manage the affiliation diversity requirement in the TC | https://review.opendev.org/c/openstack/governance/+/922512 How to contact the TC: ================== You can reach out to the TC in multiple ways: 1. Email: you can send the email with tag [tc] on this email list. 2. Ping us using 'tc-members' keyword on #openstack-tc IRC channel 3. Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [6] Upcoming Events: ============== 2024-07-01: Monthly Technical Committee Video+IRC meeting 2024-07-02: OpenInfra Board Meeting -: https://board.openinfra.dev/meetings/2024-07-02 2024-07-04: Milestone-2 of the 2024.2 release cycle Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] Emerging Technology and Inactive Projects: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [3] 2025.1 PTG: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] PTG Registration: https://ptg.openinfra.dev/#registration=1 [5] Poll for naming the 2025.1 release: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [6] CentOS-8-stream node removal: https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… [7] TC Meeting Log: https://meetings.opendev.org/meetings/tc/2024/tc.2024-06-18-18.00.log.html [8] TC Meeting Agenda: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[publiccloud-sig] Reminder - next meeting June 26th - 0700 UTC
by Felix Kronlage-Dammers 24 Jun '24

24 Jun '24

Hi everyone, This Wednesday (June 26th) the next meeting of the public cloud sig is going to happen. We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you on Wednesday! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com <mailto:fkr@osb-alliance.com>

1 0