June 2024 - openstack-discuss - lists.openstack.org

[nova] volume information was duplicated when Live Migration failed
by Junya Noguchi (Fujitsu) 29 Jun '24

29 Jun '24

To whom it may concern Hi. I encountered an issue with OpenStack where volume information was duplicated when Live Migration failed. Could you please provide a solution to this issue? [Issue] After a Live Migration failure, when I checked the volume where the error occurred using the 'openstack volume list' command, the following was displayed: +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda | I expect that the 'Attached to' display should only appear once as there is one volume connected, but it is duplicated for each failed attempt. The versions of OpenStack and nova are as follows: OpenStack Zed $ openstack --version openstack 6.0.0 $ nova --version nova CLI is deprecated and will be a removed in a future release 18.1.0 [Details] The volume display is duplicated under the following conditions: - When Live Migration of an instance fails If the communication between the ports required for Live Migration is closed, Live Migration will fail when executed. $ sudo netstat -tuln | grep 16509 $ The output of nova-compute.log at the time of Live Migration failure is as follows: --- 2024-02-28 17:04:32.119 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Live Migration failure: unable to connect to server at 'XXXXX.com:16509': Connection refused: libvirt.libvirtError: unable to connect to server at 'XXXXX.com:16509': Connection refused 2024-02-28 17:04:32.119 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation thread notification thread_finished /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10516 2024-02-28 17:04:32.607 2785 DEBUG nova.virt.libvirt.migration [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] VM running on src, migration failed _log /usr/lib/python3.9/site-packages/nova/virt/libvirt/migration.py:432 2024-02-28 17:04:32.608 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10330 2024-02-28 17:04:32.608 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation has aborted --- After Live Migration, when I check the volume where the error occurred using the 'openstack volume list' command, the following is displayed: +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda | '73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda' is displayed as many times as Live Migration failed. The duplicate display did not disappear even after Live Migration was successful. Thanks,

2 1

2025.1 E OpenStack Release Naming Poll
by Allison Price 29 Jun '24

29 Jun '24

Hi everyone, It’s that time again! Please join me in voting to name the 31st release of OpenStack, starting with the letter ‘E’! https://civs1.civs.us/cgi-bin/vote.pl?id=E_d8469898c60545f2&akey=a16aa8cad5… The deadline to vote is Friday, June 28 at 7 UTC. Cheers, Allison

2 1

Warehouse failure
by ゞPositive î 29 Jun '24

29 Jun '24

The Centos system has stopped updating now, and we hope that the official website manual Centos can be changed to Rocky Linux. Now we are using Rocky Linux to install OpenStack using the Centos system installation method. Among them, the installation source link for yum install Centos release openstack ussuri and yum install Centos release openstack xena is invalid. We hope to receive your solution. ゞPositive ? 451526204(a)qq.com  

1 0

Openstack-ansible upgrade
by Danish Khan 28 Jun '24

28 Jun '24

Dear Team, We have few openstack-ansible Train release in our environment. We want to upgrade these setup to atleast Yoga release. Currently we are doing upgrade step by step. like from Train to Ussuri then to Victoria then to wallaby and so on(in N+1 series). Can we skip few releases and directly go to Yoga or any other feasible release? Regards, Danish Khan

4 3

[release] Release countdown for week R-13, Jul 01-05
by Elõd Illés 28 Jun '24

28 Jun '24

Development Focus ----------------- The Dalmatian-2 milestone is next week, on July 4th, 2024! 2024.2 Dalmatian-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/dalmatian/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library that has not been otherwise released since milestone 1. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. Next week is also the deadline to freeze the contents of the final release. All new '2024.2 Dalmatian' deliverables need to have a deliverable file in https://opendev.org/openstack/releases/src/branch/master/deliverables and need to have done a release by milestone-2. Upcoming Deadlines & Dates -------------------------- Dalmatian-2 Milestone: July 4th, 2024 Előd Illés irc: elodilles @ #openstack-release

1 0

networking-onos plugin
by Ganesan, Sathya 28 Jun '24

28 Jun '24

Good Day, Is there an alternative integration available for ONOS and Openstack since networking-onos is archived? Sathya Ganesan Geico Tech | Data Center Core Network Engineering - SDN | sganesan(a)geico.com<mailto:sganesan@geico.com> | 240.478.2121 | M-F 9a-5:30p EST Sensitivity: Company-Internal ==================== This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message.

1 0

Re: [manila] stuck at connecting ceph backend with manila
by Goutham Pacha Ravi 28 Jun '24

28 Jun '24

Hey Shubham, Please keep the conversation on the list. On Thu, Jun 27, 2024 at 10:36 PM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: > > Hi, actually the keyring file contains the manila keyring which is mounted on manila-share pod > > 1 -> ls -l /etc/ceph (from manila-share pod) > total 12 > -rw-r--r-- 1 root root 279 Jun 26 08:02 ceph.conf > -rw-r--r-- 1 root root 62 Jun 26 08:02 keyring > -rw-r--r-- 1 root root 92 Mar 4 20:27 rbdmap > > 2 -> i have already generated manila keyring Do you mean the content of the keyring is in the "keyring" file? In your earlier email you showed the contents of your ceph.conf, it referred to the file name of the manila keyring as "/etc/ceph/manila.keyring". I don't see that file in the output above. > > On Fri, 28 Jun 2024 at 00:46, Goutham Pacha Ravi <gouthampravi(a)gmail.com> wrote: >> >> Hi, >> >> On Thu, Jun 27, 2024 at 12:26 AM Shubham Kumar Yadav >> <shubham.kumar.yadav369(a)gmail.com> wrote: >> > >> > Hi, these are the outputs you asked for >> > >> > 1 -> ls -l /etc/ceph >> > total 12 >> > -rw-r--r-- 1 root root 279 Jun 26 08:02 ceph.conf >> > -rw-r--r-- 1 root root 62 Jun 26 08:02 keyring >> > -rw-r--r-- 1 root root 92 Mar 4 20:27 rbdmap >> >> The manila keyring is missing here; Can you add it here and retry? >> >> >> > >> > >> > 2 -> ceph --verbose fs ls -n client.manila >> > parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name='client.manila', cluster=None, cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['fs', 'ls'] >> > [errno 1] RADOS permission error (error connecting to the cluster) >> > >> > 3 -> i created keyring using >> > >> > ceph auth get-or-create client.manila -o manila.keyring \ >> > mgr 'allow rw' \ >> > mon 'allow r' >> > >> > >> > Hope to hear from you soon >> > Thank you! >> > >> > >> > >> > On Thu, 27 Jun 2024 at 05:25, Goutham Pacha Ravi <gouthampravi(a)gmail.com> wrote: >> >> >> >> The client capabilities look correct; Can you paste the output of "ls >> >> -l /etc/ceph" from the manila-share pod? >> >> Also attempt to use "--verbose" with the Ceph CLI; maybe there'll be a >> >> helpful log: "ceph --verbose fs ls -n client.manila" >> >> >> >> I'm unsure if you're creating the keyring file by hand.. but if you >> >> are, you can use this command to generate a file instead: >> >> >> >> ceph auth get-or-create client.manila -o /etc/ceph/ceph.client.manila.keyring >> >> You can use this ^ command to generate it and then make sure to copy >> it into the /etc/ceph folder that's mounted into the manila-share pod. >> >> >> >> >> >> >> >> >> On Wed, Jun 26, 2024 at 12:02 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> >> > >> >> > Mmm wait, I believe I'm mistaken, I didn't read the entire config file. >> >> > Please omit my comment. >> >> > >> >> > On Wed, Jun 26, 2024, 12:54 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> >> >> >> >> >> Hi again, >> >> >> It seems you are missing osd permission on manila user authorization: >> >> >> >> >> >> Allow manila user to read/write on osd's >> >> >> >> >> >> Cheers. >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Jun 26, 2024, 7:36 AM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: >> >> >>> >> >> >>> hello, >> >> >>> I'm trying to install manila in my openstack cloud but having problem with "Authorizing the driver to communicate with Ceph" >> >> >>> >> >> >>> 1 -> i have created a configmap with these confs and mounted it on manila-share pod >> >> >>> >> >> >>> [client.manila] >> >> >>> client mount uid = 0 >> >> >>> client mount gid = 0 >> >> >>> log file = /opt/stack/logs/ceph-client.manila.log >> >> >>> admin socket = /opt/stack/status/stack/ceph-$name.$pid.asok >> >> >>> keyring = /etc/ceph/manila.keyring >> >> >>> >> >> >>> >> >> >>> 2 -> then i have created a secret with client.manila keyring and attached it to the manila-share pod at >> >> >>> >> >> >>> keyring = /etc/ceph/manila.keyring >> >> >>> >> >> >>> 3 -> inside manila-share pod i see the logs >> >> >>> >> >> >>> 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 >> >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 >> >> >>> >> >> >>> >> >> >>> 4 -> My ceph cluster is healthy >> >> >>> >> >> >>> 5 -> My manila confs look like (what could be wrong here)? >> >> >>> >> >> >>> conf: >> >> >>> manila: >> >> >>> DEFAULT: >> >> >>> debug: true >> >> >>> enabled_share_protocols: NFS,CIFS,CEPHFS >> >> >>> enabled_share_backends: cephfsnative1 >> >> >>> cephfsnative1: >> >> >>> driver_handles_share_servers: False >> >> >>> share_backend_name: CEPHFSNATIVE1 >> >> >>> share_driver: manila.share.drivers.cephfs.driver.CephFSDriver >> >> >>> cephfs_conf_path: /etc/ceph/ceph.conf >> >> >>> cephfs_protocol_helper_type: CEPHFS >> >> >>> cephfs_auth_id: manila >> >> >>> cephfs_cluster_name: rook-ceph ( my rook-ceph cluster name) >> >> >>> cephfs_filesystem_name: myfs (i created this fs manually) >> >> >>> >> >> >>> 6 -> Permissions for the user manila >> >> >>> >> >> >>> ceph auth get-or-create client.manila -o manila.keyring \ >> >> >>> mgr 'allow rw' \ >> >> >>> mon 'allow r' >> >> >>> >> >> >>> >> >> >>> 7 -> ceph --version >> >> >>> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> >> >>> >> >> >>> openstackhelm Manila Chart >> >> >>> name: manila >> >> >>> version: 0.1.7 >> >> >>> >> >> >>> 8 -> ceph -n client.manila fs ls >> >> >>> [errno 1] RADOS permission error (error connecting to the cluster) >> >> >>> (while it works with admin) >> >> >>> >> >> >>> Hope to hear from you soon with a solution >> >> >>> Thank you!

1 0

[manila] stuck at connecting ceph backend with manila
by Goutham Pacha Ravi 27 Jun '24

27 Jun '24

Hi, On Thu, Jun 27, 2024 at 12:26 AM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: > > Hi, these are the outputs you asked for > > 1 -> ls -l /etc/ceph > total 12 > -rw-r--r-- 1 root root 279 Jun 26 08:02 ceph.conf > -rw-r--r-- 1 root root 62 Jun 26 08:02 keyring > -rw-r--r-- 1 root root 92 Mar 4 20:27 rbdmap The manila keyring is missing here; Can you add it here and retry? > > > 2 -> ceph --verbose fs ls -n client.manila > parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name='client.manila', cluster=None, cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['fs', 'ls'] > [errno 1] RADOS permission error (error connecting to the cluster) > > 3 -> i created keyring using > > ceph auth get-or-create client.manila -o manila.keyring \ > mgr 'allow rw' \ > mon 'allow r' > > > Hope to hear from you soon > Thank you! > > > > On Thu, 27 Jun 2024 at 05:25, Goutham Pacha Ravi <gouthampravi(a)gmail.com> wrote: >> >> The client capabilities look correct; Can you paste the output of "ls >> -l /etc/ceph" from the manila-share pod? >> Also attempt to use "--verbose" with the Ceph CLI; maybe there'll be a >> helpful log: "ceph --verbose fs ls -n client.manila" >> >> I'm unsure if you're creating the keyring file by hand.. but if you >> are, you can use this command to generate a file instead: >> >> ceph auth get-or-create client.manila -o /etc/ceph/ceph.client.manila.keyring You can use this ^ command to generate it and then make sure to copy it into the /etc/ceph folder that's mounted into the manila-share pod. >> >> >> >> On Wed, Jun 26, 2024 at 12:02 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> > >> > Mmm wait, I believe I'm mistaken, I didn't read the entire config file. >> > Please omit my comment. >> > >> > On Wed, Jun 26, 2024, 12:54 PM Alvaro Soto <alsotoes(a)gmail.com> wrote: >> >> >> >> Hi again, >> >> It seems you are missing osd permission on manila user authorization: >> >> >> >> Allow manila user to read/write on osd's >> >> >> >> Cheers. >> >> >> >> >> >> >> >> On Wed, Jun 26, 2024, 7:36 AM Shubham Kumar Yadav <shubham.kumar.yadav369(a)gmail.com> wrote: >> >>> >> >>> hello, >> >>> I'm trying to install manila in my openstack cloud but having problem with "Authorizing the driver to communicate with Ceph" >> >>> >> >>> 1 -> i have created a configmap with these confs and mounted it on manila-share pod >> >>> >> >>> [client.manila] >> >>> client mount uid = 0 >> >>> client mount gid = 0 >> >>> log file = /opt/stack/logs/ceph-client.manila.log >> >>> admin socket = /opt/stack/status/stack/ceph-$name.$pid.asok >> >>> keyring = /etc/ceph/manila.keyring >> >>> >> >>> >> >>> 2 -> then i have created a secret with client.manila keyring and attached it to the manila-share pod at >> >>> >> >>> keyring = /etc/ceph/manila.keyring >> >>> >> >>> 3 -> inside manila-share pod i see the logs >> >>> >> >>> 2024-06-03 09:00:53.682 6 INFO manila.share.drivers.cephfs.driver [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] [CEPHFSNATIVE1] Ceph client found, connecting... >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Error encountered during initialization of driver CephFSDriver@lab-11053-26006-ceph-2@cephfsnative1: manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 421, in rados_client >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self._rados_client.connect() >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "rados.pyx", line 690, in rados.Rados.connect >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados.IOError: [errno 5] RADOS I/O error (error connecting to the cluster) >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager During handling of the above exception, another exception occurred: >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager Traceback (most recent call last): >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/manager.py", line 371, in _driver_setup >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager self.driver.do_setup(ctxt) >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 274, in do_setup >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager rados_client=self.rados_client, >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager File "/var/lib/openstack/lib/python3.10/site-packages/manila/share/drivers/cephfs/driver.py", line 424, in rados_client >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager raise exception.ShareBackendException( >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager manila.exception.ShareBackendException: [%(be)s] Ceph client failed to connect. >> >>> 2024-06-03 09:00:53.713 6 ERROR manila.share.manager >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Finished call to 'manila.share.manager.ShareManager.init_host.<locals>._driver_setup' after 2222.651(s), this was the 12th time calling it. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/after.py:44 >> >>> 2024-06-03 09:00:53.714 6 DEBUG manila.utils [None req-22cd7b29-a383-44cd-8cb8-1c639a3034dc - - - - - -] Retrying manila.share.manager.ShareManager.init_host.<locals>._driver_setup in 600.0 seconds as it raised ShareBackendException: [%(be)s] Ceph client failed to connect.. log_it /var/lib/openstack/lib/python3.10/site-packages/tenacity/before_sleep.py:65 >> >>> >> >>> >> >>> 4 -> My ceph cluster is healthy >> >>> >> >>> 5 -> My manila confs look like (what could be wrong here)? >> >>> >> >>> conf: >> >>> manila: >> >>> DEFAULT: >> >>> debug: true >> >>> enabled_share_protocols: NFS,CIFS,CEPHFS >> >>> enabled_share_backends: cephfsnative1 >> >>> cephfsnative1: >> >>> driver_handles_share_servers: False >> >>> share_backend_name: CEPHFSNATIVE1 >> >>> share_driver: manila.share.drivers.cephfs.driver.CephFSDriver >> >>> cephfs_conf_path: /etc/ceph/ceph.conf >> >>> cephfs_protocol_helper_type: CEPHFS >> >>> cephfs_auth_id: manila >> >>> cephfs_cluster_name: rook-ceph ( my rook-ceph cluster name) >> >>> cephfs_filesystem_name: myfs (i created this fs manually) >> >>> >> >>> 6 -> Permissions for the user manila >> >>> >> >>> ceph auth get-or-create client.manila -o manila.keyring \ >> >>> mgr 'allow rw' \ >> >>> mon 'allow r' >> >>> >> >>> >> >>> 7 -> ceph --version >> >>> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> >>> >> >>> openstackhelm Manila Chart >> >>> name: manila >> >>> version: 0.1.7 >> >>> >> >>> 8 -> ceph -n client.manila fs ls >> >>> [errno 1] RADOS permission error (error connecting to the cluster) >> >>> (while it works with admin) >> >>> >> >>> Hope to hear from you soon with a solution >> >>> Thank you!

1 0

Antelope: strange failure in VM migration from a Yoga HV to an Antelope one
by Michel Jouvin 27 Jun '24

27 Jun '24

Hi, In our Yoga -> Antelope upgrade, we have been able to live migrate active VMs from most Yoga HV to an Antelope one. In one case, the migration starts well but at the end of the disk migration, there is an "unexpected error" and the migration is canceled. I have not been able to find any more detailed information about the failure. In the source HV, nova-compute log says: ------------------- 2024-06-13 13:41:09.544 4108750 INFO nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Migration running for 2804 secs, memory 100% remaining (bytes processed=0, remaining=0, total=0); disk 1% remaining (bytes processed=106548297728, remaining=863371264, total=107411668992). 2024-06-13 13:41:20.436 4108750 INFO nova.compute.resource_tracker [req-db52ee9a-b0e4-4ac1-83f6-f57600ccc275 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Updating resource usage from migration 35e1a5e8-0f7d-4ff9-9075-c8da30649b4a 2024-06-13 13:41:45.678 4108750 INFO nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Migration running for 2840 secs, memory 4% remaining (bytes processed=13900749654, remaining=691298304, total=16782270464); disk 0% remaining (bytes processed=107429101568, remaining=262144, total=107429363712). 2024-06-13 13:41:51.153 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] VM Paused (Lifecycle Event) 2024-06-13 13:41:51.212 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] During sync_power_state the instance has a pending task (migrating). Skip. 2024-06-13 13:41:51.390 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] VM Resumed (Lifecycle Event) 2024-06-13 13:41:51.426 4108750 INFO nova.compute.manager [req-115e45f4-d8c5-43d7-ae70-1608818eaf70 - - - - -] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] During sync_power_state the instance has a pending task (migrating). Skip. 2024-06-13 13:41:51.772 4108750 ERROR nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Migration operation has aborted 2024-06-13 13:41:51.774 4108750 ERROR nova.virt.libvirt.driver [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Live Migration failure: internal error: QEMU unexpectedly closed the monitor (vm='instance-00042fe6'): 2024-06-13T10:54:25.319005Z qemu-kvm: warning: Machine type 'pc-i440fx-rhel7.6.0' is deprecated: machine types for previous major releases are deprecated 2024-06-13 13:41:51.806 4108750 INFO nova.compute.manager [req-565bd94e-9a3d-43b9-a42d-80ea6545c0a6 5c0b6777dafaad4f7a2e9d3b1959ff732468809f18ef35286eb123fb10865760 1b693dd6ab914fc0b5398f96e7866dd2 - 278da0eaa8cc4870a72e3a8abbf9a3f0 4828d76008524b798a4f5f632ba26adf] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Swapping old allocation on dict_keys(['0a7be64c-cff9-4fa7-93ac-ddd0a965e93a']) held by migration 35e1a5e8-0f7d-4ff9-9075-c8da30649b4a for instance 2024-06-13 13:41:52.882 4108750 WARNING nova.compute.manager [req-8c267aa3-29dc-43ec-9a1b-9c4ada34dd3f f0d2557b870a44fa90c98b507629407e 5cf79c1092c44348addf9bee098f138e - default default] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Received unexpected event network-vif-unplugged-7446e7d3-a85d-4bd3-a609-decf5e7dd591 for instance with vm_state active and task_state None. 2024-06-13 13:41:55.758 4108750 WARNING nova.compute.manager [req-59c5fb1f-4223-4e14-90cf-c97bb0b9130a f0d2557b870a44fa90c98b507629407e 5cf79c1092c44348addf9bee098f138e - default default] [instance: 81d8ce0d-3120-45ad-b8c7-c219638fbbdd] Received unexpected event network-vif-plugged-7446e7d3-a85d-4bd3-a609-decf5e7dd591 for instance with vm_state active and task_state None. ----------------- It happens for all the VMs hosted by the source HV that have no link between them (not the same project) and whatever is the destination HV (same HW model) so I guess it has to do with the source HV. And there were no such problems when migrating VMs between the other HVs with the same HW model. Any idea where to look for details that may help to identify the problem? Best regards, Michel

1 2

[swift] Weird log corruption or something more serious?
by Sam Morrison 27 Jun '24

27 Jun '24

Hi, We are debugging a weird issue with our swift cluster (currently running 2.29.2) We got a report from a user that they were getting upload failures and then sent us this from their logs: 2024-06-25 23:30:06,829 | swiftclient.service | ERROR | Object PUT failed: https://swift-host/v1/PROJECT_ID_A/prod_result/195e5e8e-334a-11ef-ba70-0a58… 401 Unauthorized [first 60 chars of response] b'<html><h1>Unauthorized</h1><p>This server could not verify t' (txn: txab25fcfcf3354bbbaf105-00667b52fe) Searching our logs for that transaction ID I found: 203.101.227.241 127.0.0.1 25/Jun/2024/23/30/06 PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/blobs/default/content/vol-27/chap-25/6bc30bae-4f5e-40a8-bc63-096679af1a58.bytes HTTP/1.0 401 - rclone/v1.59.0%2Cpython-swiftclient-4.2.0 gAAAAABme1L7tlfE... - 131 - txab25fcfcf3354bbbaf105-00667b52fe - 0.0440 - - 1719358206.765954018 1719358206.809994936 0 Note this is a completely different project ID from our side, timestamp is correct too. We also have this: ERROR WSGI: code 400, message Bad request syntax ('\x89PNG') (txn: txab25fcfcf3354bbbaf105-00667b52fe) (client_ip: XX.XX.XX.XX) Upon further analysis we see this quite a lot, mostly from attempted hackers but lots from real users and lots seem to be related to rclone. Eg. Jun 27 08:24:46 swift-host proxy-server[70043]: ERROR WSGI: code 400, message Bad request syntax ('PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/elasticsearch/nexus/nodes/0/indices/42b6a6a76dae683a647546ac02a6883108ac7df5/0/index/segments_4PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/elasticsearch/nexus/nodes/0/indices/4db62ff1327dfe236d85b1dd1a6a9e8139e7e4cf/0/index/_67t.cfs HTTP/1.1') (txn: txd57a3d00bb1942c6a390e-00667c952a) (client_ip: XX.XX.XX.XX) Anyone know what is going on here? Thanks, Sam

1 0