March 2023 - openstack-discuss - lists.openstack.org

Re: [cinder] Error when creating backups from iscsi volume
by Gorka Eguileor 09 Mar '23

09 Mar '23

On 06/03, Rishat Azizov wrote: > Hi, > > It works with smaller volumes. > > multipath.conf attached to thist email. > > Cinder version - 18.2.0 Wallaby Hi, After giving it some thought I think I may know what is going on. If you have DEBUG logs enabled in cinder-backup when it fails, how many calls do you see in the cinder-backup to "multipath -f" from os-brick, only one or do you see more? Cheers, Gorka. > > пн, 6 мар. 2023 г. в 17:35, Gorka Eguileor <geguileo(a)redhat.com>: > > > On 16/02, Rishat Azizov wrote: > > > Hello! > > > > > > We have an error with creating backups from iscsi volume. Usually, this > > > happens with large backups over 100GB. > > > > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > [req-f6619913-6f96-4226-8d75-2da3fca722f1 > > 23de1b92e7674cf59486f07ac75b886b > > > a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message > > handling: > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > while > > > running command. > > > Command: multipath -f 3624a93705842cfae35d7483200015ec6 > > > Exit code: 1 > > > Stdout: '' > > > Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a > > > multipath device\n' > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Traceback > > > (most recent call last): > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line > > 165, > > > in _process_incoming > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server res = > > > self.dispatcher.dispatch(message) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line > > > 309, in dispatch > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > self._do_dispatch(endpoint, method, ctxt, args) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line > > > 229, in _do_dispatch > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server result = > > > func(ctxt, **new_args) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > func(self, *args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in > > > create_backup > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > volume_utils.update_backup_error(backup, str(err)) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in > > > __exit__ > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > self.force_reraise() > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in > > > force_reraise > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server raise > > > self.value > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in > > > create_backup > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server updates > > = > > > self._run_backup(context, backup, volume) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in > > > _run_backup > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > ignore_errors=True) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066, > > in > > > _detach_device > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > force=force, ignore_errors=ignore_errors) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in > > > trace_logging_wrapper > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > f(*args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line > > 360, > > > in inner > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > f(*args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > line 880, in disconnect_volume > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > is_disconnect_call=True) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", > > > line 942, in _cleanup_connection > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > self._linuxscsi.flush_multipath_device(multipath_name) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line > > > 382, in flush_multipath_device > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > root_helper=self._root_helper) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in > > > _execute > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server result = > > > self.__execute(*args, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line > > > 172, in execute > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > execute_root(*cmd, **kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line > > 247, > > > in _wrap > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return > > > self.channel.remote_call(name, args, kwargs) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File > > > "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in > > > remote_call > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server raise > > > exc_type(*result[2]) > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error > > while > > > running command. > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command: > > > multipath -f 3624a93705842cfae35d7483200015ec6 > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit code: 1 > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: '' > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: 'Feb > > > 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath > > device\n' > > > 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server > > > > > > Could you please help with this error? > > > > Hi, > > > > Does it work for smaller volumes or does it also fail? > > > > What are your defaults in your /etc/multipath.conf file? > > > > What Cinder release are you using? > > > > Cheers, > > Gorka. > > > > > defaults { > user_friendly_names no > find_multipaths yes > enable_foreign "^$" > } > > blacklist_exceptions { > property "(SCSI_IDENT_|ID_WWN)" > } > > blacklist { > } > > devices { > device { > vendor "PURE" > product "FlashArray" > fast_io_fail_tmo 10 > path_grouping_policy "group_by_prio" > failback "immediate" > prio "alua" > hardware_handler "1 alua" > max_sectors_kb 4096 > } > }

1 0

OpenInfra Live March 9, 2023, at 9 a.m. CT
by Kristin Barrientos 08 Mar '23

08 Mar '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by OpenMetal Episode: Examination of Cost Differences b/w Private Cloud and Hyperscalers Todd Robinson, president of OpenMetal, will be discussing the examination of cost differences between private cloud and hyperscalers. Date and time: March 9, 2023, at 9 a.m. CT (15:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=6GCGhuRpPqM LinkedIn: https://www.linkedin.com/events/7038909988492242944/comments/ WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Todd Robinson Have an idea for a future episode? Share it now at ideas.openinfra.live. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[Magnum] Magnum service status in Openstack Antelope
by A Tom 08 Mar '23

08 Mar '23

Hi, Will magnum be supported in Antelope and further releases of Openstack? I was looking for some informations about this and I’m quite confused, because here I don’t see Magnum service: https://releases.openstack.org/antelope/index.html and here Magnum is mentioned: https://docs.openstack.org/2023.1.antelope/projects.html Which version of kubernetes is supported in Zed, and which (if) will be in Antelope? Because there’s no such information in compatibility matrix: https://wiki.openstack.org/wiki/Magnum Best regards, Adam Tomas

3 2

[magnum] security groups for magnum nodes
by Vivian Rook 08 Mar '23

08 Mar '23

Is there an option for adding security groups to a given magnum template, and thus the nodes that such a template would create? I have an NFS server, and it is setup to only allow connections from nodes with the "nfs" security group. A few pods in my cluster mount the NFS server, and are blocked as a result. Is it possible to setup magnum so that it adds the "nfs" security group to the worker nodes (it would be alright if it has to be worker and control nodes)? Thank you! -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

2 2

[neutron] New Neutron releases for Xena, Yoga and Zed
by Rodolfo Alonso Hernandez 08 Mar '23

08 Mar '23

Hello Neutrinos: Please check the patches for the new stable releases for the Neutron projects: * Xena: https://review.opendev.org/c/openstack/releases/+/876827 * Yoga: https://review.opendev.org/c/openstack/releases/+/876828 * Zed: https://review.opendev.org/c/openstack/releases/+/876835 Feel free to comment on the patch if you found something wrong or a missing pending patch. Regards.

1 0

[largescale-sig] Next meeting: March 8, 9utc
by Thierry Carrez 08 Mar '23

08 Mar '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC, our APAC+EU-friendly time. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230308T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[horizon] Cancelling Today's Weekly meeting
by vishal manchanda 07 Mar '23

07 Mar '23

Hello Team, As it is a holiday for me, I will not be able to host today's horizon weekly meeting. So let's cancel today's weekly meeting. If anything urgent, please reach out to the horizon core team. Thanks & regards, Vishal Manchanda

1 0

[cinder] Could I use system lvm volume group for cinder-volume
by 韩光宇 07 Mar '23

07 Mar '23

Hello, I have a physical node and the system uses lvm partitions. With the service already running above, is it possible for me to deploy cinder-volume that uses the lvm backend without affecting the existing service? The physical disk space is basically allocated to the /dev/sda4 physical volume, and the pv wad added to the `uniontechos` volume group. All the space of `uniontechos` vg is allocated to the system logical volume (/dev/uniontechos/root), there is 5T space (a lot of free space). I want to try to reduce the system logical volume——/dev/uniontechos/root, and let cinder-volume use the uniontechos volume group at the same time , I don't know if it is feasible. Could I get some advices? Thank for any help! ```shell # pvdisplay ... --- Physical volume --- PV Name /dev/sda4 VG Name uniontechos PV Size <5.43 TiB / not usable 2.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 1422313 Free PE 0 Allocated PE 1422313 PV UUID 8LF270-LYD1-kuP1-iWZb-BZEQ-LdA8-UwEa58 # lvdisplay --- Logical volume --- LV Path /dev/uniontechos/root LV Name root VG Name uniontechos LV UUID cEypcY-xcbC-JFoO-d3MS-uMPU-ntQa-WEeE1c LV Write Access read/write LV Creation host, time compute2, 2022-04-28 15:21:27 +0800 LV Status available # open 1 LV Size 5.42 TiB Current LE 1421289 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0 ``` Best wishes, Han

1 0

[magnum] certificate authority key location
by Vivian Rook 07 Mar '23

07 Mar '23

If I want to create a credential for a user to access a magnum cluster I can do so as described in https://docs.openstack.org/magnum/latest/user/#id4 Namely by running: openstack coe ca sign secure-k8s-cluster client.csr > cert.pem I would like to do this without calling the openstack cli. Where does magnum store its ca key file? I could not find it on the control node under /etc/kubernetes/certs (the ca.crt is there, though no ca.key) Thank you! -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

[nova][ptg] Strawman proposal for vPTG timeslots
by Sylvain Bauza 07 Mar '23

07 Mar '23

Hi team, We recently discussed this in the Nova meeting [1] but I'd like to reemphasize here and now that I proposed to book 4 timeslots of one hour during 4 days for the next vPTG. https://ptg.opendev.org/ptg.html As you see, the proposed timeline will be : Tuesday, Wednesday, Thursday, Friday between 13:00UTC and 17:00UTC. May you have concerns with this proposal, please express them by replying to this thread. As a reminder, please add the topics you'd like to cover during the vPTG in the PTG etherpad : https://etherpad.opendev.org/p/nova-bobcat-ptg Thanks, -Sylvain [1] https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-07-16.00.log.h…

1 0