June 2023 - openstack-discuss - lists.openstack.org

Strange Cinder rpc timeouts after Xena upgrade
by Colby Walsworth 09 Jun '23

09 Jun '23

Hey Everyone, We just upgraded from Wallaby to Xena. We use Ceph as our volume/image backend. All instances from before the upgrade seem to be unable to be shelved/unshelved or live migrated (have not tried offline migration yet). Any new instances that are created work fine and all those tasks work fine on those. The problem seems to be with cinder: cinder.api.v3.attachments [req-2af34fab-a4b5-43f3-bd9a-94f360201533 e28435e0a66740968c523e6376c57f68 179f701c810c425ab004548cc1f76bc9 - default default] Unable to create attachment for volume.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 8eea6a00568a477c89ced8db17f1179d the migrations and shelve/unshelve processes start on the hypervisor but eventually error out with the above message. Any new instances I create do not have this issue at all. I have done all the db upgrades so I don't believe this is a db record update issue. Any ideas of what might be causing this? Thanks, Colby

2 1

[ironic][barbican][qa][tripleo][nova][opendev] Next steps for Fedora jobs in OpenStack
by Tony Breeds 08 Jun '23

08 Jun '23

Hello all, Following on from "Future of Fedora and CentOS Stream Test Images"[1] discussion on service-discuss the opendev infra team have removed the unused Fedora content from the AFS mirrors. Now it's time to look at removing the Fedora-36 content from the mirrors. I'd like to, again, point out the Fedora 36 isn't the latest release and is infact EOL[4]. Removal of the Fedora 36 content will start with something like "[configure-mirrors] Allow per distribution disabling of mirrors"[2]. Once merged, and activated, this means that any/all Fedora based jobs will pull OS content from the Fedora project mirrors directly. There isn't an ETA but "soon" is a fair estimate. Any Fedora jobs will potentially run slower and will be more subject to network failures, this is also the situation with existing rocky linux jobs and there hasn't been any major problems[3]. Projects using Fedora are encouraged to switch to CentOS Stream or Rocky Linux. As at some stage the Fedora nodepool images may also need to go away. OpenStack users of Fedora (master) appear to be: * name: bifrost-integration-tinyipa-fedora-latest where: https://opendev.org/openstack/bifrost/src/branch/master/zuul.d/bifrost-jobs… recent builds: https://zuul.openstack.org/builds?job_name=bifrost-integration-tinyipa-fedo… * name: bifrost-integration-redfish-uefi-fedora-latest where: https://opendev.org/openstack/bifrost/src/branch/master/zuul.d/bifrost-jobs… recent builds: https://zuul.openstack.org/builds?job_name=bifrost-integration-redfish-uefi… * name: barbican-dogtag-tox-functional where: https://opendev.org/openstack/barbican/src/branch/master/.zuul.yaml#L16 recent builds: https://zuul.openstack.org/builds?job_name=barbican-dogtag-tox-functional&b… * name: devstack-plugin-ceph-tempest-fedora-latest where: https://opendev.org/openstack/devstack-plugin-ceph/src/branch/master/.zuul.… recent builds: https://zuul.openstack.org/builds?job_name=devstack-plugin-ceph-tempest-fed… * name: devstack-platform-fedora-latest where: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L861 recent builds: https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&… * name: devstack-platform-fedora-latest-virt-preview where: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L868 recent builds: https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest-… Notes: This COPR has CentOS Stream packages * name: validations-libs-podified-podman where: https://opendev.org/openstack/validations-libs/src/branch/master/.zuul.yaml… recent builds: https://zuul.openstack.org/builds?job_name=validations-libs-podified-podman… * name: novajoin-functional where: https://opendev.org/x/novajoin/src/branch/master/.zuul.yaml#L13 recent builds: https://zuul.openstack.org/builds?job_name=novajoin-functional&branch=maste… * name: bindep-fedora-latest where: https://opendev.org/opendev/bindep/src/branch/master/.zuul.yaml#L24 recent builds: https://zuul.openstack.org/builds?job_name=bindep-fedora-latest&branch=mast… Looking at these jobs they're all non-voting or disabled apart from: * validations-libs-podified-podman, which is part of which is part of tripleo and as such the future is hazy * bindep-fedora-latest, This job would seem to be a loss, however from a funcational test POV Fedora and Centos Stream are compatible [1] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… [2] https://review.opendev.org/c/zuul/zuul-jobs/+/884935?tab=change-view-tab-he… [3] This may be in part due to the scale/number of those jobs [4] https://docs.fedoraproject.org/en-US/releases/eol/ Yours Tony.

2 1

[cinder] Active-Active cluster support for allocated_capacity_gb
by Ilya Popov 08 Jun '23

08 Jun '23

Hello, All ! There is one issue with Active/Actvie cinder cluster. Each cinder-volume instance in cluster holds it's own local allocated_capacity_gb. When cinder-volume service starts it counts all volumes created on particular pool and calculates allocated_capacity_gb accordingly. If one instance of cinder-volume create volume - it increase its local allocated_capacity_gb. If other instance delete volume - it decrease its local allocated_capacity_gb. And each instance of cinder-volume in cluster reports its own local allocated_capacity_gb. It cause incorrect allocated_capacity_gb presented to cinder get-pools --detail. And it is able to see even negative value of allocated_capacity_gb for pool. There is reported issue about it: https://bugs.launchpad.net/cinder/+bug/1927186 There is document about different approaches of Active-Active cinder support, but there is nothing about incorrect allocated_capacity_gb report on it: https://docs.openstack.org/cinder/latest/contributor/high_availability.html Looks like without fixing allocated_capacity_gb we have to get all valumes and calculate this value by portal (it isnt quite effective and resource consuming by the way). Are there any plans to complete Active/Active support for cinder ? For example as the coordinator is demand for Active/Active support (“My recommendation is to do this right: configure the cluster option, remove the backend_host, and configure the coordinator.”, https://lists.openstack.org/pipermail/openstack-discuss/2020-November/01885…) - it is possible to move allocated_capacity_gb for each pool to redis or to listen notification queue and change local allocated_capacity_gb for each instance of cinder-volume accordingly Bug #1927186 “the value of allocated_capacity_gb is Incorrect wh...” : Bugs : Cinder<https://bugs.launchpad.net/cinder/+bug/1927186> the value of allocated_capacity_gb is incorrect when the number of replaca of cinder-volume is more than one and all of them configurated same backend, We set up more than one cinder-volume service ,and all of them have the same config file with the same storage backend. When all cinder-volume services complete initialization, the value of 'allocated_capacity_gb' that command of 'cinder get pools -detail' returned is correct.But once any creating or deleting volumes has been done, the value... bugs.launchpad.net 

1 0

[kolla-ansible][Yoga] Manila share creation stuck in creating status cephfs backend
by wodel youchi 08 Jun '23

08 Jun '23

Hi, We are facing a strange problem, when creating manila shares, we use cephfs as backend, the creation is stuck in creation state. If we try to delete the share, we get and error "you are not authorized to delete share ...." We have to use force-delete in cli to be able to delete it. We don't have any indication in manila's log files, there no error or exception. It seems like an access right problem, but we don't know where to look. Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 1

[security-sig] PTG schedule
by Jeremy Stanley 08 Jun '23

08 Jun '23

I've booked a PTG table for the two "slots" immediately prior to Thursday's lunch break (11:00-12:50 Vancouver time). See https://ptg.opendev.org/etherpads.html for a link to the notepad for it; I've added some ideas for things we could do or discuss but add anything you'd like to cover and I'm happy to pivot based on attendee interest. Given the timing, we can also turn it into an early lunch outing or whatever. -- Jeremy Stanley

1 0

[manila] Cancelling June 8th and June 15th weekly meetings
by Carlos Silva 07 Jun '23

07 Jun '23

Hello, Since June 8th is a holiday in some countries and that will impact the weekly meeting's attendance numbers, we are cancelling this meeting. The weekly meeting of June 15th is also being cancelled, as some members will be attending the OpenInfra Summit. The next upstream weekly meeting will be on June 22nd. Regards, carloss

1 0

[manila][ptg] Vancouver PTG
by Carlos Silva 07 Jun '23

07 Jun '23

Hello, zorillas and interested stackers. In the previous weekly meeting I mentioned the tentative time slot for next week's PTG and I got confirmation. We will host a hybrid session for our PTG on Thursday 9:40AM to 12:10 PM PDT. More details about the location, topics and connecting to the meeting bridge will be available in the PTG etherpad [1]. You can still add your topics until Friday (June 9th) to the etherpad. [1] https://etherpad.opendev.org/p/manila-openinfra-2023 Regards, carloss

1 0

[all][ops][rbac] RBAC feedback sessions in Vancouver Summit
by Ghanshyam Mann 07 Jun '23

07 Jun '23

Hello Everyone, If you are planning to attend Vancouver Summit, we will have a forum discussion on Tuesday to provide the RBAC updates and collect operators' feedback. "OpenStack RBAC: updates & operators feedback" - Tue, June 13, 1:45pm - 2:15pm | Vancouver Convention Centre - Room 9 I have created the etherpad to collect the feedback, feel free to write your feedback if you cannot attend this session. - https://etherpad.opendev.org/p/rbac-operator-feedback-vancouver2023 -gmann

1 0

[cinder] Vancouver PTG scheduling
by Brian Rosmaita 07 Jun '23

07 Jun '23

Hello Vancouver-bound Argonauts, In trying to put together a doodle pool, I found it was way easier to just use an etherpad. So please navigate on over to it and block out your unavailability before the Cinder Spec Review meeting at 1400 UTC on Friday 9 June. https://etherpad.opendev.org/p/vancouver2023-cinder-ptg-scheduling thanks, brian

1 0

Question About BGP Dynamic Routing, Floating IP, and SNAT
by Alex Z 07 Jun '23

07 Jun '23

Hi Everyone, Hope you are all doing well. I’m a beginner to Openstack and Neutron and now run into an issue about SNAT and shared floating IP. I’ve already deployed a neutron network which uses BGP to announce floating IP to PE (Provider Edge router), and everything works as expected when I assign the public floating IP (e.g., 123.0.0.10/24) to VMs. But when I tried to use floating IP port-forwarding function with floating IP 123.0.0.20/24 and rule (internal_ip 10.10.10.10, internal_port 5555, external_port 64000), and assign a private IP (10.10.10.10/24) to a VM. The floating IP 123.0.0.20 won’t be advertised through BGP. May I have some suggestions about how I could get this fixed, or the neutron just won’t work this way? FYI, 1. Per my understanding, the port_forwardings rule will make the port act like a SNAT role and forward any packets that reach it with destination 123.0.0.20:64000 to the private IP 10.10.10.10/24. 2. The IP address could be reached in the neutron network. 3. PE IP address, CE IP address, and floating IP gateway are using the same subnet A and subnet pool (192.168.123.0/24), while floating IP belongs to subnet B and subnet pool (123.0.0.0/24), both subnets belong to the provider network. 4. Only floating IP that assigned to the specific VM will be advertised to PE through BGP 5. Floating IP that is assigned to the port of a router in the neutron network won’t be advertised, even if the IP is activated and is reachable internally. Sincerely, Alex

1 0