- openstack-discuss - lists.openstack.org

[tacker] Skip weekly IRC meeting
by Yasufumi Ogawa 11 Jun '23

11 Jun '23

Hi team, Since we'll have PTG this week, I would like to skip the next IRC meeting on Jun 13. Thanks, Yasufumi

1 0

[magnum] kubectl loses access after 31 days
by Vivian Rook 09 Jun '23

09 Jun '23

After 31 days I lose kubectl access to magnum clusters. This has happened consistently for any cluster that I have deployed. The clusters run just fine, though around 31 days of operation kubectl cannot connect, and the web service shows the service as down (Though the web service on the cluster is responding enough to say that nothing is working, so the cluster has not completely crashed) All kubectl commands have a long pause (about 10 minutes) then gives errors like: Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get deployments.apps) Unable to connect to the server: stream error: stream ID 11; INTERNAL_ERROR; received from peer I have a little more information in https://phabricator.wikimedia.org/T336586 It feels like a cert is expiring as it always seems to happen right about 31 days after deployment. Does magnum have some kind of certificate like that? I checked the kubectl certs, they were set to be fine for years, so I don't think it is them unless I didn't check them correctly (Let's not discount that possibility, I totally could have read the wrong bit of the cert). I can still generate a new kubectl config file with openstack coe cluster config <cluster> Though the resulting configuration will have the same issue as the original config (long pause, then timeout errors). I have also tried to run: openstack coe ca rotate <cluster> Which is accepted and seems to run fine, but after that point if I regenerate a kubeconfig file as above I get new errors when running kubectl: Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "<cluster>") If the key rotation would work, and I'm not doing it correctly, I would be delighted to hear how to run it correctly. Though ideally I would like to find where the original key is failing, and if it is an expiration, how to set it to a longer time. Thank you! -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 1

[release] Release countdown for week R-16, June 12-16
by Thierry Carrez 09 Jun '23

09 Jun '23

Development Focus ----------------- The bobcat-2 milestone will happen in next month, on July 6, 2023. 2023.2 Bobcat-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/bobcat/schedule.html for details. General Information ------------------- Please remember that libraries need to be released at least once per milestone period. At milestone 2, the release team will propose releases for any library that has not been otherwise released since milestone 1. Other non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. At milestone-2 we also freeze the contents of the final release. If you have a new deliverable that should be included in the final release, you should make sure it has a deliverable file in: https://opendev.org/openstack/releases/src/branch/master/deliverables/bobcat You should request a beta release (or intermediary release) for those new deliverables by milestone-2. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. See the MembershipFreeze description for more details: https://releases.openstack.org/bobcat/schedule.html#b-mf Finally, now may be a good time for teams to check on any stable releases that need to be done for your deliverables. If you have bugfixes that have been backported, but no stable release getting those. If you are unsure what is out there committed but not released, in the openstack/releases repo, running the command "tools/list_stable_unreleased_changes.sh <cycle_name>" gives a nice report. Upcoming Deadlines & Dates -------------------------- OpenInfra Summit in Vancouver: From June 13 to June 15, 2023 Bobcat-2 Milestone: July 6, 2023 -- Thierry Carrez (ttx)

1 0

[ironic][ops] Forum Session at OIS Vancouver 2023
by Iury Gregory 09 Jun '23

09 Jun '23

Hello everyone, I would like to invite everyone interested in ironic to join our Forum Session next week during the Open Infrastructure Summit in Vancouver! The forum session *Ironic: Our Future starts Now! *will be held on *Wed, June 14, 10:20am - 10:50am | Vancouver Convention Centre - Room 9* This is a chance to provide feedback to our work, see you all next week in Vancouver! -- *Att[]'s* *Iury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Ironic PTL * *Senior Software Engineer at Red Hat Brazil* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 0

[glance][ops][ptg] glance operator event at the summit/PTG
by Brian Rosmaita 09 Jun '23

09 Jun '23

Hello operators, Come by and meet members of the Glance project team and give us feedback, unusual use cases, feature requests, etc. Wednesday 14 June 15:10-15:40 (UTC-7) Table #13 in the PTG area etherpad: https://etherpad.opendev.org/p/vancouver-june2023-glance

1 0

[cinder][ops] cinder operator events at the summit/PTG
by Brian Rosmaita 09 Jun '23

09 Jun '23

Hello operators, Just want to alert you about some opportunities to give feedback to the Cinder project team while you're at the summit or PTG: Wednesday 14 June, 11:40-12:10 Forum session: Cinder, the OpenStack Block Storage service ... how are we doing? (Looking for feedback from operators, vendors, and end-users) Vancouver Convention Centre - Room 9 Wednesday 14 June, 15:50-16:20 PTG session: Operator Half-Hour (Focus on operator feedback) Table #13 in the PTG area Thursday 15 June, 16:40-17:10 PTG session: Open Discussion with the Cinder project team (Focus on developers, but willing to discuss anything, really) Table #13 in the PTG area

1 0

[cinder][ptg] cinder PTG schedule
by Brian Rosmaita 09 Jun '23

09 Jun '23

Hello Argonauts, We're going to have a small presence at the PTG next week, so we've only scheduled 3 sessions: Wednesday 14 June 10:20-10:50 Support for NVMe-OF in os-brick 15:50-16:20 Cinder Operator Half-Hour Thursday 15 June 16:40-17:10 Open Discussion with the Cinder project team All times are Vancouver time (UTC-7) and all will take place at Table #13 in the PTG area. Here's the Cinder PTG etherpad: https://etherpad.opendev.org/p/vancouver-ptg-june-2023 cheers, brian

1 0

[cinder][all] EOL EM branches
by Rajat Dhasmana 09 Jun '23

09 Jun '23

Hi, We had a discussion in the last cinder meeting regarding making EM branches EOL for the cinder project[1]. The discussion started because of the CVE fixes where we backported to active stable branches i.e. Yoga, Zed and 2023.1 but there were no backports to further EM stable branches like Xena, Wallaby ... all the way to Train. Cinder team doesn't see much merit in keeping these EM stable branches alive since there is rarely any activity that requires collaboration and if there is, it is usually by the core cinder team. Following are some of our reasons to EOL these branches: 1) We have less review bandwidth even for active stable branches (Yoga, Zed and 2023.1) 2) No one, apart from the project team, does backport of critical fixes implying that those branches aren't used much for collaboration 3) Will save gate resources for some periodic jobs and the patches proposed 4) Save project team's time to fix gate issues We, as the cinder team, have decided to EOL all the existing EM branches that go from Train to Xena. It was agreed upon by the cinder team and no objections were raised during the upstream cinder meeting. We would like to gather more feedback outside of the cinder team regarding if this affects other projects, deployers, operators, vendors etc. Please reply to this email with your concerns, if you have any, so we can discuss again and reconsider our decision. Else the week after the summit we will be moving forward with our current decision. If you will be at the Vancouver Summit, you can also give us feedback at the cinder Forum session at 11:40 am on Wednesday June 14. [1] https://meetings.opendev.org/irclogs/%23openstack-meeting-alt/%23openstack-… Thanks Rajat Dhasmana

8 14

[kolla] Image building process getting stuck
by Satish Patel 09 Jun '23

09 Jun '23

Folks, I am trying to build images using 2023.1 kolla tag but every time it's getting stuck here and there and I have to ctrl+c and re-run the build. It's very frustrating. How do CI jobs build images without getting stuck? Because of that I have to build images role-by-role instead of all images in a single shot. Example, This is stuck here since 24 hours and not moving at all : https://paste.opendev.org/show/bf1rRhg9V3BCy4KmHPRg/

1 0

[cinder] Spec Review Day 09 June
by Rajat Dhasmana 09 Jun '23

09 Jun '23

Hello Argonauts, Inspired from the nova team, we will be having our day of spec review. As decided in today's cinder meeting, we will be having spec review day on 09th June i.e. this Friday. Following are the details: Date: 09 June, 2023 Time: 1400-1500 UTC Connection Information: Meeting link will be shared in the cinder channel before the event Thanks Rajat Dhasmana

1 1

[gate][tripleo] gate blocker
by Ananya Banerjee 09 Jun '23

09 Jun '23

Hello, Jobs on Centos 9 deploying standalone or multinode are failing standalone deploy or overcloud deploy with https://bugs.launchpad.net/tripleo/+bug/2023019 . Fixes are in progress, please hold rechecks for the moment if you hit this. Thanks, Ananya -- Ananya Banerjee, RHCSA, RHCE-OSP Software Engineer Red Hat EMEA <https://www.redhat.com/> anbanerj(a)redhat.com M: +491784949931 IM: frenzy_friday @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

2 1

Octavia does not work properly.
by GOPAL RAJORA 09 Jun '23

09 Jun '23

Hi everyone. Hope you are well. This is Gopal. I am contacting you as I need your help. So, recently I installed the Octavia service on the Openstack server. After installation of the Octavia, several services such as Nova, Neutron and Keystone started not to work and I can not access the horizon page either. So, I have checked the logs, for the moment, keystone service does not work after the Octavia installation. *Unable to establish connection to http://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens <http://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens>: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))* So, I tried to install the Octavia service several times, so if I disable the health manager, it seems everything works properly, but as you know, Octavia can not work properly if the heath manger is not installed. So, can you please help me regarding this problem? Thanks.

3 6

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 09 Jun '23

09 Jun '23

Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Please remember that all meetings will be cancelled next week too because of the Vancouver PTG. Have a nice weekend! [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

Strange Cinder rpc timeouts after Xena upgrade
by Colby Walsworth 09 Jun '23

09 Jun '23

Hey Everyone, We just upgraded from Wallaby to Xena. We use Ceph as our volume/image backend. All instances from before the upgrade seem to be unable to be shelved/unshelved or live migrated (have not tried offline migration yet). Any new instances that are created work fine and all those tasks work fine on those. The problem seems to be with cinder: cinder.api.v3.attachments [req-2af34fab-a4b5-43f3-bd9a-94f360201533 e28435e0a66740968c523e6376c57f68 179f701c810c425ab004548cc1f76bc9 - default default] Unable to create attachment for volume.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 8eea6a00568a477c89ced8db17f1179d the migrations and shelve/unshelve processes start on the hypervisor but eventually error out with the above message. Any new instances I create do not have this issue at all. I have done all the db upgrades so I don't believe this is a db record update issue. Any ideas of what might be causing this? Thanks, Colby

2 1

[ironic][barbican][qa][tripleo][nova][opendev] Next steps for Fedora jobs in OpenStack
by Tony Breeds 08 Jun '23

08 Jun '23

Hello all, Following on from "Future of Fedora and CentOS Stream Test Images"[1] discussion on service-discuss the opendev infra team have removed the unused Fedora content from the AFS mirrors. Now it's time to look at removing the Fedora-36 content from the mirrors. I'd like to, again, point out the Fedora 36 isn't the latest release and is infact EOL[4]. Removal of the Fedora 36 content will start with something like "[configure-mirrors] Allow per distribution disabling of mirrors"[2]. Once merged, and activated, this means that any/all Fedora based jobs will pull OS content from the Fedora project mirrors directly. There isn't an ETA but "soon" is a fair estimate. Any Fedora jobs will potentially run slower and will be more subject to network failures, this is also the situation with existing rocky linux jobs and there hasn't been any major problems[3]. Projects using Fedora are encouraged to switch to CentOS Stream or Rocky Linux. As at some stage the Fedora nodepool images may also need to go away. OpenStack users of Fedora (master) appear to be: * name: bifrost-integration-tinyipa-fedora-latest where: https://opendev.org/openstack/bifrost/src/branch/master/zuul.d/bifrost-jobs… recent builds: https://zuul.openstack.org/builds?job_name=bifrost-integration-tinyipa-fedo… * name: bifrost-integration-redfish-uefi-fedora-latest where: https://opendev.org/openstack/bifrost/src/branch/master/zuul.d/bifrost-jobs… recent builds: https://zuul.openstack.org/builds?job_name=bifrost-integration-redfish-uefi… * name: barbican-dogtag-tox-functional where: https://opendev.org/openstack/barbican/src/branch/master/.zuul.yaml#L16 recent builds: https://zuul.openstack.org/builds?job_name=barbican-dogtag-tox-functional&b… * name: devstack-plugin-ceph-tempest-fedora-latest where: https://opendev.org/openstack/devstack-plugin-ceph/src/branch/master/.zuul.… recent builds: https://zuul.openstack.org/builds?job_name=devstack-plugin-ceph-tempest-fed… * name: devstack-platform-fedora-latest where: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L861 recent builds: https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&… * name: devstack-platform-fedora-latest-virt-preview where: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L868 recent builds: https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest-… Notes: This COPR has CentOS Stream packages * name: validations-libs-podified-podman where: https://opendev.org/openstack/validations-libs/src/branch/master/.zuul.yaml… recent builds: https://zuul.openstack.org/builds?job_name=validations-libs-podified-podman… * name: novajoin-functional where: https://opendev.org/x/novajoin/src/branch/master/.zuul.yaml#L13 recent builds: https://zuul.openstack.org/builds?job_name=novajoin-functional&branch=maste… * name: bindep-fedora-latest where: https://opendev.org/opendev/bindep/src/branch/master/.zuul.yaml#L24 recent builds: https://zuul.openstack.org/builds?job_name=bindep-fedora-latest&branch=mast… Looking at these jobs they're all non-voting or disabled apart from: * validations-libs-podified-podman, which is part of which is part of tripleo and as such the future is hazy * bindep-fedora-latest, This job would seem to be a loss, however from a funcational test POV Fedora and Centos Stream are compatible [1] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… [2] https://review.opendev.org/c/zuul/zuul-jobs/+/884935?tab=change-view-tab-he… [3] This may be in part due to the scale/number of those jobs [4] https://docs.fedoraproject.org/en-US/releases/eol/ Yours Tony.

2 1

[cinder] Active-Active cluster support for allocated_capacity_gb
by Ilya Popov 08 Jun '23

08 Jun '23

Hello, All ! There is one issue with Active/Actvie cinder cluster. Each cinder-volume instance in cluster holds it's own local allocated_capacity_gb. When cinder-volume service starts it counts all volumes created on particular pool and calculates allocated_capacity_gb accordingly. If one instance of cinder-volume create volume - it increase its local allocated_capacity_gb. If other instance delete volume - it decrease its local allocated_capacity_gb. And each instance of cinder-volume in cluster reports its own local allocated_capacity_gb. It cause incorrect allocated_capacity_gb presented to cinder get-pools --detail. And it is able to see even negative value of allocated_capacity_gb for pool. There is reported issue about it: https://bugs.launchpad.net/cinder/+bug/1927186 There is document about different approaches of Active-Active cinder support, but there is nothing about incorrect allocated_capacity_gb report on it: https://docs.openstack.org/cinder/latest/contributor/high_availability.html Looks like without fixing allocated_capacity_gb we have to get all valumes and calculate this value by portal (it isnt quite effective and resource consuming by the way). Are there any plans to complete Active/Active support for cinder ? For example as the coordinator is demand for Active/Active support (“My recommendation is to do this right: configure the cluster option, remove the backend_host, and configure the coordinator.”, https://lists.openstack.org/pipermail/openstack-discuss/2020-November/01885…) - it is possible to move allocated_capacity_gb for each pool to redis or to listen notification queue and change local allocated_capacity_gb for each instance of cinder-volume accordingly Bug #1927186 “the value of allocated_capacity_gb is Incorrect wh...” : Bugs : Cinder<https://bugs.launchpad.net/cinder/+bug/1927186> the value of allocated_capacity_gb is incorrect when the number of replaca of cinder-volume is more than one and all of them configurated same backend, We set up more than one cinder-volume service ,and all of them have the same config file with the same storage backend. When all cinder-volume services complete initialization, the value of 'allocated_capacity_gb' that command of 'cinder get pools -detail' returned is correct.But once any creating or deleting volumes has been done, the value... bugs.launchpad.net 

1 0

[kolla-ansible][Yoga] Manila share creation stuck in creating status cephfs backend
by wodel youchi 08 Jun '23

08 Jun '23

Hi, We are facing a strange problem, when creating manila shares, we use cephfs as backend, the creation is stuck in creation state. If we try to delete the share, we get and error "you are not authorized to delete share ...." We have to use force-delete in cli to be able to delete it. We don't have any indication in manila's log files, there no error or exception. It seems like an access right problem, but we don't know where to look. Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 1

[security-sig] PTG schedule
by Jeremy Stanley 07 Jun '23

07 Jun '23

I've booked a PTG table for the two "slots" immediately prior to Thursday's lunch break (11:00-12:50 Vancouver time). See https://ptg.opendev.org/etherpads.html for a link to the notepad for it; I've added some ideas for things we could do or discuss but add anything you'd like to cover and I'm happy to pivot based on attendee interest. Given the timing, we can also turn it into an early lunch outing or whatever. -- Jeremy Stanley

1 0

[manila] Cancelling June 8th and June 15th weekly meetings
by Carlos Silva 07 Jun '23

07 Jun '23

Hello, Since June 8th is a holiday in some countries and that will impact the weekly meeting's attendance numbers, we are cancelling this meeting. The weekly meeting of June 15th is also being cancelled, as some members will be attending the OpenInfra Summit. The next upstream weekly meeting will be on June 22nd. Regards, carloss

1 0

[manila][ptg] Vancouver PTG
by Carlos Silva 07 Jun '23

07 Jun '23

Hello, zorillas and interested stackers. In the previous weekly meeting I mentioned the tentative time slot for next week's PTG and I got confirmation. We will host a hybrid session for our PTG on Thursday 9:40AM to 12:10 PM PDT. More details about the location, topics and connecting to the meeting bridge will be available in the PTG etherpad [1]. You can still add your topics until Friday (June 9th) to the etherpad. [1] https://etherpad.opendev.org/p/manila-openinfra-2023 Regards, carloss

1 0

[all][ops][rbac] RBAC feedback sessions in Vancouver Summit
by Ghanshyam Mann 07 Jun '23

07 Jun '23

Hello Everyone, If you are planning to attend Vancouver Summit, we will have a forum discussion on Tuesday to provide the RBAC updates and collect operators' feedback. "OpenStack RBAC: updates & operators feedback" - Tue, June 13, 1:45pm - 2:15pm | Vancouver Convention Centre - Room 9 I have created the etherpad to collect the feedback, feel free to write your feedback if you cannot attend this session. - https://etherpad.opendev.org/p/rbac-operator-feedback-vancouver2023 -gmann

1 0

[cinder] Vancouver PTG scheduling
by Brian Rosmaita 07 Jun '23

07 Jun '23

Hello Vancouver-bound Argonauts, In trying to put together a doodle pool, I found it was way easier to just use an etherpad. So please navigate on over to it and block out your unavailability before the Cinder Spec Review meeting at 1400 UTC on Friday 9 June. https://etherpad.opendev.org/p/vancouver2023-cinder-ptg-scheduling thanks, brian

1 0

Question About BGP Dynamic Routing, Floating IP, and SNAT
by Alex Z 07 Jun '23

07 Jun '23

Hi Everyone, Hope you are all doing well. I’m a beginner to Openstack and Neutron and now run into an issue about SNAT and shared floating IP. I’ve already deployed a neutron network which uses BGP to announce floating IP to PE (Provider Edge router), and everything works as expected when I assign the public floating IP (e.g., 123.0.0.10/24) to VMs. But when I tried to use floating IP port-forwarding function with floating IP 123.0.0.20/24 and rule (internal_ip 10.10.10.10, internal_port 5555, external_port 64000), and assign a private IP (10.10.10.10/24) to a VM. The floating IP 123.0.0.20 won’t be advertised through BGP. May I have some suggestions about how I could get this fixed, or the neutron just won’t work this way? FYI, 1. Per my understanding, the port_forwardings rule will make the port act like a SNAT role and forward any packets that reach it with destination 123.0.0.20:64000 to the private IP 10.10.10.10/24. 2. The IP address could be reached in the neutron network. 3. PE IP address, CE IP address, and floating IP gateway are using the same subnet A and subnet pool (192.168.123.0/24), while floating IP belongs to subnet B and subnet pool (123.0.0.0/24), both subnets belong to the provider network. 4. Only floating IP that assigned to the specific VM will be advertised to PE through BGP 5. Floating IP that is assigned to the port of a router in the neutron network won’t be advertised, even if the IP is activated and is reachable internally. Sincerely, Alex

1 0

[i18n] upstream investment: translations infrastructure engineer
by Brian Rosmaita 07 Jun '23

07 Jun '23

I've proposed an OpenStack "upstream investment opportunity" for an engineer to handle the Zanata -> Weblate re-plumbing for gerrit/zuul: https://review.opendev.org/c/openstack/governance/+/885379 Please leave comments if you have any. cheers, brian

1 0

Cinder Bug Report 2023-06-07
by Sofia Enriquez 07 Jun '23

07 Jun '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> *Medium* - Volume creation fails if an old signature is found when using the LVM driver. <https://bugs.launchpad.net/cinder/+bug/2023073> - *Status*: Unassigned. - LIO target doesn't support discard on thin volumes. <https://bugs.launchpad.net/cinder/+bug/2023168> - *Status*: Fix proposed to master. <https://review.opendev.org/c/openstack/cinder/+/885451> *Low* - Cinder PowerMax driver volume attach/detach hang issues. <https://bugs.launchpad.net/cinder/+bug/2021999> - *Status*: Fix proposed to master <https://review.opendev.org/c/openstack/cinder/+/884957>. - Huawei Oceanstor volume driver is not up to date with official Huawei's driver <https://bugs.launchpad.net/cinder/+bug/2022152>. - *Status*: Please propose the topic to the cinder meeting <https://wiki.openstack.org/wiki/CinderMeetings>. *Incomplete* - [huawei] wrong hostname passed to os_brick. <https://bugs.launchpad.net/cinder/+bug/2017575> - *Status*: Waiting on <hillpd> keep the conversation on the Launchpad bug or propose a WIP patch to discuss the changes on the patch. -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0