February 2022 - openstack-discuss

[stable] sphinx docs jobs fail on old stable branches (no more py2 support)
by Előd Illés 08 Feb '22

08 Feb '22

Hi stable maintainers, This mail is about to get some attention to build-openstack-sphinx-docs job that recently started to fail on old stable branches (Queens and Pike) and block gates for about ~30 repositories. (The failures are also visible in periodic stable jobs). This is because py27 support from sphinx jobs were removed [1] recently. (Python 3 (py35) support was mostly done in Rocky so that branch is not affected) To unblock the gates I see the following options: 1. fix sphinx jobs to work with py35 (most probably requires some patches to backport - in a squashed patch - and maybe some resolution of requirement conflicts) 2. set sphinx jobs to non-voting (even better to remove them to do not use resources unnecessarily) 3. revert the patch [1] that removed the py2 support of sphinx (maybe the easiest thing to do, though i understand that py2 support is something that shouldn't be kept forever) 4. teams examine their old branches and if they are not really maintained, haven't merged patches recently, then probably it is a good time to propose their EOL transition according to the EOL process [1] Some additional thoughts: - When Extended Maintenance was introduced the community agreed that certain CI jobs can be removed, CI coverage can be reduced as time goes. - To keep open an old branch in Extended Maintenance helps the cooperation between vendors / users as instead of maintaining it by themselves it can be done with a common effort, which could be beneficial for everyone. On the other hand, if there are no maintainers or invested efforts to keep the branch and CI alive, then it is a reasonable decision to transition the branch to End of Life. Please consider the above options and act to eliminate the gate failures / periodic stable job failures. [1] https://review.opendev.org/c/zuul/zuul-jobs/+/827588 <https://review.opendev.org/#/q/I98c3da5b02a4ac7fb9d7bd8e00170762e77b9f40>[2] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l… Thanks, Előd

2 1

[election][ptl][cinder] PTL non-candidacy
by Brian Rosmaita 08 Feb '22

08 Feb '22

Hello Argonauts, With the election season rapidly approaching [0], I want to communicate my intentions to the team, to wit, I will not be standing for PTL for the Z cycle. I've served as PTL for five cycles, which conflicts with my belief that it's good for open source projects to rotate leadership. I think the Cinder project is fairly healthy, but it will be good to have someone take the Cinder helm who has a fresh perspective and who will be as excited to be PTL as I was way back in Ussuri. We have an excellent project team, and I have no doubt that whoever steps up will have plenty of support from the Cinder community (including me, I don't have any intention of going anywhere in the near term). cheers, brian [0] http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027070…

7 6

Floating IP issue regarding Latest upgrade to XENA
by Adivya Singh 08 Feb '22

08 Feb '22

Hi Team, We have the latest upgrade to XENA, and we are facing an issue with XENA release , where a few times, even Floating IP attached to the instance are not reachable. I tried various ways to troubleshoot the issue, Regarding The IP is reachable from the router Namespace, The router IP is reachable, L3 agent is working Fine, as other Floating IP are reachable which are hosted through router on L3 agent, All ports are up, Interfaces are up, SNAT and DNAT are working Fine, But as soon as i dissociate and associate the Floating IP , it started working Fine Any Idea, why this Happening , and later on Fixed when Floating IP is dissociate and associate again, and later again starts happening Regards Adivya Singh

1 0

[ironic] Each deploy overwrites the entire target disk
by Radosław Piliszek 08 Feb '22

08 Feb '22

Dears, I have a question regarding the default behaviour of Ironic Python Agent - it seems that each deploy causes the entire target disk to be overwritten. I have noticed this after replacing disks with larger ones, the deployment process started taking considerably longer and I observed that each sector is rewritten on the target device. Is that expected? Can I somehow avoid it? I am using whole-disk, qcow2 images that are much smaller than the disk size (even in raw form). I am running Ironic Wallaby. The deployment method is direct. Kind regards, -yoctozepto

2 4

[openstack-helm] No Meeting Tomorrow
by Gage Hugo 07 Feb '22

07 Feb '22

Hey team, Since there's nothing on the agenda for tomorrow's meeting, it has been cancelled. We will meet again next week.

1 0

[nova][ops] Problem with nova policies for resume operation
by Massimo Sgaravatto 07 Feb '22

07 Feb '22

Dear all I am running a Xena installation I have modified the nova policy fail so that certain operations can be done only by the user who created the instance, or by the administrator This [*] is my policy.yaml file. While the suspend operation works as intended (I can suspend only my instances and I am not allowed to suspend an instance created by another user) I am not able to resume an instance that I own and that I have previously suspended. I get this error: ERROR (Forbidden): Policy doesn't allow os_compute_api:os-suspend-server:suspend to be performed. (HTTP 403) (Request-ID: req-c57458bc-b1ea-4b40-a1d2-0f67608ef673) Only removing the line: "os_compute_api:os-suspend-server:suspend": "rule:admin_api or user_id:%(user_id)s" from the policy file, I am able to resume the instance. I am not able to understand what is wrong with that policy. Any hints ? Thanks, Massimo [*] # Pause a server # POST /servers/{server_id}/action (pause) # Intended scope(s): system, project "os_compute_api:os-pause-server:pause": "rule:admin_api or user_id:%(user_id)s" # Delete a server # DELETE /servers/{server_id} # Intended scope(s): system, project "os_compute_api:servers:delete": "rule:admin_api or user_id:%(user_id)s" # Resize a server # POST /servers/{server_id}/action (resize) # Intended scope(s): system, project "os_compute_api:servers:resize": "rule:admin_api or user_id:%(user_id)s" # Rebuild a server # POST /servers/{server_id}/action (rebuild) # Intended scope(s): system, project "os_compute_api:servers:rebuild": "rule:admin_api or user_id:%(user_id)s" # Stop a server # POST /servers/{server_id}/action (os-stop) # Intended scope(s): system, project "os_compute_api:servers:stop": "rule:admin_api or user_id:%(user_id)s" # Resume suspended server # POST /servers/{server_id}/action (resume) # Intended scope(s): system, project "os_compute_api:os-suspend-server:resume": "rule:admin_api or user_id:%(user_id)s" # Suspend server # POST /servers/{server_id}/action (suspend) # Intended scope(s): system, project "os_compute_api:os-suspend-server:suspend": "rule:admin_api or user_id:%(user_id)s"

4 8

[kolla-ansible][mariadb]error (galera cluster problem ?)
by Franck VEDEL 07 Feb '22

07 Feb '22

Hello, I am in an emergency situation, quite catastrophic situation because I do not know what to do. I have an Openstack cluster with 3 servers (serv1, serv2, serv3). He was doing so well… A network admin came to me and told me to change an MTU on the cards. I knew it shouldn't be done...I shouldn't have done it. I did it. Of course, it didn't work as expected. I went back to my starting configuration and there I have a big problem with mariadb which is set up on serv1 and serv2. Here are my errors: 2022-02-04 17:40:36 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():160 2022-02-04 17:40:36 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -110 (Connection timed out) 2022-02-04 17:40:36 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1475: Failed to open channel 'openstack' at 'gcomm://10.0.5.109:4567,10.0.5.110:4567': -110 (Connection timed out) 2022-02-04 17:40:36 0 [ERROR] WSREP: gcs connect failed: Connection timed out 2022-02-04 17:40:36 0 [ERROR] WSREP: wsrep::connect(gcomm://10.0.5.109:4567,10.0.5.110:4567) failed: 7 2022-02-04 17:40:36 0 [ERROR] Aborting I do not know what to do. My installation is done with kolla-ansible, mariadb docker restarts every 30 seconds. Can the "kolla-ansible reconfigure mariadb" command be a solution? Could the command "kolla-ansible mariadb recovery" be a solution? Thanks in advance if you can help me. Franck

3 6

[ops][cinder] resource count survey
by Brian Rosmaita 07 Feb '22

07 Feb '22

Hello operators, The Cinder project is working on the cinder quotas system in an effort to improve its accuracy and remove the necessity for operators to go through and clean it up occasionally when it gets out of sync with reality. One way to do this is to use dynamic database counting (instead of storing the usage info in a table where it can become outdated). Whether this will be a good change depends on how many records the database has to process during a transaction, which in turn depends on how many Cinder resources a project has. So we need to get some information from you. Please take this brief survey: https://rosmaita.wufoo.com/forms/cinder-resource-count-survey/ The survey closes on Wednesday 30 March at 1200 UTC, so that we can have data to discuss during the "Z" PTG. But please don't procrastinate, it's only 6 questions and the answers are all positive integers. It shouldn't take long to fill out! Thanks! brian

1 0

Re: Recently Upgraded to XENA and getting this error
by Mohammed Naser 07 Feb '22

07 Feb '22

The fix hasn’t landed yet since then. On Fri, Feb 4, 2022 at 8:38 AM Adivya Singh <adivya1.singh(a)gmail.com> wrote: > hi Mohammed, > > This is for pike release, I am talking about Xena > > Regards > Adivya Singh > > On Thu, Feb 3, 2022 at 4:37 AM Mohammed Naser <mnaser(a)vexxhost.com> wrote: > >> I think you're running into this which has been broken for a hot minute: >> >> https://review.opendev.org/c/openstack/horizon/+/808102 >> >> that patch works for us locally. >> >> On Wed, Feb 2, 2022 at 10:37 AM Adivya Singh <adivya1.singh(a)gmail.com> >> wrote: >> > >> > Hello, >> > >> > We have recently upgraded to XENA release of Openstack, and getting >> this error while trying to Resize instance using GUI, it is telling "Danger >> , Please try Later" >> > Same is working properly from CLI without any issue >> > >> > Any thoughts on this >> > >> > Regards >> > Adivya Singh >> >> >> >> -- >> Mohammed Naser >> VEXXHOST, Inc. >> > -- Mohammed Naser VEXXHOST, Inc.

2 1

[baremetal-sig][ironic] Tue Feb 8, 2022, 2pm UTC: Scaling Ironic
by Arne Wiebalck 07 Feb '22

07 Feb '22

Dear all, The Bare Metal SIG will meet tomorrow Tue Feb 8, 2022, at 2pm UTC on zoom. The meeting will feature a "topic-of-the-day" presentation by Julia Kreger on "Scaling Ironic" providing details on the massive performance and scaling improvements Ironic has seen in the latest releases. Come along and bring your questions, everyone is welcome! As usual, all details on https://etherpad.opendev.org/p/bare-metal-sig Cheers, Arne

1 0

[all][elections][ptl][tc] Combined PTL/TC Election Feb-Mar 2022 Season
by Ghanshyam Mann 06 Feb '22

06 Feb '22

Hello Everyone! This is time for the Z development cycle Technical elections. Election details: https://governance.openstack.org/election/ The nomination period officially begins Feb 8, 2022 23:45. Please read the stipulations and timelines for candidates and electorate contained in this governance documentation. The PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. If teams do not produce any PTL candidate during the nomination period, or are interested in considering alternatives prior to nominations, the TC may consider requests to switch to the new distributed leadership model they recently documented[1]. In keeping with the established plan[1 ] to gradually reduce the size of the Technical Committee, we will only fill five (5) seats in this coming election. There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you which to discuss in private please email any of the election officials[3] so that we may address your concerns. [1] https://governance.openstack.org/tc/resolutions/20200803-distributed-projec… [2] https://governance.openstack.org/tc/reference/charter.html#number-of-seats-… [3] https://governance.openstack.org/election/#election-officials -gmann & The OpenStack Technical Election Officials

1 0

[designate] Tentative PTG time slots for Designate
by Michael Johnson 04 Feb '22

04 Feb '22

Hello Designate community, I have tentatively put Designate down for two time slots for the PTG. Tuesday April 5th 15:00 - 17:00 UTC Wednesday April 6th 15:00-17:00 UTC Please let me know if those times do not work for you and you would like to attend. Michael

1 0

[release] Release countdown for week R-7, Feb 07-11
by Előd Illés 04 Feb '22

04 Feb '22

Development Focus ----------------- We are entering the last weeks of the Yogadevelopment cycle. From now until the final release, we'll send a countdown email like this every week. It's probably a good time for teams to take stock of their library and client work that needs to be completed yet. The non-client library freeze is coming up, followed closely by the client lib freeze. Please plan accordingly to avoid any last minute rushes to get key functionality in. General Information ------------------- Next week is the Extra-ATC freeze, in preparation for elections. All contributions to OpenStack are valuable, but some are not expressed as Gerrit code changes. Please list active contributors to your project team who do not have a code contribution this cycle, and therefore won't automatically be considered an Active Technical Contributor and allowed to vote. This is done by adding extra-atcs to https://opendev.org/openstack/governance/src/branch/master/reference/projec… before the Extra-ATC freeze on February 10th, 2022. A quick reminder of the upcoming freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (February 17th, 2022). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (February 24th, 2022) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, February 24th, 2022. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (March 7th, 2022) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (March 24th, 2022) Finally, now is also a good time to start planning what highlights you want for your deliverables in the cycle highlights. The deadline to submit an initial version for those is set to Feature freeze (February 24th, 2022). Background on cycle-highlights: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125613.html Project Team Guide, Cycle-Highlights: https://docs.openstack.org/project-team-guide/release-management.html#cycle… knelson [at] openstack.org/diablo_rojo on IRC is available if you need help selecting or writing your highlights Upcoming Deadlines & Dates -------------------------- Extra-ATC freeze: February 10th, 2022(R-7 week) Non-client library freeze: February 17th, 2022(R-6 week) Client library freeze: February 24th, 2022(R-5 week) Yoga-3 milestone: February 24th, 2022(R-5 week) Yoga final release: March 30th, 2022 Next PTG: April 4 - 8, 2022 (virtual) ElődIllés irc: elodilles

1 0

[oslo] feature freeze - Feb 07 - Feb 11 (R-7)
by Herve Beraud 04 Feb '22

04 Feb '22

Hello Osloers, Just a friendly reminder that oslo's feature freeze will happen next week (R-7 - Feb 07 - Feb 11). It's earlier than everyone else, and if you're wondering why, we have a policy[1] that discusses it. The main thing is that if you have features you want to get into Oslo libraries this cycle, please make sure they merge by Friday. After that we'll need to go through the FFE process and there's no guarantee we can land them. Feel free to ping us on IRC if you need reviews. Thanks for your attention. [1] http://specs.openstack.org/openstack/oslo-specs/specs/policy/feature- freeze.html -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

openstack swift and local path filename
by admin＠gibdev.ru 03 Feb '22

03 Feb '22

I have installed openstack swift 2.25.2 object-server writes and serves local files from disk It receives http requests from a proxy-server, for example: http get: /sdb4/525/AUTH_xxxxxxxxxxxxxxxxxxxxxxxx/panda/2c34a941f581d848e2799e7a0956ea14f3cb27b0e516aef99a438fdaebbd7592 reads a file from disk: /srv/node/sdb4/objects/525/346/835ee70dce07217d8e33147e0864f346/1643910916.71556.data how can I calculate the path by which the file will be read by object-server? thanks

2 1

[PTG] April 2022 Team Signup
by Kendall Nelson 03 Feb '22

03 Feb '22

Hello Everyone! Last week, we announced the next virtual PTG will be held virtually from Monday, April 4th to Friday, April 8th, 2022. We will have the same schedule set up available as last time with three windows of time spread across the day to cover all timezones with breaks in between. To signup your team, you must complete BOTH the survey[1] AND reserve time in the ethercalc[2] by end of day March 11th. We ask that the PTL/SIG Chair/Team lead sign up for time to have their discussions with 3 rules/guidelines: 1. Cross project discussions (like SIGs or support project teams) should be scheduled towards the start of the week so that any discussions that might shape those of other teams happen first. 2. No team should sign up for more than 4 hours per UTC day to help keep participants actively engaged. 3. No team should sign up for more than 16 hours across all time slots to avoid burning out our contributors and to enable participation in multiple teams discussions. Again, you need to fill out BOTH the ethercalc AND the survey to complete your team's sign up. If you have any issues with signing up your team, due to conflict or otherwise, please let me know! While we are trying to empower you to make your own decisions as to when you meet and for how long (after all, you know your needs and teams timezones better than we do), we are here to help! Once your team is signed up, please register[3]! And remind your team to register! Registration is free, but it's important that you sign up to let us know you'll be attending because that's how you'll receive event details, passwords, and other relevant information about the PTG. Continue to visit openinfra.dev/ptg for updates. -Kendall (diablo_rojo) [1] Team Survey: https://openinfrafoundation.formstack.com/forms/april2022_vptg_survey [2] Ethercalc Signup: https://ethercalc.openstack.org/7yxdas7suqnd [3] PTG Registration: https://openinfra-ptg.eventbrite.com/

1 0

[cinder][drivers] third-party CI compliance check
by Brian Rosmaita 03 Feb '22

03 Feb '22

Hello Argonauts, particularly vendor driver maintainers, This is a reminder that the third-party CI compliance check is imminent [0]. You can prevent any unpleasantness by checking now to verify that your CI is compliant. [0] https://releases.openstack.org/yoga/schedule.html#cinder-3rd-party-ci-compl… [1] https://docs.openstack.org/cinder/latest/drivers-all-about.html#driver-comp…

1 0

[cinder] new driver freeze & exceptions
by Brian Rosmaita 03 Feb '22

03 Feb '22

Hello Argonauts, The new driver merge deadline passed at 20:00 UTC today. I'm extending the new driver merge deadline to Friday 28 January at 20:00 UTC for two drivers: 1. Lightbits: It has both a cinder and os-brick patch, and I want the team to have more time to look at the os-brick patch. I think the driver patch is close to ready; the developers have been quick to respond to comments and make revisions. Also, the third-party CI is functioning and responding on patches. cinder: https://review.opendev.org/c/openstack/cinder/+/821602 os-brick: https://review.opendev.org/c/openstack/os-brick/+/821603 2. NEC Storage V Series: the driver patch has one +2 and the third party CI is functioning and responding on patches; I don't see any reason to make this one wait for the Z release. https://review.opendev.org/c/openstack/cinder/+/815614 Cinder core reviewers: please continue to review the above patches. With respect to the other proposed drivers: - Pure Storage NVMe-RoCE: vendor decided to hold it for Z - Dell EMC PowerStore NFS driver: vendor decided to hold it for Z - HPE XP Storage FC & iSCSI: the code is very straightforward, but the CI is not ready yet, so this will have to wait for Z - YADRO Tatlin.UNIFIED driver: initial patch arrived yesterday (and has a -1 from Zuul); the CI appears to be running, but the cinder team needs to move on to reviewing os-brick and Yoga feature patches, so this will have to wait for Z also cheers, brian

3 4

[neutron] Drivers meeting - Friday 4.2.2022 - cancelled
by Lajos Katona 03 Feb '22

03 Feb '22

Hi Neutron Drivers! Due to the lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. Lajos Katona (lajoskatona)

1 0

[nova][gate] test_tagged_attachment failing close to 100% in nova-next
by Artom Lifshitz 03 Feb '22

03 Feb '22

https://bugs.launchpad.net/nova/+bug/1959899 The test times out verifying the tags in the metadata API after attaching a NIC and volume. This is happening almost 100% of the time, but I've seen at least a couple of passing runs as well. We have yet to determine what the exact failure mode is - is the metadata API timing out (unlikely, since every other test passes)? Or are the expected tags just not present (more likely)? I'm trying to figure some things out at [1], but other than that this is very much under investigation with no root cause identified yet. [1] https://review.opendev.org/c/openstack/nova/+/827549

2 1

[nova][tempest] nova-next gate blocker
by Artom Lifshitz 03 Feb '22

03 Feb '22

nova-next (and possibly others) is broken because of https://bugs.launchpad.net/tempest/+bug/1959682 The fix is at https://review.opendev.org/c/openstack/tempest/+/827305, with a Nova DNM patch to exercise it at https://review.opendev.org/c/openstack/nova/+/827306 Please hold your rechecks until this is resolved. Cheers!

2 2

bypass sphinx conf.py from CLI
by Stone, Ronald 03 Feb '22

03 Feb '22

Hi, The StarlingX docs team is implementing the default sphinxcontrib.spelling spelling extension for use from tox via a virtualenv set up for this purpose. We do not want sphinxcontrib.spelling loaded into other tox jobs that won't use it. This means that it cannot be loaded from the standard conf.py. Instead I am bypassing conf.py and setting parameters from the sphinx-build command in the appropriate tox job: sphinx-build -a -E -C --keep-going -d doc/build/doctrees -D extensions=sphinxcontrib.spelling -D spelling_word_list_filename=spelling_wordlist.txt -D rst_prolog="open('./shared/strings.txt', 'r').read()" -t starlingx -t openstack -b spelling doc/source doc/build/spelling {posargs} This works except that we also need to load some strings into the rst prolog for various purposes, and that is failing without warning or errors. I can open the file from a python console. Sphinx-build escapes the single quotes as: open('"'"'./shared/strings.txt'"'"', '"'"'r'"'"').read()' Any suggestions on the correct parameter format appreciated. Ron

3 2

[os-brick][nova][cinder][glance] Question about lock_path
by Radosław Piliszek 03 Feb '22

03 Feb '22

Hello Fellow OpenStackers! I have noticed os-brick started using the file locks (in lock_path) to avoid race conditions. I have a question regarding that lock_path - it seems all the cases I have found use separate lock_paths per service, yet the problematic cases listed in the bug report (and the commit message of [1]) suggest that, in case of co-hosted nova-compute, glance-api, cinder-volume and cinder-backup services (or any combination thereof), the lock_path should be shared. Should the deployment projects adapt to actually fix the race condition issue? [1] https://review.opendev.org/c/openstack/os-brick/+/814139 Kind regards, -yoctozepto

4 7

[stein][neutron] Static rules losed
by Gargano Andrea 03 Feb '22

03 Feb '22

Hi all, we faced an inssue in our openstack, we have some private clouds with static rules, if the controller that own namespace with the static rule goes down, namespace moves to other controller but it loses the rule. I can still see the rules inside the router on openstack but it doesn't work until recreated. Have you seen something like this? Thank you in advance. Andrea Gargano

1 1

[all][tc] Technical Committee next weekly meeting is today (Feb 3rd) at 1500 UTC
by Radosław Piliszek 02 Feb '22

02 Feb '22

Hello OpenStackers, Below is the agenda for today's TC IRC meeting scheduled at 1500 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for today's TC meeting == * Roll call * Follow up on past action items * Gate health check ** Fixing Zuul config error in OpenStack *** https://etherpad.opendev.org/p/zuul-config-error-openstack * Progress check on Yoga tracker ** https://etherpad.opendev.org/p/tc-yoga-tracker * Z Release Cycle Name ** Election results: https://civs1.civs.us/cgi-bin/results.pl?id=E_693bd8bbe63ca52f ** Next step, the foundation has started the trademark checks. * Z cycle Technical Elections ** Proposed Election schedule, need to merge it if no objection. *** https://review.opendev.org/c/openstack/election/+/825017 * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -yoctozepto

1 0