- openstack-discuss - lists.openstack.org

[sdk][osc][ptl][election] OpenStackSDK PTL Nomination
by Monty Taylor 27 Mar '20

27 Mar '20

Hi everybody! I'd like to run for PTL of OpenStackSDK again. We just merged the OpenStackClient and OpenStackSDK teams. While on the one hand this is a nice big new challenge, I think it also presents an excellent opportunity to accelerate our long-standing goal of integrating the two more seamlessly. I'm excited to get some motion on that. In this last cycle we finally got started on the long-awaited python-*client-ectomy from OpenStackClient by removing glanceclient. I'm hoping in the next cycle we can at least remove keystoneclient - but I think we can pick up the pace now. We need to streamline how config and arguments are parsed so that it's 100% consistent between OSC and SDK and Ansible. This is going to mean using openstack.config in OpenStackClient more directly - and I think it's going to mean an end to the double-auth dance. That, in turn, will likely tie in to attempting to unwind the plugin structure from entrypoints and laying in an optimization so that we can skip entrypoints calls in most cases. We've recently been more aggressive with adding new project-specific cores and I'd like to continue that. We need to trust people from project teams to help maintain relevant code in SDK and OSC, and as a result get into the habit of getting support for new things into SDK and OSC as a part of shipping them in the service. That leads to our need to get to the point where both SDK and OpenStackClient can speak to the latest microversion for the projects supporting microversions. This has been mentioned as one of the things people from the projects don't like about OSC vs the project-specific CLIs. The thing is, we will already use the most recent microversion we understand - so we already behave like people want us to. The thing we need to get to is understanding the MV as it's added. The momentum is picking up to get a lot accomplished, and I'd love to keep helping to pushing these rocks uphill. Thanks! Monty

1 0

[openstack-ansible] proposing Erik Berg to openstack-ansible core
by Mohammed Naser 27 Mar '20

27 Mar '20

Hi everyone, I'd like to propose adding Erik Berg to the OSA core team. They've been doing solid contributions with some activity in reviews and I think they'd be an excellent addition to the OSA team If no one opposes, I'll be adding them to the core team in the next few days. Regards, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. https://vexxhost.com

5 5

[tripleo] TripleO CI Summary: Sprint 44
by Rafael Folco 27 Mar '20

27 Mar '20

Greetings, The TripleO CI team has just completed Sprint 44 / Unified Sprint 23 (Mar 05 thru Mar 25). The following is a summary of completed work during this sprint cycle: - Migrated stable branches from rdo-cloud to vexxhost - Upstream TripleO migrated to CentOS-8. - Upstream component pipeline <http://tripleo-cockpit.usersys.redhat.com/d/2tivP9BWz/component-pipeline?or…> done. - RDO content unpinned - First CentOS-8 promotion on March 16th - Promotion server is online and ready for CentOS-8 - Completed development of unified logging with tripleo-quickstart and infrared, roll out in progress Ruck/Rover Notes: - https://hackmd.io/7MBqFHurTA2e5H8kYRwgag The planned work for the next sprint [1] extends the work started in the previous sprint and focuses on the following: - TripleO-Operator-Ansible: contribute w/ code, docs, tests and zuul jobs. - Internal component pipeline: starting work - Unified tempest test skip list The Ruck and Rover for this sprint are Wesley Hayutin (weshay) and Sandeep Yadav (ysandeep). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes to be tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-24 [2] https://hackmd.io/XMd_vskSTdGZl7ruZ1UckA

1 0

[tc][elections] Nomination
by Donny Davis 27 Mar '20

27 Mar '20

Hello Openstack Humanoids, My name is Donny Davis and I would like to announce my candidacy for TC. I built my first Openstack cloud by hand on the Grizzly release to provide a training environment for fellow employees who lacked Satellite equipment to train on. I have spent subsequent years of my life building, maintaining, operating, and teaching other people how to Openstack. I contribute Openstack CI resources via Fort Nebula (last cycle), and this cycle via Open Edge. This is a cloud that has been tuned specifically for the Openstack Community CI workload. I support many of the obscure requirements to run non-standard CI jobs on this cloud. This Openstack deployment is all my personal equipment and is 100% privately funded. In the last cycle my commit for test instances was mostly 50 concurrent jobs, and this cycle is mostly 40. As a member of the TC, I bring close to a decade of experience as an Operator. I feel the Openstack community can continue to deliver quality code while focusing on easing the burden of getting Openstack clouds into production. For a long time the focus of this community has been to develop bleeding edge features to deliver technical value to consumers. Now this community should focus on making those features easier to consume. I can view the TC through the eyes of a builder, tinkerer, trainer, and long time operator of Openstack. Thank you for your consideration. -- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"

1 0

[ironic][ops] Stable Ocata and Pike branches are now Unmaintained
by Iury Gregory 27 Mar '20

27 Mar '20

Greetings Ironicers and Operators, During the ironic upstream meeting held on 09 March 2020, we decided to move stable/ocata and stable/pike to unmaintained [1]. Apologies for sending this email almost 3 weeks after the decision was made. We have documented what unmaintained means in our contributor docs [2]. Feel free to rise any questions here or in the irc channel (#openstack-ironic) [1] http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-iro… [2] https://docs.openstack.org/ironic/latest/contributor/#our-policy-for-stable… Best regards, -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 0

openstack-dev Tacker
by Akshata Jitendra Gage 27 Mar '20

27 Mar '20

Hello, I am new to Tacker. As mentioned in the installation guide i have installed tacker on my server. I need to know that is there any way available by which we can create "service function chaining" via Horizon? And is there any forum related to tacker on which i can raise my doubts and questions? Thanks & Regards Akshata Gage

2 1

[election][neutron] PTL nomination
by Slawek Kaplonski 27 Mar '20

27 Mar '20

Hi, I want to propose my candidacy for Neutron PTL in the Victoria cycle. I have been serving this great team as PTL in the Ussuri cycle already and I would love to continue this in the next cycle as well. Ussuri was my first time as PTL and I learned a lot during this cycle. I think that this new experience and knowledge which I now have will help me serve as PTL in this next cycle as well. In Ussuri I proposed a couple major goals which I wanted to focus on: * merge networking-ovn driver into main Neutron repository - in Ussuri we actually merged networking-ovn code into neutron and made ovn driver as one of the main in-tree drivers for ML2. We also merged the networking-ovn core team into the neutron core team. * Improve overall stability of our CI - we are continuously working to improve CI stability: during this cycle we added couple OVN related jobs to Neutron CI, and we also removed some unnecessary jobs. Unfortunately we didn’t make enough progress with our CI and some jobs still need a lot of work to improve their stability. I think that this is a continuous effort and we will need to focus on it in the next cycle as well. * Improve test coverage for existing features - we introduced many new test cases in the neutron-tempest-plugin repository. Many of them are covering some additional scenarios and advanced functionality: multicast traffic, TCP and UDP port forwarding, and others. I think that this is a continuous effort and we will need to focus on that one in the next cycle as well. * Finish the transition to the new engine facade - this is something which we didn’t make enough progress so far and I would like to treat it as a priority in Victoria. * Improve the stability of Neutron when it is running under uWSGI - recently we promoted our uWSGI jobs to be voting. Those jobs are now pretty stable. I think that in the next cycle we need to finally focus on making uwsgi as the default in Devstack, so that it will be used in most Neutron CI job. In addition to that I wanted to continue Miguel’s way of mentoring potential new core reviewers and contributors, and I think that this has been working pretty fine. In addition to the networking-ovn core team which is now part of neutron core, we also have new core reviewer Lajos Katona and new member of drivers team: Nate Johnston. Our team is now in good shape and we have many active core reviewers and contributors. That’s my summary of what we actually achieved in the Ussuri cycle. Now let’s talk about what are my goals for the Victoria cycle. * Continue adoption of the OVN driver in Neutron - now that the OVN is in-tree, we need to think about making it the default in Devstack. * Finish adoption of new engine facade - it’s unfinished goal from the Ussuri cycle and I think that we should treat it as a priority now, * Metadata over IPv6 - we still have not finished the RFE about support for Metadata service in IPv6 only networks. In the Ussuri cycle we made some progress with the spec [1], and we also had some discussions about it with cloud-init developers. Now I think it’s time to finally get this spec merged and implement it in Neutron, * Continue work on test coverage and CI stability - this is a never ending story and I think that it should be treated with high priority all the time. It is my continuing desire to do my best to help our outstanding team to deliver better software and to grow. [1] https://review.opendev.org/#/c/315604/ — Slawek Kaplonski Senior software engineer Red Hat

1 0

[release] Release countdown for week R-6, March 30 - April 3
by Sean McGinnis 27 Mar '20

27 Mar '20

Development Focus ----------------- Work on libraries should be wrapping up, in preparation for the various library-related deadlines coming up. Now is a good time to make decisions on deferring feature work to the next development cycle in order to be able to focus on finishing already-started feature work. General Information ------------------- We are now getting close to the end of the cycle, and will be gradually freezing feature work on the various deliverables that make up the OpenStack release. This coming week is the deadline for general libraries (except client libraries): their last feature release needs to happen before "Non-client library freeze" on April 2. Only bugfixes releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/ussuri branching request with the review (as an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/…) In the next weeks we will have deadlines for: * Client libraries (think python-*client libraries), which need to have their last feature release before "Client library freeze" (April 9) * Deliverables following a cycle-with-rc model (that would be most services), which observe a Feature freeze on that same date, April 9. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. As we are getting to the point of creating stable/ussuri branches, this would be a good point for teams to review membership in their $project-stable-maint groups. Once the stable/ussuri branches are cut for a repo, the ability to approve any necessary backports into those branches for ussuri will be limited to the members of that stable team. If there are any questions about stable policy or stable team membership, please reach out in the #openstack-stable channel. Upcoming Deadlines & Dates -------------------------- Non-client library freeze: April 2 (R-6 week) Client library freeze: April 9 (R-5 week) Ussuri-3 milestone (including cycle highlights): April 9 (R-5 week) Cycle Highlights Due: April 9 (R-5 week) Ussuri final release: May 13

1 0

[neutron] Final release of non-client libraries
by Slawek Kaplonski 27 Mar '20

27 Mar '20

Hi, Next week is deadline for final releases for non-client libraries and I was checking what patches we are still in review and potentially we would need to have in Ussuri. *Neutron-lib* I see only https://review.opendev.org/#/c/712657/ which potentially may be good to get before final release. IMO it’s in good shape, please review it if You have few minutes (it’s small). *Ovsdbapp* There are 3 opened patches on https://review.opendev.org/#/q/project:openstack/ovsdbapp+branch:master+sta… But I think that maybe only https://review.opendev.org/#/c/683355/ may be potentially needed, mostly by ovn subteam. So please check if this is needed and we should update it to make it in before final Ussuri release. Os-ken I don’t see any important patches waiting in the list https://review.opendev.org/#/q/project:openstack/os-ken+branch:master+statu… so it should be good to go. If I missed anything or You have something else which You need to get in before this final release, please ping me by email or on irc. — Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][tc] Meeting Agenda April 2nd
by Jean-Philippe Evrard 27 Mar '20

27 Mar '20

Hello everyone, Here are the report topics for the tc next meeting, April 2nd. A few of those are most likely noop, tbh. Feel free to join us! - Report on tc/uc merge (ttx) - Report on the analysis of the 2019 survey (jungleboyj), and survey 2020. - Report on telemetry after Rico's convo with the PTL (ricolin) - report on stable branch policy work: mnaser to push an update to distill what happened on the ML in terms of stable branch policy (mnaser) - report on the community goals for U and V, py2 drop (gmann) - Moving OpenDev discussion forward (clarkb) - Report on the OSF board initiatives (mnaser) Please note that, as usual, the office hours are happening after the meeting, so if you want to chat over new things, it's a perfect place! Regards, Jean-Philippe Evrard

1 0

[tripleo][mistral] Farewell
by Dougal Matthews 27 Mar '20

27 Mar '20

Hey all, I am leaving Red Hat and my last day is tomorrow. As a result I will be moving on from OpenStack. I just wanted to say my farewells here and say thanks to everyone for all the fun over the years. As my last core duty I will be de-coring myself tomorrow :) If anyone wants to reach me beyond tomorrow, my email is probably best: dougal(a)dougalmatthews.com Cheers, Dougal

2 1

[nova][cinder] Why nova-api uses volume's size as bdm.volume_size?
by Zhi CZ Chang 27 Mar '20

27 Mar '20

1 0

[neutron] Drivers meeting - 27.03.2020
by Slawek Kaplonski 26 Mar '20

26 Mar '20

Hi, Due to lack of agenda for our next drivers meeting I will cancel it. I would like You to check list of opened, not triaged, old RFEs [1] and check if there are maybe some RFEs which You would like to continue work on, or maybe there are some which You think we should close as not needed anymore. Also if RFE on this list belongs to You, please respond to any questions from comments in this RFE or just let us know that You don’t want/don’t have time to continue work on it. Thx in advance [1] https://bugs.launchpad.net/neutron/+bugs?field.tag=rfe — Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][elections][ptl][tc] Combined PTL/TC Nominations Kickoff
by Kendall Nelson 26 Mar '20

26 Mar '20

Hello Everyone! Nominations for OpenStack PTLs (Project Team Leads) and TC (Technical Committee) positions (5 positions) are now open and will remain open until Mar 31, 2020 23:45 UTC. All nominations must be submitted as a text file to the openstack/election repository as explained at https://governance.openstack.org/election/#how-to-submit-a-candidacy Please make sure to follow the candidacy file naming convention: candidates/victoria/<Project_Name or "TC">/<email_address> (for example, "candidates/victoria/TC/stacker(a)example.org"). The name of the file should match an email address for your current OpenStack Foundation Individual Membership. Take this opportunity to ensure that your OSF member profile contains current information: https://www.openstack.org/profile/ Any OpenStack Foundation Individual Member can propose their candidacy for an available, directly-elected seat on the Technical Committee. In order to be an eligible candidate for PTL you must be an OpenStack Foundation Individual Member. PTL candidates must also have contributed to the corresponding team during the Train to Ussuri timeframe, Mar 22, 2019 00:00 UTC - Apr 07, 2020 00:00 UTC. Your Gerrit account must also have a verified email address matching the one used in your candidacy filename. Both PTL and TC elections will be held from Apr 07, 2020 23:45 UTC through to Apr 14, 2020 23:45 UTC. The electorate for the TC election are the OpenStack Foundation Individual Members who have a code contribution to one of the official teams over the Train to Ussuri timeframe, Mar 22, 2019 00:00 UTC - Apr 07, 2020 00:00 UTC, as well as any Extra ATCs who are acknowledged by the TC. The electorate for a PTL election are the OpenStack Foundation Individual Members who have a code contribution over the Train to Ussuri timeframe, Mar 22, 2019 00:00 UTC - Apr 07, 2020 00:00 UTC, in a deliverable repository maintained by the team which the PTL would lead, as well as the Extra ATCs who are acknowledged by the TC for that specific team. The list of project teams can be found at https://governance.openstack.org/tc/reference/projects/ and their individual team pages include lists of corresponding Extra ATCs. Please find below the timeline: nomination starts @ Mar 24, 2020 23:45 UTC nomination ends @ Mar 31, 2020 23:45 UTC campaigning starts @ Mar 31, 2020 23:45 UTC campaigning ends @ Apr 07, 2020 23:45 UTC elections start @ Apr 07, 2020 23:45 UTC elections end @ Apr 14, 2020 23:45 UTC Shortly after election officials approve candidates, they will be listed on the https://governance.openstack.org/election/ page. The electorate is requested to confirm their email addresses in Gerrit prior to 2020-03-31 00:00:00+00:00, so that the emailed ballots are sent to the correct email address. This email address should match one which was provided in your foundation member profile as well. Gerrit account information and OSF member profiles can be updated at https://review.openstack.org/#/settings/contact and https://www.openstack.org/profile/ accordingly. If you have any questions please be sure to either ask them on the mailing list or to the elections officials: https://governance.openstack.org/election/#election-officials -Kendall Nelson (diablo_rojo) & the Election Officials

3 2

[oslo] PTL Non-Candidacy
by Ben Nemec 26 Mar '20

26 Mar '20

This won't come as a surprise to most of you, but in case you missed the announcement at the beginning of this cycle I won't be able to serve as Oslo PTL for the next one. While I very much wish this were not the case, I do feel good about the current state of the Oslo team and I'm confident it will continue to do good things under someone else's leadership. Thanks to all of the Oslo contributors who have made the past two years a great experience! -Ben

4 3

[all][ptl] How to Contribute to the OpenStack Blog & Superuser Magazine
by Allison Price 26 Mar '20

26 Mar '20

Hey Stackers— We have received more questions recently about where the community can contribute content, especially on official OpenStack or community channels. Accompanied by the amount of events that have been transformed to virtual experiences, the OpenStack Foundation has been working on ways for the community to centrally share learnings online, and create a place where the community can send project, SIG, and overall community updates. Superuser and the OpenStack blog are community channels—for the community, by the community, so we would really like to get more contributors. We even had a user survey completed yesterday from an operator saying they wish the blog had more information on bug fixes and software updates, so here we are. Here’s how you can contribute: OpenStack Blog <https://www.openstack.org/blog/> We are reviving the OpenStack blog! Here, we want to publish OpenStack bug fixes, software updates, and project milestones that would be beneficial for users to know more about that may be more in depth. Often these are posted on the mailing list, which is great, but could also double as blog posts so that they are searchable in the future when operators stumble upon an issue. Once you have an idea of what you would like to share or have questions, please email community(a)openstack.org <mailto:community@openstack.org>. We can also keep track of a running list of ideas / calls for content for people who have questions that would be great to get answered on the blog. PTLs—heading into the Ussuri cycle, this is a great time to share your ideas on the etherpad around what you can write about. The OpenStack Foundation will reach out to folks who put their names or nicks in the etherpad as well as those who proactively reach out. Superuser <https://superuser.openstack.org/> Superuser has evolved to cover the open infrastructure landscape covering different use cases and open source projects. Whether it’s a video or an article, this content is typically more in depth than what you would see on a project blog and includes things like hands on tutorials, case studies, thought leadership articles, Q&As with open source community leaders, and updates and highlights from open source events around the world. If you have an idea you would like to share, fill out this form <https://superuser.openstack.org/contribute/> and the editorial team will reach out. If you have questions about whether your topic is the right fit, email editor(a)openstack.org <mailto:editor@openstack.org>. You can also attend the OSF community meeting <http://lists.openstack.org/pipermail/foundation/2020-March/002856.html>next week to learn more about how you can get involved and contribute! Thanks! Allison Allison Price OpenStack Foundation allison(a)openstack.org

1 0

[ironic] The Sanity Preservation Un-Conference, or SPUC - An Ironic gathering!
by Julia Kreger 26 Mar '20

26 Mar '20

Greetings Humans, Vulcans, AIs, and any other form of life! Today in the Ironic team meeting, we decided that we needed to have an un-conference! In the theme of an un-conference, I think we will start with two simple questions: * How is everyone? * What ironic (or non-ironic) things shall we discuss? >From there, we will identify topics of discussion, and possibly jump into separate video calls for ~15 minutes each, depending on the number of topics and general insanity. During the last ?15? minutes or so, we shall reconvene into the same video call to enjoy a little time sipping coffee Coffee, Tea or other delicious beverages while we determine who is wearing the silliest hat! So if you have a steampunk hat with gears or even some kind of hat you would wear to a fancy horse race, go right ahead! Only have a rose from your growing quarantine garden! You can wear that too! So what do you need to do! 1) Fill out the doodle[0] so we can actually schedule a time that works for a group of people! 2) Brainstorm some ideas of things you might want to talk about/present or LEARN! 3) Dust off your silly hats! NB: I know the windows on this doodle overlap with Baremetal SIG doodle that I sent out a little while ago. We will figure it out. -Julia [0]: https://doodle.com/poll/krtn9p4z7xx9pa29

6 8

[nova] API updates week 20-13
by Ghanshyam Mann 26 Mar '20

26 Mar '20

Hello Everyone, Please find the Nova API updates of this week. Please add if I missed any BPs/API related work. API Related BP : ============ COMPLETED: 1. Add image-precache-support spec: - https://blueprints.launchpad.net/nova/+spec/image-precache-support Code Ready for Review: ------------------------------ 1. Nova API policy improvement - Topic: https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+(status:ope… - Weekly Progress: Lot of Policies work merged and I am working on remaining ones. - Review guide over ML - http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008504.h… 2. Non-Admin user can filter their instance by availability zone: - Topic: https://review.opendev.org/#/q/topic:bp/non-admin-filter-instance-by-az+(st… - Weekly Progress: Nova change is merged, novaclient change is in-progress - https://review.opendev.org/#/c/713089/ 3. Boot from volume instance rescue - Topic: https://review.opendev.org/#/q/topic:bp/virt-bfv-instance-rescue+(status:op… - Weekly Progress: Code Review is in progress. 4. Add action event fault details -Topic: https://review.opendev.org/#/q/topic:bp/action-event-fault-details+(status:… - Weekly Progress: Few discussion on how we can minimize the info leak to non-admin. discussion is going on. Almost good. 5. Support re-configure deleted_on_termination in server -Topic: https://review.opendev.org/#/q/topic:bp/destroy-instance-with-datavolume+(s… - Weekly Progress: Discussion happened not to use PATCH for this and use existing PUT method. gibi will discuss this in today nova meeting on process and FF wise. Specs are merged and code in-progress: ------------------------------ ------------------ - None Spec Ready for Review or Action from Author: --------------------------------------------------------- -Marking as None here as Ussuri spec deadline is already over. Others: 1. None Bugs: ==== I started fixing policy bugs while working on policy-defaults-refresh BP. 5 bugs have been identified till now and fix up for review. - https://bugs.launchpad.net/nova/+bugs?field.tag=policy-defaults-refresh NOTE- There might be some bug which is not tagged as 'api' or 'api-ref', those are not in the above list. Tag such bugs so that we can keep our eyes. - https://etherpad.openstack.org/p/nova-bug-triage-ussuri -gmann

1 0

Regarding Split network plane for live migration
by Md. Hejbul Tawhid MUNNA 26 Mar '20

26 Mar '20

Hi, We are using Openstack Rocky . We are observing internal network is using while we do live migration. our internal network interface is 1G. So interface getting full and migration never getting complete. (nova server-migration-list ) 'Remaining Memory Bytes' increasing and decreasing but not getting completed. We have 10G interface for public/external network. I have tried following solution but no luck. still using internal network https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/s… Some Information: [DEFAULT] my_ip = Internal_network_IP [libvirt] live_migration_uri=qemu+ssh://nova@%s.external /system?keyfile=/var/lib/nova/.ssh/id_rsa&no_verify=1 Please let us know if any further information is required. Please advice us if we can solve the issue. Regards, Munna

3 4

[openstack-ansible][elections] Nomination
by Mohammed Naser 26 Mar '20

26 Mar '20

I'd like to announce my candidacy for OpenStack Ansible PTL. As usual, I want to go over all the points I talked about last time and their progress. # Simplifying scenarios for usage and testing We probably haven't progressed much in this, our testing has improved, we did add upgrade testing but unfortuantely that hasn't been as stable. # Using integrated repository for testing and dropping role tests This is pretty much done for the expect of a few corner projects. # Progress on switching all deployments to use Python3 This is also done with some packaging workarounds to make it work with CentOS 7. # Eventual addition of CentOS 8 option We made some progress in this, but it still needs a lot more work, big thanks to jrosser for driving a bunch of this work too. # Reduction in number of config variables (encouraging overrides) This has been done over a few roles now that are using overrides in hopes of minimizing the number of variables to speed up runs. # Increase cooperation with other deployment projects (i.e. TripleO) The Ansible SIG is moving and alive, TripleO uses python_venv_build and os_tempest as well as our connection plugins, so yay. We've worked on getting some new cores as well and making sure the project continues to be sustainable. Thank you for your consideration.

1 0

[tc][elections] Nomination
by Mohammed Naser 26 Mar '20

26 Mar '20

Hi everyone, I want to run again for the OpenStack technical committee for the upcoming cycle. I have been a long time operator of OpenStack with a focus on never using a local for, which has resulted in my involvement with many different projects by making quality-of-life improvements for these services that can help other deployers. I've contributed and worked with different sizes of teams inside OpenStack. >From our big projects such as Nova to projects providing specific services such as Magnum and pretty much most deployments tools at this point. On the operator side, I've helped run one of the longest-running OpenStack public clouds and many private clouds. I've also interacted with many other OpenStack operators that are not as involved in our community, which gives me a significant exposure to our operator story and what that looks like outside our community of known operators. I believe it's time now for me to start driving significant changes in our community to simplify the operator experience, and I think that should start from the technical committee. We're seeing the world evolve around us with new technologies, but we've remained relatively stable and stuck to our "known and trusted" stack. Operators have historically had a lot of frustrations with systems like RabbitMQ and their clustering issues, which end up making the OpenStack experience disappointing. There are many new ways of doing RPC, which we can leverage and help make it even easier to run OpenStack (things such as gRPC). We've also had a history of making it complicated to use containers to deploy OpenStack and historically refused to ship "official" containers. The OpenDev community has built our a step of tooling that can allow us to ship containers very quickly, with nothing more than a well written bindep.txt file and a simple multi-stage build. It's about time that we start having official images and easy ways to deploy OpenStack to reduce those barriers. We define OpenStack as a "cloud operating system." We struggle to deliver much more than the essential infrastructure components. The drive behind OpenStack was to allow people to get more than the necessary infrastructure components. Still, large amounts of different efforts lead us into complicated ways of having to wrangle VMs to run services. With popular container orchestrators and schedulers, OpenStack has a perfect platform to be able to launch those services. By setting up a proper framework, we can start delivering many services such as database as a service, memory store as a service, and so much more by building out Kubernetes operators. It also opens up a whole domain of possible contributors that can use those operators without OpenStack. In closing, I don't expect everyone to agree with all the different ideas I'm putting out. We may only drive just a few of them, or maybe none of them. However, I want to convince you that we need to change. I can understand they sound like big leaps, and they may seem uncomfortable. However, they are genuinely some of the things we need to do to continue to be relevant and put us at the leading edge of technology like we used to be years ago. Thank you for reading this. Regards, Mohammed

1 0

[keystone][horizon] Two approaches to "multi-region" OpenStack deployment
by Krzysztof Klimonda 26 Mar '20

26 Mar '20

Hi, I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds. The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc. Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica. This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails. The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources. While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks: * In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role) Additional thoughts Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon? In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense? In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require. -Chris

6 8

[tripleo] ptg topics
by Wesley Hayutin 25 Mar '20

25 Mar '20

Greetings, It's that time of year to start collecting topics for the PTG [1]. The in person PTG has been switched to a virtual meeting [2]. So the bad news is that the world is upside down atm, but the good news is that you may have a better lunch at the sessions ( I kid.. ) You can help plan the virtual ptg by signing up @ https://etherpad.openstack.org/p/Virtual_PTG_Planning Thanks all!! [1] https://etherpad.openstack.org/p/tripleo-victoria-topics [2] http://lists.openstack.org/pipermail/foundation/2020-March/002854.html

1 0

[all] Collecting Virtual Midcycle Best Practices
by Kendall Nelson 25 Mar '20

25 Mar '20

Hello Everyone! I wanted to collect best practices and pitfalls to avoid wrt projects experiences with virtual midcycles. I know of a few projects that have done them in the past and with how travel is hard for a lot of people right now, I expect more projects to have midcycles. I think it would be helpful to have all of the data we can collect in one place for those not just new to virtual midcycles but the whole community. I threw some categories into this etherpad[1] and filled in some options. Please add to it :) -Kendall (diablo_rojo) [1] https://etherpad.openstack.org/p/virtual-midcycle-best-practices

4 8

[tripleo] rdo registry migration
by Wesley Hayutin 25 Mar '20

25 Mar '20

Greetings, FYI.. the rdo registry is migrating from RDO to Vexxhost. You may find some of your jobs hitting "retry_limit" [1] This was caused by a default to log into registries for jobs running in rdo. The default has been updated. Jobs should return to their "GREENISH" status shortly [1] https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-…

1 0

[heat][magnum][tripleo] Official build and test plan for heat container agent
by Rico Lin 25 Mar '20

25 Mar '20

Dear all tl;dr: Shall we build, publish, and test heat container agents on heat-agents directly?:) Please help to review https://review.opendev.org/#/q/topic:build-container-agent+(status:open) Current I think we encounter with following issues: ** No functional/scenario test for heat-agents but only unit tests* Right now, whatever patch people send up in heat-agents, we only vote against unit tests. This makes no sense for long term maintenance. Which makes me thinking of the second issue. ** No enable scenario tests for software deployment:* We have tests defined, but since we didn't build any customized image. We didn't enable it for a long time. To build a container image for heat-agents (aka heat container agent) is (IMO) the best way for us to enable these teats. Which we can also use it to cross gating on heat and heat-agents (or even heat-template). The no much to change for the test itself but only update the config script to enable the container. ** Can't guarantee cross-project usage:* We currently provide no guarantee for any usage of heat agents from other projects, especially Magnum and TripleO (I didn't personally check if TripleO still using it, but since they still have experiment CI running) who uses heat container agents. It will be nice if we provide basic images for other teams can build their additional requirements. To resolve the above issues, I propose [1]. Which a lot of logic was copied from the current magnum heat container agent scripts [2] so we can start with a higher possibility to allow magnum to reuse our works. For a more detail approach in [1]. The patch defines a build job in `check` pipeline. And a publish job in `post-check`. Uses zuul secrets to save docker authN information. As for the account, I created a new `openstackheat` account [3] for it (I specifically recall there ware discussion about having an official image hub for OpenStack, but not quite sure what happened since). The current approach also can support multiple architectures (like amd64 arm arm64 ppc64le s390x, etc). But for now, I think we should have tested before we publish for other architectures. You can run the build or publish job easily by running `make build-images-all-archs` to build images for all architectures or `make upload-images-all-archs` to build and publish images for all architectures. [1] https://review.opendev.org/#/q/topic:build-container-agent+(status:open) [2] https://opendev.org/openstack/magnum/src/branch/master/dockerfiles/heat-con… [3] https://hub.docker.com/r/openstackheat -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 1

[cinder][qa] propose Liang Fang for devstack-plugin-open-cas-core
by Brian Rosmaita 25 Mar '20

25 Mar '20

Liang Fang is driving the effort to create the devstack-plugin-open-cas and use it to test the cinder/nova volume-local-cache feature with tempest. Since he's our SME on open-cas, I propose that he be made a member of devstack-plugin-open-cas-core. (Currently, the only members of that group are the members of cinder-core and devstack-core.) In the absence of objections, I'll make the change before the next Cinder team meeting (Wednesday, 25 March at 1400 UTC in its new location of #openstack-meeting-alt). cheers, brian

4 4

Re: [Release-job-failures] Release of openstack/python-openstackclient for ref refs/tags/5.1.0 failed
by Thierry Carrez 25 Mar '20

25 Mar '20

zuul(a)openstack.org wrote: > Build failed. > > - release-openstack-python https://zuul.opendev.org/t/openstack/build/4519cd03309c4b5e834a9089183f6b3b : SUCCESS in 4m 00s > - announce-release https://zuul.opendev.org/t/openstack/build/7a30f87bded640f0a1f6c418d1dcce66 : FAILURE in 4m 34s > - propose-update-constraints https://zuul.opendev.org/t/openstack/build/eb428894cf24407a9991d22f40de98d7 : SUCCESS in 4m 25s During python-openstackclient 5.1.0 release: Error while sending announcement email: 'Temporary local problem - please try later' It's not the first time this happens, so maybe we should have a retry in there. -- Thierry Carrez (ttx)

1 0

[largescale-sig] Next meeting: Mar 25, 9utc
by Thierry Carrez 25 Mar '20

25 Mar '20

Hi everyone, The Large Scale SIG will have a meeting this week on Wednesday, Mar 25 at 9 UTC[1] in #openstack-meeting on IRC: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200325T09 Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting A reminder of the TODOs we had from last meeting, in case you have time to make progress on them: - amorin to create a wiki page for large scale documentation - amorin to propose patch against Nova doc - all to check/comment on https://etherpad.openstack.org/p/LargeScaleOps_OpenDev - all to review new patchset of oslo.metrics spec https://review.opendev.org/#/c/704733/ - oneswig to contribute a scaling story on bare metal cluster scaling Talk to you all on Wednesday, -- Thierry Carrez

1 1

[ops] need help configuring glance policies
by rajiv mucheli 25 Mar '20

25 Mar '20

Hi there, In regards to https://answers.launchpad.net/glance/+question/689450, i need help configuring glance policies. I did review the excellent videos but had no response from the authors over IRC. The 403 errors occur with admin_role = '' Please suggest how to proceed. Regards, Rajiv

1 0

10 Years of OpenStack Survey
by Sunny Cai 24 Mar '20

24 Mar '20

Hello Stackers! To celebrate 10 years of OpenStack, we are launching a campaign that not only celebrates the community’s achievement in the past 10 years, but also showcases the top 10 favorite things about OpenStack from our community members. We want to hear from you about what your top 10 favorite things related to OpenStack are! Go to this survey <https://openstackfoundation.formstack.com/forms/10_years_of_openstack> and choose one or more topics to answer. The topics range from your top 10 most memorable moments of OpenStack, your top 10 most used features in OpenStack to your top 10 favorite cities you visited for OpenStack. Top 10 favorites of OpenStack survey link: https://openstackfoundation.formstack.com/forms/10_years_of_openstack <https://openstackfoundation.formstack.com/forms/10_years_of_openstack> Please also share it to your networks including folks who were around at the beginning of OpenStack. We are looking forward to hearing your favorites, and we invite you all to join us and celebrate 10 awesome years of OpenStack! Thanks, Sunny Cai OpenStack Foundation sunny(a)openstack.org

1 0

[baremetal-sig][ironic] Lets finish the whitepaper doodle
by Julia Kreger 24 Mar '20

24 Mar '20

Greetings fellow humans! During today's ironic meeting, the idea of just getting on a video call and hammering out the remainder of the baremetal SIG whitepaper was raised. After some quick discussion in the community, I'm sending out a doodle[0] for us to find a few time windows that works, where we can try to gather together and hammer out words, cut/paste/massage content, and overall just try and get it done. In the next 24-48 hours, I'll contact the respondents to and schedule a call, where I'll also send out call details. Please respond to the doodle quickly. -Julia [0] https://doodle.com/poll/k4wnmuay34mvh94v

1 2

nova-api missing heartbeats to rabbitmq
by Satish Patel 24 Mar '20

24 Mar '20

Folks, Recently i am seeing lots of error message in rabbitmq logs saying missing heartbeats from nova-api nodes, I am not seeing any issue at functionality level as everything working fine but just noticed those error and trying to find root cause of it. 172.28.15.125 nova-api server 172.28.15.192 rabbitmq server on rabbit.log 2020-03-24 12:21:41.389 [error] <0.29772.4418> closing AMQP connection <0.29772.4418> (172.28.15.125:42656 -> 172.28.15.192:5671 - uwsgi:32419:9b8a323b-653d-4585-9916-d52b3fd81d59): missed heartbeats from client, timeout: 60s on nova-api.log 2020-03-24 12:19:06.554 32435 ERROR oslo.messaging._drivers.impl_rabbit [-] [4b8adff0-ff9f-4863-a939-537d391e5d9e] AMQP server on 172.28.15.192:5671 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer

4 3

[tc] April 2nd meeting agenda
by Jean-Philippe Evrard 24 Mar '20

24 Mar '20

Hello folks, If you have to update the agenda of our next monthly meeting, it's still time! I will send the definitive agenda on the ML Thursday. Regards, JP [1]: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[neutron] OVS inactivity probes
by James Denton 24 Mar '20

24 Mar '20

Hello all, Like others, we have seen an increase in the amount of messages like those below, followed up by disconnects: 2020-03-19T07:19:47.414Z|01614|rconn|ERR|br-int<->tcp:127.0.0.1:6633: no response to inactivity probe after 10 seconds, disconnecting 2020-03-19T07:19:47.414Z|01615|rconn|ERR|br-ex<->tcp:127.0.0.1:6633: no response to inactivity probe after 10 seconds, disconnecting 2020-03-19T07:19:47.414Z|01616|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 10 seconds, disconnecting We have since increased the value of of_inactivity_probe (https://bugs.launchpad.net/neutron/+bug/1817022) and confirmed the Controller connection reflects this new value: --- # ovs-vsctl list Controller ... _uuid : b4814677-c6f3-4afc-9c9e-999d5a5ac78f connection_mode : out-of-band controller_burst_limit: [] controller_rate_limit: [] enable_async_messages: [] external_ids : {} inactivity_probe : 60000 is_connected : true local_gateway : [] local_ip : [] local_netmask : [] max_backoff : [] other_config : {} role : other status : {last_error="Connection refused", sec_since_connect="1420", sec_since_disconnect="1423", state=ACTIVE} target : "tcp:127.0.0.1:6633" --- However, we also see disconnects on the manager side, which the config option does not address: 2020-03-23T11:01:02.871Z|00443|reconnect|ERR|tcp:127.0.0.1:50098: no response to inactivity probe after 5 seconds, disconnecting This bug (https://bugs.launchpad.net/neutron/+bug/1627106) and related commit (https://opendev.org/openstack/neutron/commit/1698bee770b84a2663ba940a6ded5d…) appear to leverage the ovs_vsctl_timeout value (since renamed to ovsdb_timeout), but the inactivity_probe for the Manager connection does not appear to be implemented. Honestly, I'm not sure if that code path is used. --- # ovs-vsctl list Manager _uuid : d61519ba-93fc-4fe5-b05c-b630778a44b0 connection_mode : [] external_ids : {} inactivity_probe : [] is_connected : true max_backoff : [] other_config : {} status : {bound_port="6640", n_connections="2", sec_since_connect="0", sec_since_disconnect="0"} target : "ptcp:6640:127.0.0.1" --- Running this by hand sets the inactivity_probe timeout on manager connection, but we'd prefer to use a built-in method, if possible: # ovs-vsctl set manager d61519ba-93fc-4fe5-b05c-b630778a44b0 inactivity_probe=30000 Any suggestions? Thanks, James

2 2

[glance] weekly meeting IRC channel change
by Abhishek Kekane 24 Mar '20

24 Mar '20

Hi All, Beginning with this Thursday (26 March), the Glance team weekly meeting will be held in: #openstack-meeting The time continues to be 1400 UTC on Thursdays. Thanks & Best Regards, Abhishek Kekane

1 0

[nova][ptg] PTG participation
by Balázs Gibizer 24 Mar '20

24 Mar '20

Hi, I've just got the news from my employer that due to COVID19 I cannot travel to Vancouver in June. cheers, gibi

3 4

[nova] feasibility to keep the two-company rule
by Balázs Gibizer 24 Mar '20

24 Mar '20

Hi, Nova has an unwritten rule that requires to have at least two companies involved in any new feature development (or even bugfix?). In the current Nova core diversity situation this rule puts extra burden to the remaining non Red Hat cores and I guess it also makes any Red Hat driven feature development harder. In parallel to working on increasing the size of the core team I suggest to reconsider dropping this rule. Some discussion happened already on the today's meeting[1]. Cheers, gibi [1] http://eavesdrop.openstack.org/meetings/nova/2020/nova.2020-03-19-16.00.log…

10 10

[nova] FYI for out-of-tree virt drivers : new argument for finish_migration() and finish_revert_migration()
by Sylvain Bauza 24 Mar '20

24 Mar '20

I wrote two changes [1] for passing down target allocations to virt drivers given we need to use them for closing some bugfix [2] Given the Nova contributor policy is saying that the public API is not related to virt drivers, we won't support virt drivers that won't accept this new argument. If you're a contributor that works on any out-of-tree Nova virt driver, please make sure that you modify your own two above methods. Thanks, -Sylvain [1] https://review.opendev.org/#/c/589085/7 and https://review.opendev.org/#/c/712118/2 [2] https://bugs.launchpad.net/nova/+bug/1778563

2 1

Summary: Ironic Mid-cycle at CERN
by Julia Kreger 23 Mar '20

23 Mar '20

A couple weeks ago (February 25-26), the Ironic community convened its first mid-cycle in quite a long time at the invitation of and encouragement of CERN. A special thanks goes to Arne Wiebalck for organizing the gathering. We spent two days discussing, learning, sharing, and working together to form a path forward. As long time contributors, some of us were able to bring context of not just how, but why. Other community members brought questions and requirements, meanwhile one of the hardware vendors brought their context and needs. And CERN was kind enough to show us how our work matters and makes a difference, which was the most inspiring part of all! Special thanks goes to Dmitry Tantsur, Riccardo Pittau, and Iury Gregory for helping me keep momentum moving forward on this summary. --------------------------------------- Deploy Steps ========== Discussed issues related to the actual workflow, concerns out process efficiency and path forward. The issue in question was the advance validation of deploy steps when some of the steps come from the ironic-python-agent ramdisk and are not reflected in the server code. Creating the whole list of steps for validation and execution requires information from the ramdisk, but it’s only available when the ramdisk is booted. We have discussed the following alternatives: * Start ramdisk before deploy step execution. This was ruled out for the following reasons: ** Some steps need to be executed out-of-band before the ramdisk is running. This is already an issue with iDRAC clean steps. ** The first deploy steps validation happens in the node validation API when the ramdisk is clearly not running. * Using cached deploy steps from the previous cleaning run. It was ruled out because: ** Some deployments disable automated cleaning. ** The deploy step list can change in between, e.g. because of hardware changes or any external input. * Accept that we cannot provide early validation of deploy steps and validate them as we go. This involves booting the ramdisk as one of the deploy steps (no special handling), with only out-of-band steps executed before that (ignoring any in-band steps with higher priority). We decided to go with the third option. In a side discussion we decided to file an RFE for a driver_info flag preventing booting the ramdisk during manual cleaning. Solving the cleaning issues completely probably requires making booting the ramdisk a separate clean step, similarly to deploy steps above. No final plan has been made for it, but we have more clarity than we did before. Security ====== Firmware Management -------------------------------- We entered a discussion about creating a possible “meta step”. After some back and forth discussions, we reached a consensus that it is likely not possible given different vendor parameters and requirements. During this topic, we also reached the point of discussing changes to “Active node” configuration as it is related in large part to firmware updates, and is necessary for larger fleet management, and eventually even for attestation process integration. The consensus kind of revolved around that the way to help enable some of this process was to leverage rescue, this is only theory. Hopefully we’ll have operator feedback in the next year on this subject and can make more informed decisions. By then, we should have deploy steps in a form that one doesn't need to be a python developer to leverage, and the team should be bandwidth to explore this further with operators. Attestation --------------- This is a topic that Julia has been raising for a while because there is a logical and legitimate reason to go ahead and implement some sort of integration with an attestation platform to perform system measurement and attestation during cleaning and deployment processes in order to help identify if machines were tampered with. In our case, remote attestation is likely the way forward, and inspiration can come looking at Keylime (TPM based highly boot attestation and run-time integrity measurement solution, and most important, opensource). We’ll need an implementation to cover at least Clean/Deploy steps, to be able to run and validate TPM measurement, and fail the deployment if attestation fails. We still need to figure out what the actual impact to firmware upgrade is, and how to safely ensure that a re-measurement is valid, or not, and when to trust the measurement is actually valid. Ironic’s next step is to begin to talk to the Keylime folks in more depth. Also one of our contributors, Kaifeng, who read our notes etherpad indicated that he is working towards the same area as well, so we may see some interesting and fruitful collaboration because ultimately we all have some of the same needs. Agent Tokens ------------------ Agent tokens was possibly the quickest topic that we visited with Dmitry suggesting we just needed to add a unit test and merge the code. To be further secure, we need the agent ramdisk to begin using TLS. TODO: Julia is to send out an email to the mailing list to serve as notice to operators that ironic intends to break backwards IPA compatibility next cycle by removing support for agents that do not support agent tokens. NOTE: As we type/refine this summary for distribution, the agent token code has largely merged, and should be completely merged before the end of the current development cycle. TLS in virtual media --------------------------- In order to secure Agent token use, we need to secure their transmission to the ironic-python-agent when commands are issued to the agent from the conductor. Ultimately we’re hoping to begin work on this soon in order to better secure interactions and communications with machines in remote “edge” environments. An RFE has been filed to automatically generate certificates and exchange them with the ramdisk: https://storyboard.openstack.org/#!/story/2007214. Implementing it may require downstream consumers to update their USA export control certification. FIPS 140-2 --------------- This was a late addition topic that was added while other discussion was coming up, and largely more for the purposes of community visibility. In short, we know based on some recent bugs that were fixed, that operators are starting to try and deploy Ironic in environments and on hosts that are configured for a FIPS 140-2 operating mode, in short a much more strict cryptography configuration. We ought to make sure that we don’t have any other surprises waiting for us, so the call went out for someone to at some point review the standard, and sanity check Ironic and components. Post-IPMI universe =============== The decline of IPMI is one that we, as a community, need to plan ahead for as some things become a little more difficult. Discovery ------------- Node discovery, as a feature, is anticipated to become a little more complicated. While we should still be able to identify a BMC address,that address may be the in-band communications channel address once vendors are supporting the Redfish host interface specification. This spurred discussion of alternatives, and one of the items that was raised was possibly supporting the discovery of BMCs using the SSDP and UPnP. This raises an interesting possibility in that the UUID of the BMC is retrievable through these means. It seems logical for us to one day consider the detection and enrollment of machines using an operator tool of some sort. This functionality is defined by the Redfish standard and, as everything in Redfish, is optional. The DMTF provided Redfish library contains an example implementation: https://github.com/DMTF/python-redfish-library/blob/master/src/redfish/disc…. Using System ID/GUID ------------------------------- The discovery topic spurred another topic of if we should be using the System UUID or GUID identifier in addition to or instead of a MAC address on the chassis. Ironic Inspector folks have been considering having additional or even plug-able matches for a long time. The system UUID can be discovered in-band via SMBios before launching inspection/discovery. Supporting this would largely be more of a feature matching a physical machine, but some of our code requires network information anyway, so it may not bring a huge benefit upfront beyond trying to match BMC<->Host. IPMI cipher changes ---------------------------- CERN folks were kind enough to raise an issue that has brought themselves some headaches as of recent which is that some BMC vendors have changed cipher logic and keying so they’ve had to put in some workarounds and modified ipmitool builds on their systems. As far as we’re aware as a community, there is really nothing we can directly do to help them remove this work around, but ultimately this headache may cause them to begin looking at Redfish and cause some development on serial console support for Redfish. Making inspector data a time series =========================== One of the challenges in data centers is identifying when the underlying hardware changes. When a disk is replaced, the serial number changes, and if that disk is in a running system, it would traditionally have needed to be re-inspected in order for information about the machine to be updated. But we added the ability to manually execute and submit this data in the last development cycle, so if there was any time series nature to inspection data, then it allows for changes to be identified, new serial numbers recorded, etc. The overwhelming response during the discussion was “Yes Please!”, in that such a feature would help a number of cases. Of course, what we quickly reached was disagreement over meaning. Turns out, the purpose is more about auditing and identifying changes, so even if there are only two copies, the latest and the previous inspection data, then differences could be identified by external tooling. A spec document or some sort of written MVP will ultimately be required, but the overall concept was submitted to Outreachy. DHCP-less deploy ============== In regard to the DHCP-less deploy specification (https://review.opendev.org/#/c/672780/) we touched upon several areas of the specification. We settled on Nova's network metadata format (as implemented by Glean) as the API format for this feature. Ilya has voiced concerns that it will tie us to Glean closer than we may want. Scalability of rebuilding ISO images per node. The CERN folks rightfully expressed concern that a parallel deployment of several hundred nodes can put a significant load on conductors, especially in terms of disk space. * In case of hardware that has more than one usable virtual media slot, we can keep the base ISO intact and use a 2nd slot (e.g. virtual USB) to provide configuration. * The only other option is documenting it as a limitation of our virtual media implementation. To get rid of the MAC address requirements for DHCP-less virtual media deployments, we determined that it will be necessary to return to sending node UUID and other configuration to the ramdisk via boot parameters. This way we can avoid the requirement for MAC addresses, although this has to be navigated carefully and with Operator feedback. An additional concern, beyond parallel deployment load, was “Rescue” support. The consensus seemed to be that we put giant security warnings in the documentation to signal the security risk of the ramdisk being exposed to a potentially un-trusted network. Agent token work _does_ significantly help improve operational security in these cases, but operators must be cognizant of the risks and potentially consider that rescue may be something they might not want to use under normal circumstances with network edge deployments. OOB DHCP-less deploy =================== We briefly touched on OOB dhcp-less deployment. This is HTTPBoot asserted through to the BMC with sufficient configuration details, ultimately looking a lot like DHCP-less deployments. Interest does seem to exist on this topic, but we can revisit once the base DHCP-less deployment work is done and hopefully an ironic contributor has access hardware where this is an explicit feature of the BMC. CI Improvements and Changes ======================== The upstream CI, and ways to make it more stable and just improve its efficiency, is a recurrent main discussion argument not only as part of the meetup. The impact of the CI in the day-to-day work is very high, and that’s why we took our time to talk about different aspects and do a proper analysis of the different jobs involved. The discussion started with the proposal of reducing the usage of the ironic-python-agent images based on TinyCoreLinux (the so-called tinyipa images), and rely more on the images built using diskimage-builder (DIB) and specifically CentOS 8 as base. This proposal is based on the fact that DIB-built images are what we recommend for production usage, while tinyIPA images have known issues on real hardware. Their only real benefit is a much smaller memory footprint (roughly 400MiB vs 2GiB of a CentOS 8 image). We have agreed to switch all jobs that use 1 testing VM to pre-built CentOS 8 images. This covers all jobs, except for the standalone, multi-node and grenade (upgrade testing) ones. While reviewing the current list of jobs, we realized that there is a lot of duplication between them. Essentially, most of the image type - deploy interface combinations are already tested in the standalone job. As these combinations are orthogonal to the management technology, we can use Redfish instead of IPMI for some of the tests. We decided to split the ironic-standalone job since it covers a lot of scenarios from tempest and it has a high failure rate. The idea to have one job testing software raid, manual cleaning and rescue, the other tests that consists in the combination of image type - deploy interface will be split in two jobs (one using IPMI and the other using Redfish). One other point that we reached was some consensus that more exotic, non-openstack focused CI jobs, were likely best to be implemented using Bifrost as opposed to Tempest. Third Party CI/Driver Requirements ----------------------------------------------- The question has been raised with-in the community if we reconsider 3rd Party CI requirements. For those that are unaware, it has been a requirement for drivers to merge into ironic to have Third-Party operated CI. Operating Third Party CI helps exercise drivers to ensure that the driver code is functional, and provides the community information in the event that there is a breaking changer or enhancement made. The community recognizes third party CI is difficult, and can be hard at times to keep working as the entire code base and dependencies evolve as time moves on. We discussed why some of these things are difficult, and what can we, and the larger community do to try and make it easier. As one would expect, a few questions arose: Q: Do we consider “supported = False” and keeping drivers in-tree until we know they no longer work? A: The consensus was that this is acceptable. That the community can keep unit tests working and code looking sane. Q: Do we consider such drivers as essentially frozen? A: The consensus is that drivers without third party CI will be functionally frozen unless changes are required to the driver for the project to move forward. Q: How do we provide visibility into the state of the driver? A: The current thought is to return a field in the /v1/drivers list to signal if the driver has upstream testing. The thought is to use a field named “Tested” or something similar as opposed to the internal name in the driver interface which is “supported” Q: Will we make it easier to merge a driver? A: Consensus was that we basically want to see it work at least once before we merge drivers. It was pointed out that this helped provide visibility with some of the recently proposed driver code which was developed originally against a much older version of ironic. Q: Do third party CI systems need to run on every patch? A: Consensus is No! A number of paths in the repository can be ignored from changes. In other words, there is no reason to trigger an integration test of Third Party CI for a documentation change or an update to a release note, or even a change to other vendor’s drivers. In summary Drivers without Third Party CI are “use at own risk” and removal is moving towards a model of “don’t be brutal”. This leaves us with a number of tasks in the coming months: * Update the contributor documentation with the Questions and Answers above. * Author an explicit exception path for the process of bringing CI back up as it pertains to drivers, essentially focusing on communication between the ironic community and driver maintainers. * Author a policy stating that unsupported drivers shall be removed immediately upon being made aware that the driver is no longer functional and without a clear/easy fix or path to resolution. * Solicit pain points from driver maintainers who have recently setup or do presently maintain Third Party CI and try to aggregate the data and maybe find some ways of improving the situation. “Fishy Politics”: Adapting sushy for Redfish spec versus implementation reality ============================================================ Everyone’s favorite topic is how implementations differ from specification documents. In part, the community is increasingly seeing cases where different vendors have behavior oddities in their Redfish implementations. We discussed various examples, such as https://storyboard.openstack.org/#!/story/2007071, and the current issues we’re observing on two different vendors around setting the machine boot mode and next boot device at the same time. For some of these issues, the idea of having some sort of Redfish flavor indicator suggested so the appropriate plugin could be loaded which might be able to handle larger differences like major field name differences, or possibly endpoint behavior differences like “PUT” instead of “PATCH”. This has not yet been explored but will likely need to be explored moving forward. Another item for the ironic team to be mindful moving forward, is that newer UEFI specific boot setting fields have been created, which we may want to explore using. This could give us a finer level of granularity of control, and at the same time may not be really usable in other vendor’s hardware due to the data in the field and how or what to correspond it back to. Kexec (or "Faster booting, yes?") ========================= This topic concerns using the kexec mechanism instead of rebooting from the ramdisk to the final instance. Additionally, if it is acceptable to run the agent on the user instance, it can be used for rebuilding and tearing down an instance. Potentially saving numerous reboots and "Power On Self Tests" in the process. We have one potential issue: with multi-tenant networking there is a possibility of a race between kexec and switching from the provisioning to the tenant network(s). In a normal deploy we avoid it by powering the node off first, then flipping the networks, then booting the final instance (on success). There is no such opportunity for kexec, meaning that this feature will be restricted to flat network cases. The group has expressed lots of interest in providing this feature as option for advanced operators, in other words “those who run large scale computing farms”. Julia proposed making a demo of super-fast deployment using fast-track and kexec as a goal moving forward and this received lots of positive feedback. Partitioning, What is next? ==================== Dmitry expressed that there seemed to be lots of interest from EU operators in supporting disk partitioning. This has been long sought, but with minimal consensus. We discussed some possible cases how this could be supported and we reached conclusion that the basis is largely just to support Linux Logical Volume Manager in the simplest possible configuration. At the same time the point was raised that parity basically means some mix of software RAID through LVM and UEFI boot. We soon realized we needed more information! So the decision was reached to start by creating a poll, with questions in three different languages to try and identify community requirements using some simple and feasible scenarios. Such as LVM on a single disk, LVM on multiple disks, LVM + image extraction on top of the LVM. The partitioning topic was actually very power in that we covered a number of different topics that we were not otherwise planning to explicitly cover. Network booting ---------------------- One of them being why not use network booting, and the point was made that Network is our legacy and fundamental for iSCSI based boot and ramdisk booting (such as the deployment ramdisk). During this dive into ironic’s history, we did reach an important point of consensus which is that Ironic should switch the default boot mode, as previously planned, and still keep at least one scenario test running in CI which uses network booting. Stated operator wants in terms of Partitioning ------------------------------------------------------------- Dmitry was able to provide us some insight into what the Russian operator community was seeking from Ironic, and Julia confirmed she had heard similar wants from public cloud operators wanting to offer Bare Metal as a Service. Largely these wants revolve around wanting LVM capability of the most basic possible scenario such as a single disk or a partition image with a LVM, or even Software RAID with partition images. Likely what has stalled some of these discussions in the past is the immediate focus on the more complex partitioning scenarios sought by some operators in the past, which has resulted in these discussions stalling due to complexity of requirements. Traits/Scheduling/Flavor Explosion =========================== Arne with CERN raised this topic to bring greater awareness. CERN presently has greater than 100 flavors representing their hardware fleet as each physical machine type gets its own flavor. This has resulted in pain from the lack of flavor specific quotas. What may help in this area is for Resource Class based quotas, but presently the state of that work is unknown. The bottom line: A user does not have clarity into their resource usage. The question then shifted to being able to report utilization since the current quota model is based on cores/RAM/instances but not resource_class consumption as a whole. The question largely being “How many am I allowed to create? [before I eat someone else’s capacity]”. https://github.com/stackhpc/os-capacity was raised as a reporting tool that may help with these sorts of situations with bare metal cloud operators. Another point raised in this discussion was the lack of being able to tie consumer and project ID to consumed resources, but it turns out the allocation list functionality in Placement now has this functionality. In the end of this discussion, there was consensus that this should be brought back to the Nova community radar. Machine Burn-in ============= Burning in machines is a regular topic that comes up, and it looks like we’re getting much closer to being able to support such functionality. Part of the reason behind discussing this was to determine how to move forward what organizations like CERN can offer the community. There are two use cases. The most important thing is to ensure that hardware does not fail. That doesn’t seem like a “cloudy” thing to have to worry about, but when you're building your cloud, you kind of want to make sure that suddenly half your hardware is not going to fail as soon as you put a workload on it. The second use case is nearly just as important, which is to be able to ensure that you are obtaining the performance you expect from the hardware. This brought a bit of discussion because there are fundamentally two different paths that could be taken. The first is to leverage the inspector, whereas the second is to use clean steps. Both have very useful possible configurations, largely there is no sophisticated data collection in terms of performance data. That being said, the consensus seemed to be that actual data collection was less of a problem than flexibility to invoke as part of cleaning and preparation of a machine into the deployment. In other words, the consensus seemed to be clean-steps would be ideal for community adoption and code acceptance. IPv6/Dual Stack ============ TL;DR, we need to remove the ip_version setting field. This is mostly a matter of time to sift through the PXE code, and determine the code paths that need to be taken. I.E. for IPv6, we would likely only want to signal flags for it if the machine is in UEFI mode. The dhcp-less work should provide some of the API side capabilities this will really need in terms of interaction with Neutron. Graphical Console ============== The question was raised to “what would it take to finally get this moving forward again?” This is because there is initial interface code, two proofs of concept, and it should be relatively straightforward to implement redfish support, OEM capability dependent. The answer was functionally “Someone needs to focus on this for a couple of months, keep it rebased, and engage the community”. The community expressed an absolute willingness to mentor. Software RAID - Specifying devices =========================== We have briefly discussed RFE https://storyboard.openstack.org/#!/story/2006369 that proposes a way to define which physical devices participate in software RAID. A similar functionality already exists in the RAID configuration format for hardware RAID, but software RAID always spans all hard drives, no matter how many. The RFE proposes re-using the same dictionary format as used for root device hints in the “physical_disks” field of the RAID configuration. This idea has been accepted by the audience, with Arne proposing to extend supported hints with a new “type” hint with values like “rotational”, “nvme” or “ssd”. Stickers ====== Yes, we really did discuss next steps for stickers. We have ideas. Lots of ideas… and we are all very busy. So we shall see if we’re able to make some awesome and fun stickers appear for the Berlin time frame.

2 1

[tempest][gate] tempest-tox-plugin-sanity-check broken by whitebox-tempest-plugin
by Artom Lifshitz 23 Mar '20

23 Mar '20

Hey all, A recent change in the whitebox-tempest-plugin [1] has broken the tempest-tox-plugin-sanity-check job (ex: [2]). The cause is a skipCheck that uses a whitebox-specific config option that had no default value and is obviously not set in any jobs outside of whitebox itself. When attempting to compare a None and an int, a TypeError is thrown. This has been fixed in [3]. The disruption should be minimal, but please don't: 1. recheck tempest until [3] merges 2. blacklist whitebox from the tempest-tox-plugin-sanity-check job :) Cheers! [1] https://review.opendev.org/#/c/699864/ [2] https://review.opendev.org/#/c/711049/8 [3] https://review.opendev.org/#/c/714539/

2 2

[cinder] reminder: weekly meeting location change
by Brian Rosmaita 23 Mar '20

23 Mar '20

(If you missed last week's meeting, this email may be the first you've heard about this change.) Beginning with this Wednesday (25 March), the Cinder team weekly meeting will be held in: #openstack-meeting-alt The time continues to be 1400 UTC on Wednesdays. cheers, brian

1 0

[neutron] Week 13 - bug deputy report
by Slawek Kaplonski 23 Mar '20

23 Mar '20

Hi, Last week I was on bug deputy. Below is my summary of it. *Critical* None *High* https://bugs.launchpad.net/neutron/+bug/1867936 - test_update_delete_extra_route failing due to timeout when creating subnets - HIGH as it’s gate failure This has no assignment yet. For me it looks like generall issue which we have from time to time with some very slow responses from API but it has to be verified. https://bugs.launchpad.net/neutron/+bug/1868237 - [tempest] Error in "test_multicast_between_vms_on_same_network" test case - HIGH as this is gate-failure No assignment yet. https://bugs.launchpad.net/neutron/+bug/1868110 - [OVN] neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log randomly fails - HIGH as it’s gate failure maciejjozefczyk is working on it *Medium* https://bugs.launchpad.net/neutron/+bug/1868100 - ncat process isn't always spawned on guest vm in scenario tests - MEDIUM Already fixed https://bugs.launchpad.net/neutron/+bug/1868125 - [ovn] Metadata agent spawns haproxy quickly twice on a single new port binding - MEDIUM Already fixed https://bugs.launchpad.net/neutron/+bug/1868137 - default geneve overhead is set not enough - MEDIUM Someone from ovn subteam can take a look *Low* https://bugs.launchpad.net/neutron/+bug/1867869 - [OVN] Remove dependency on fip_object - LOW Patch proposed https://review.opendev.org/#/c/713585/ *Incomplete* https://bugs.launchpad.net/neutron/+bug/1868098 - Neutron openvswitch agent error - INCOMPLETE But would be good if someone can take a look *Stadium projects* https://bugs.launchpad.net/neutron/+bug/1868515 - vpnaas ike policy is not allowed to be updated No assignment yet. Needs someone from vpnaas team to take a look. -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][privsep] Migrate from oslo.rootwrap to oslo.privsep
by Rodolfo Alonso 22 Mar '20

22 Mar '20

Hello all: With this mail I would like to propose the goal to move away from oslo.rootwrap and migrate to oslo.privsep. The last one offers a superior security model, faster and more secure. During the last cycles and since privsep was released, the Community has encouraged the usage of privsep and the deprecation of any existing code still using rootwrap. For any developer willing to collaborate, there are plenty of code examples, as I’ll provide later, implementing and using privsep for new methods and migrations. If this goal is approved, I'll open a Story (https://storyboard.openstack.org/) and any developer will be able to add a task for each patch or set of them related. This would be the tracker for this common effort. PROJECTS TO MIGRATE. -------------------- Projects that are still using rootwrap: http://codesearch.openstack.org/?q=rootwrap&i=nope&files=.*.py&repos= neutron os-brick designate cinder ironic-inspector neutron-vpnaas nova solum glance_store ironic zun magnum manila networking-bagpipe sahara ceilometer cinderlib freezer ironic-lib monasca-agent tacker tripleo-common USAGE DOCUMENTATION ABOUT PRIVSEP. ---------------------------------- How to create a privsep context, assign privileges and use it as a decorator: https://docs.openstack.org/oslo.privsep/latest/user/index.html HOW TO MIGRATE FROM ROOTWRAP TO PRIVSEP. ---------------------------------------- >From the same link provided previously, in the section “Converting from rootwrap to privsep”: https://docs.openstack.org/oslo.privsep/latest/user/index.html#converting-f… oslo.privsep provides a class, PrivContext, that can be used to create a decorator function. The instance created is a context of execution and has defined a list of capabilities, matching the Linux capabilities. The privsep context decorator should contain the minimum needed capabilities to execute the decorated function. For example: default = priv_context.PrivContext( __name__, cfg_section='privsep', pypath=__name__ + '.default', capabilities=[caps.CAP_SYS_ADMIN, caps.CAP_NET_ADMIN, caps.CAP_DAC_OVERRIDE, caps.CAP_DAC_READ_SEARCH, caps.CAP_SYS_PTRACE], ) The function “entrypoint” of this instance can be used as a decorator for another function: @privileged.default.entrypoint def delete_interface(ifname, namespace, **kwargs): _run_iproute_link("del", ifname, namespace, **kwargs) As commented in the given link, a straight 1:1 filter:function replacement generally results in functions that are still too broad for good security. It is better to replace each chmod rootwrap call with a narrow privsep function that will limit it to specific files. MIGRATION EXAMPLES. ------------------- Nova: https://review.opendev.org/#/q/project:openstack/nova+branch:master+topic:m… Neutron: https://review.opendev.org/#/q/status:merged+project:openstack/neutron+bran… os-vif: https://review.opendev.org/#/c/287725/ Thank you and regards.

6 7

[qeeens][neutron] migrating from iptables_hybrid to openvswitch
by Ignazio Cassano 22 Mar '20

22 Mar '20

Hello All, I am facing some problems migrating from iptables_hybrid frirewall to openvswitch firewall on centos 7 queens, I am doing this because I want enable security groups logs which require openvswitch firewall. I would like to migrate without restarting my instances. I startded moving all instances from compute node 1. Then I configured openvswitch firewall on compute node 1, Instances migrated from compute node 2 to compute node 1 without problems. Once the compute node 2 was empty, I migrated it to openvswitch. But now instances does not migrate from node 1 to node 2 because it requires the presence of qbr bridge on node 2 This happened because migrating instances from node2 with iptables_hybrid to compute node 1 with openvswitch, does not put the tap under br-int as requested by openvswich firewall, but qbr is still present on compute node 1. Once I enabled openvswitch on compute node 2, migration from compute node 1 fails because it exprects qbr on compute node 2 . So I think I should moving on the fly tap interfaces from qbr to br-int on compute node 1 before migrating to compute node 2 but it is a huge work on a lot of instances. Any workaround, please ? Ignazio

5 31

Diskimage-builder complex disk setup
by Stephen Nemeth 21 Mar '20

21 Mar '20

Hi all, We’re attempting to create user images to deploy with metal^3/ironic. I’m making almost no headway starting from any of the examples provided for the DIB_BLOCK_DEVICE_CONFIG setup. Is there a repository with practical values located anywhere? I’m finding the documentation insufficient to do anything more complicated than a single partition setup. Here’s what I’ve gotten so far. Help greatly appreciated: ``` [ { "local_loop": { "name": "image0", "size": "20GiB" } }, { "partitioning": { "base": "image0", "label": "gpt", "partitions": [ { "name": "ESP", "type": "EF00", "size": "16MiB" }, { "name": "boot", "type": "EF02", "size": "6GiB" }, { "name": "lvm", "type": 8, "size": "100%" } ] } }, { "lvm": { "name": "lvm2", "pvs": [ { "name": "pv", "options": [ "--force" ], "device": "lvm", "base": "image0" } ], "vgs": [ { "name": "vg", "base": [ "pv" ], "options": [ "--force" ] } ], "lvs": [ { "name": "root", "base": "vg", "extents": "100%FREE", "options": [ "--zero=n" ] } ] } }, { "mkfs": { { "name": "mkfs_root", "base": "root", "type": "btrfs", "label": "root", "opts": "-f", "mount": { "mount_point": "/" } }, { "name": "mkfs_efi", "base": "ESP", "type": "vfat", "label": "efi", "opts": "-f", "mount": { "mount_point": "/boot/efi/" } }, { "name": "mkfs_boot", "base": "boot", "type": "btrfs", "label": "boot", "opts": "-f", "mount": { "mount_point": "/boot/" } } } } ] ```

2 1

[tripleo] launchpad bug policy
by Wesley Hayutin 20 Mar '20

20 Mar '20

6 8

[neutron] How to fix break of connectivity in case of L3 HA after reboot of backup node
by Slawek Kaplonski 20 Mar '20

20 Mar '20

Hi, We have bug [1] to solve. Basically, when node which is backup node for some router, connectivity to external gateway may be broken for some time. It's like that because when host is up and L3 agent is configuring qrouter namespace, it flush all IPv6 addresses from the qg- interface. And due to that some MLDv2 packets are sent from this interface which may cause that ToR switch will learn qg port's mac address on wrong (backup) node. This don't happens every time and for every router because it is a race between L3 agent and OVS agent. When L3 agent creates qg interface in br-int, it sets tag 4095 for it and traffic sent with such vlan tag is always dropped in br-int. So if L3 agent will flush IPv6 addresses before OVS agent wires the port and sets correct tag for it, then all is fine. But if OVS agent is first, then MLDv2 packets are sent to the wire and there is this connectivity break. There are proposed 2 ways of fixing this: - [2] which propsoes to add some kind of "communication" between L3 agent and OVS agent and tell OVS agent that tag can be changed only after IPv6 config is finished by L3 agent. Downside of this solution is that it works for OVS agent only, Linuxbridge agent may still hit the same issue. But plus is that after initial configuration of the router, everything else regarding to failover is handled by keepalived only - in same way like it is now. - [3] which sets qg NIC to be DOWN always on backup nodes. So when keepalived failover active router to new node, L3 agent needs to come and switch interfaces to be UP before it will work. The plus of this solution is that it works for all OVS and Linuxbridge L2 agents (and probably for others too) but downside is that failover process is a bit longer and there may be potentially another race condition between L3 agent and keepalived. Keepalived tries to sent gARP packets after switch node to be active, first attempt will always fail as interface is still DOWN. But keepalived will retry those gARPs after some time and this should be fine if L3 agent will already bring interface to be UP. Both patches are waiting for pretty long time in gerrit and I want to bring more visibility for both of them. Please check them and maybe You will have some opinions about which solution would be better and which we should go with. [1] https://bugs.launchpad.net/neutron/+bug/1859832 [2] https://review.opendev.org/#/c/702856/ [3] https://review.opendev.org/#/c/707406/ -- Slawek Kaplonski Senior software engineer Red Hat

5 4

Neutron RabbitMQ issues
by Grant Morley 20 Mar '20

20 Mar '20

Hi all, We are currently experiencing some fairly major issues with our OpenStack cluster. It all appears to be with Neutron and RabbitMQ. We are seeing a lot of time out messages in responses to replies and because of this instance creation or anything to do with instances and networking is broken. We are running OpenStack Queens. We have already tuned Rabbit for Neutron by doing the following on neutron: heartbeat_timeout_threshold = 0 rpc_conn_pool_size = 300 rpc_thread_pool_size = 2048 rpc_response_timeout = 3600 rpc_poll_timeout = 60 ## Rpc all executor_thread_pool_size = 64 rpc_response_timeout = 3600 What we are seeing in the error logs for neutron for all services (l3-agent, dhcp, linux-bridge etc ) are these timeouts: https://pastebin.com/Fjh23A5a We have manually tried to get everything in sync by forcing fail-over of the networking which seems to get routers in sync. We are also seeing that there are a lot of "unacknowledged" messages in RabbitMQ for 'q-plugin' in the neutron queues. Some times restarting of the services on neutron gets these back acknowledged again, however the timeouts come back. The RabbitMQ servers themselves are not loaded at all. All memory, file descriptors and errlang processes have plenty of resources available. We are also seeing a lot of rpc issues: Timeout in RPC method release_dhcp_port. Waiting for 1523 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 965fa44ab4f6462fa378a1cf7259aad4 2020-03-10 19:02:33.548 16242 ERROR neutron.common.rpc [req-a858afbb-5083-4e21-a309-6ee53582c4d9 - - - - -] Timeout in RPC method release_dhcp_port. Waiting for 3347 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 7937465f15634fbfa443fe1758a12a9c Does anyone know if there is anymore tuning to be done at all? Upgrading for us at the moment to a newer version isn't really an option unfortunately. Because of our setup, we also have roughly 800 routers enabled and I know that will be putting a load on the system. However these problems have only started to happen roughly 1 week ago and have steadily got worse. If anyone has any use cases for this or any more recommendations that would be great. Many thanks,

3 25

[release] Release countdown for week R-7, March 23-27
by Sean McGinnis 20 Mar '20

20 Mar '20

Development Focus ----------------- We are entering the last weeks of the Ussuri development cycle. From now until the final release, we'll send a countdown email like this every week. It's probably a good time for teams to take stock of their library and client work that needs to be completed yet. The non-client library freeze is coming up, followed closely by the client lib freeze. Please plan accordingly so avoid any last minute rushes to get key functionality in. General Information ------------------- Next week is the Extra-ATC freeze, in preparation for elections. All contributions to OpenStack are valuable, but some are not expressed as Gerrit code changes. Please list active contributors to your project team who do not have a code contribution this cycle, and therefore won't automatically be considered an Active Technical Contributor and allowed to vote. This is done by adding extra-atcs to https://opendev.org/openstack/governance/src/branch/master/reference/projec… before the Extra-ATC freeze on March 26. A quick reminder of the upcoming freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (April 2). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (April 9) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, April 9. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (April 23) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (May 7) Finally, now is also a good time to start planning what highlights you want for your deliverables in the cycle highlights. The deadline to submit an initial version for those is set to Feature freeze (April 9). Background on cycle-highlights: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125613.html Project Team Guide, Cycle-Highlights: https://docs.openstack.org/project-team-guide/release-management.html#cycle… knelson [at] openstack.org/diablo_rojo on IRC is available if you need help selecting or writing your highlights Note that cycle highlights are per-team, not per-deliverable. Upcoming Deadlines & Dates -------------------------- Extra-ATC freeze: March 26 (R-7 week) Non-client library freeze: April 2 (R-6 week) Client library freeze: April 9 (R-5 week) Ussuri-3 milestone: April 9 (R-5 week) Cycle Highlights Due: April 9 (R-5 week)

1 0