March 2020 - openstack-discuss - lists.openstack.org

Re: [openstack-dev] Are we ready to put stable/ocata into extended maintenance mode?
by Alex Schultz 03 Mar '21

03 Mar '21

On Tue, Sep 18, 2018 at 1:30 PM Alex Schultz <aschultz(a)redhat.com> wrote: > > On Tue, Sep 18, 2018 at 1:27 PM, Matt Riedemann <mriedemos(a)gmail.com> wrote: > > The release page says Ocata is planned to go into extended maintenance mode > > on Aug 27 [1]. There really isn't much to this except it means we don't do > > releases for Ocata anymore [2]. There is a caveat that project teams that do > > not wish to maintain stable/ocata after this point can immediately end of > > life the branch for their project [3]. We can still run CI using tags, e.g. > > if keystone goes ocata-eol, devstack on stable/ocata can still continue to > > install from stable/ocata for nova and the ocata-eol tag for keystone. > > Having said that, if there is no undue burden on the project team keeping > > the lights on for stable/ocata, I would recommend not tagging the > > stable/ocata branch end of life at this point. > > > > So, questions that need answering are: > > > > 1. Should we cut a final release for projects with stable/ocata branches > > before going into extended maintenance mode? I tend to think "yes" to flush > > the queue of backports. In fact, [3] doesn't mention it, but the resolution > > said we'd tag the branch [4] to indicate it has entered the EM phase. > > > > 2. Are there any projects that would want to skip EM and go directly to EOL > > (yes this feels like a Monopoly question)? > > > > I believe TripleO would like to EOL instead of EM for Ocata as > indicated by the thead > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.ht… > Bringing this backup to see what we need to do to get the stable/ocata branches ended for the TripleO projects. I'm bringing this up because we have https://review.openstack.org/#/c/647009/ which is for the upcoming rename but CI is broken and we have no interest in continue to keep the stable/ocata branches alive (or fix ci for them). Thanks, -Alex > Thanks, > -Alex > > > [1] https://releases.openstack.org/ > > [2] > > https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… > > [3] > > https://docs.openstack.org/project-team-guide/stable-branches.html#extended… > > [4] > > https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… > > > > -- > > > > Thanks, > > > > Matt > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 11

[tc][uc][all] Starting community-wide goals ideas for V series
by Ghanshyam Mann 15 Jun '20

15 Jun '20

Hello everyone, We are in R14 week of Ussuri cycle which means It's time to start the discussions about community-wide goals ideas for the V series. Community-wide goals are important in term of solving and improving a technical area across OpenStack as a whole. It has lot more benefits to be considered from users as well from a developers perspective. See [1] for more details about community-wide goals and process. We have the Zuulv3 migration goal already accepted and pre-selected for v cycle. If you are interested in proposing a goal, please write down the idea on this etherpad[2] - https://etherpad.openstack.org/p/YVR-v-series-goals Accordingly, we will start the separate ML discussion over each goal idea. Also, you can refer to the backlogs of community-wide goals from this[3] and ussuri cycle goals[4]. NOTE: TC has defined the goal process schedule[5] to streamlined the process and ready with goals for projects to plan/implement at the start of the cycle. I am hoping to start that schedule for W cycle goals. [1] https://governance.openstack.org/tc/goals/index.html [2] https://etherpad.openstack.org/p/YVR-v-series-goals [3] https://etherpad.openstack.org/p/community-goals [4] https://etherpad.openstack.org/p/PVG-u-series-goals [5] https://governance.openstack.org/tc/goals/#goal-selection-schedule -gmann

10 22

[tripleo][operators] Removal of mistral from the TripleO Undercloud
by Kevin Carter 26 May '20

26 May '20

Hello stackers, In the pursuit to remove Mistral from the TripleO undercloud, we've discovered an old capability that we need to figure out how best to handle. Currently, we provide the ability for an end-user (operator / deployer) to pass in "N" Mistral workflows as part of a given deployment plan which is processed by python-tripleoclient at runtime [0][1]. From what we have documented, and what we can find within the code-base, we're not using this feature by default. That said, we do not remove something if it is valuable in the field without an adequate replacement. The ability to run arbitrary Mistral workflows at deployment time was first created in 2017 [2] and while present all this time, its documented [3] and intra-code-base uses are still limited to samples [4]. As it stands now, we're on track to making Mistral inert this cycle and if our progress holds over the next couple of weeks the capability to run arbitrary Mistral workflows will be the only thing left within our codebase that relies on Mistral running on the Undercloud. So the question is what do we do with functionality. Do we remove this ability out right, do we convert the example workflow [5] into a stand-alone Ansible playbook and change the workflow runner to an arbitrary playbook runner, or do we simply leave everything as-is and deprecate it to be removed within the next two releases? Although it would be perfectly acceptable to keep Mistral around a little while longer, it would also be a bummer if this one capability was the only reason we were not able to remove Mistral from the Undercloud. Any and all feedback from operators, deployers, developers, former developers, etc would be greatly appreciated. Thank you in advance. [0] https://opendev.org/openstack/python-tripleoclient/src/commit/af719b795bc9b… [1] https://opendev.org/openstack/python-tripleoclient/src/commit/af719b795bc9b… [2] https://review.opendev.org/#/c/457874 [3] https://specs.openstack.org/openstack/tripleo-specs/specs/pike/tripleo-deri… [4] https://opendev.org/openstack/tripleo-heat-templates/src/commit/48da6a13936… [5] https://opendev.org/openstack/tripleo-common/src/commit/cc4f9c57b708259e9e9… Kevin Carter IRC: kecarter

5 9

[swift] Rolling upgrade, any version relationships?
by Mark Kirkwood 24 May '20

24 May '20

Hi, we are looking at upgrading our 2.7.0 Swift cluster. In the past I've modeled this on a dev system by upgrading storage nodes one by one (using 2.17 as the target version). This seemed to work well - I deliberately left the cluster half upgraded for an extended period to test for any cross version weirdness (didn't see any). However I'm wanting to check that I have not missed something important. So my questions are: - If upgrading from 2.7.0 is it safe to just grab the latest version (e.g 2.23)? - If not is there a preferred version to jump to first? - Is it ok for the upgrade to take extended time (e.g weeks) and therefore be running with some new and some old storage nodes for that time? regards Mark

3 5

[swift] Erasure code hardware offloading
by Mark Kirkwood 22 May '20

22 May '20

Hi, Can Swift make use of an Erasure Code offload card (e.g https://community.mellanox.com/s/article/understanding-erasure-coding-offlo… regards Mark

2 1

[swift] Adding disks - one by one or all lightly weighted?
by Mark Kirkwood 14 May '20

14 May '20

Hi, We are wanting to increase the number of disks in each of our storage nodes - from 4 to 12. I'm wondering whether it is better to: 1/ Add 1st new disk (with a reduced weight)...increase the weight until full, then repeat for next disk etc 2/ Add 'em all with a (much i.e 1/8 of that in 1/ ) reduced weight...increase the weights until done Thoughts? regards Mark

2 2

[oslo][nova][vmware] Replacement for suds library
by Stephen Finucane 14 May '20

14 May '20

The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests. /nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts) These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work? Stephen [1] https://docs.python.org/3.9/whatsnew/3.9.html#you-should-check-for-deprecat… [2] https://www.python.org/dev/peps/pep-0596/ [3] https://bitbucket.org/jurko/suds/src/default/ [4] https://github.com/suds-community/suds/pull/32 [5] https://python-zeep.readthedocs.io/en/master/

5 6

[Tempest][Restack]Call for help on 2020:06 Certificaion
by prakash RAMCHANDRAN 08 May '20

08 May '20

Hi all, Appreciate all your inputs in mail and one-on-one conversations I had with few of you. Like to set proper expectations before we deal with extending our goals. First we must deliver on continuity of Refstack, while we evaluate the value it adds to end users through survey. I. Every Release - There is a new criterion for Certification - Do we have one for 2020.06? - Who has volunteered or would like to volunteer for this effort? II. Can we have previous WG Chair or anyone can brief us (by call or by email ) as what the process was and how it was managed. |||. Who were last release OSF Reviewers? Can they volunteer for 2020:06 release definition for Refstack - https://opendev.org/openstack/refstack/ VI. Given the circumstances (CoViD-19) will it be possible to deliver Refstack in June /July 2020? (Need some estimates from TC or others who managed this in past years) V. Is Zun - ?, Magnum -? Are both OCI and/or CNI compliant? - Can they be tested using Tempest and add those two or any other container projects to refstack core? For Survey (suggest any specifics Questions to ask- may be few questions to - Customer / Vendors/ new feedback Q1-Value of Certification , Q2. How Often customers ask for it in RFP or sales cycle?, Q3. What additional Tests, process, frequency etc will add value for them?Q4. Do you think we should continue the efforts of Refstack?) Let me know through email and have added this to TC meeting for Thursday April 2.https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting ThanksPrakash Sent from Yahoo Mail on Android

2 1

[Syntribos] Does anyone still use this?
by Gage Hugo 07 May '20

07 May '20

The Security SIG has recently been looking into updating several sites/documentation and one part we discussed the last couple meetings was the security tools listings[0]. One of the projects listed there, Syntribos, doesn't appear to have much activity[1] outside of zuul/python/docs maintenance and the idea of retiring the project was mentioned. However, there does seem to be an occasional bug-fix submitted, so we are sending this email out to see if anyone is still utilizing Syntribos. If so, please either respond here, reach out to us in the #openstack-security irc channel, or fill out the section in the Security SIG Agenda etherpad[2]. Thanks! [0] https://security.openstack.org/#security-tool-development [1] https://review.opendev.org/#/q/project:openstack/syntribos [2] https://etherpad.openstack.org/p/security-agenda

2 2

[openstack-ansible] dropping suse support
by Mohammed Naser 05 May '20

05 May '20

Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct… [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct…

6 9

[all] A call for consolidation and simplification
by Thierry Carrez 01 May '20

01 May '20

Hi all, I'd like to issue a call for consolidation and simplification for OpenStack development. In the early years of the project, we faced a lot of challenges. We had to spread the development load across manageable-size groups, so we encouraged the creation of a lot of project teams. We wanted to capture all the energy that was sent towards the project, so we passed project structure reforms (like the big tent) that would aggressively include new community groups in the "official" OpenStack community. We needed to remove bottlenecks, so we encouraged decentralized decision making. And we had to answer unique challenges, so we created software to match them (Zuul). In summary, we had a lot of people, and not enough systems to organize them, so we created those. Fast-forward to 2020, and our challenges are different. The many systems that we created in the early days have created silos, with very small groups of people working in isolation, making cross-project work more difficult than it should be. The many systems that we created generate a lot of fragmentation. Like we have too many meetings (76, in case you were wondering), too much energy spent running them, too much frustration when nobody joins. Finally, the many systems that we created represent a lot of complexity for newcomers to handle. We have 180 IRC channels, most of them ghost towns where by the time someone answers, the person asking the question is long gone. So I think it's time to generally think about simplifying and consolidating things. It's not as easy as it sounds. Our successful decentralization efforts make it difficult to make the centralized decision to regroup. It's hard to justify time and energy spent to /remove/ things, especially those that we spent time creating in the first place. But we now have too many systems and not enough people, so we need to consolidate and simplify. Back around Havana, when we had around the same number of active contributors as today, we used to have 36 meetings and 20 teams. Do we really need 180 IRC channels, 76 meetings, 63 project teams (not even counting SIGs)? Yes, we all specialized over time, so it's hard to merge for example Oslo + Requirements, or QA + Infrastructure, or Stable + Release Management, or Monasca + Telemetry. We are all overextended so it's hard to learn new tricks or codebases. And yet, while I'm not really sure what the best approach is, I think it's necessary. Comments, thoughts? -- Thierry Carrez (ttx)

9 16

[manila][ptg] Victoria Cycle PTG in Vancouver - Planning
by Goutham Pacha Ravi 30 Apr '20

30 Apr '20

Hello Zorillas and other interested Stackers, We've requested space to gather at the Open Infrastructure PTG event in Vancouver, BC between June 8th-11th 2020. This event gives us the opportunity to catch up in person (and via telepresence) and discuss the next set of goals for manila and its ecosystem of projects. As has been the norm, there's now a planning etherpad for us to collect topics and vote upon [1]. Depending on how much space/time we are budgeted, we will select and schedule these topics. Please take a look and add your topics to the etherpad. Thanks, Goutham [1] https://etherpad.openstack.org/p/vancouver-ptg-manila-planning

1 1

Virtual OpenDev Update
by Ashlee Ferguson 22 Apr '20

22 Apr '20

Hi everyone, As we continue to gather feedback and discuss what a virtual OpenDev will look like, we wanted to provide you with an update around the event. While we are still attempting to solve some of the riddles of virtual event world, we did come up with some decisions that give us something to build on, which are based on input and experience from members of the Programming Committee. Decisions: - OpenDev will happen after the Virtual PTG (exact format and timing TBD) - Will not be simultaneous/parallel Tracks so everyone has the opportunity to participate in more than one - 3 Tracks, each on different date: - Hardware Automation - Large Scale Ops - Containers in Production We also discussed the original intent and goals of OpenDev to make sure we’re still heading in a direction that’s beneficial for the community, especially since it will no longer directly precede the upcoming PTG. Similar to past OpenDev events, the goal is to identify the questions we don’t have the answers for, and to use that time to dive in and determine the work ahead of us, working together, sharing use cases, and learning from other people asking those same questions. High-level decisions, goals, and some areas we’re looking for feedback are in the etherpad below. As you continue to experience this new amazing world of virtual events, please add any feedback or ideas to the etherpad below. We’re looking forward to creating virtual OpenDev with you all! https://etherpad.openstack.org/p/Virtual_OpenDev_Planning <https://etherpad.openstack.org/p/Virtual_OpenDev_Planning> Ashlee Ashlee Ferguson Community & Events Coordinator OpenStack Foundation

1 2

Re: [nova][cinder][ops] question/confirmation of legacy vol attachment migration
by Brett Milford 22 Apr '20

22 Apr '20

On Tue, Dec 10, 2019 at 11:15 PM Gorka Eguileor <geguileo(a)redhat.com> wrote: > > On 06/12, Brett Milford wrote: > > On Fri, Nov 22, 2019 at 3:54 AM Matt Riedemann <mriedemos(a)gmail.com> wrote: > > > > > > On 10/17/2019 5:24 AM, Gorka Eguileor wrote: > > > > I stand by my initial recommendation, being able to update the existing > > > > attachment to add the connection information from Nova. > > > > > > OK, thanks for the input and thoughtfulness on this. I've abandoned my > > > change since I'm not going to be pushing this boulder anymore but left > > > notes in the change in case someone else wants to pick it up some day. > > > > > > Note to nova cores: this means we'll have legacy volume attachment > > > compat code around forever. > > > > > > -- > > > > > > Thanks, > > > > > > Matt > > > > > > > Hi Groka, Cinder & Nova devs, > > > > Following up this thread from the context of > > https://review.opendev.org/#/c/579004/ > > > > To summarise the discussion: > > - initialize_connection shouldn't be called more than once, as it > > may mess up some drivers. > > - To (safely) refresh connection_info a new api for cinder is required. > > Hi, > > It may be a new API or a modification of one of the existing ones. > > > > - a patch to nova, such as #579004 could make a call to this new api > > to refresh connection info on events such as reboot. > > > > Is there any other context to this issue I've missed or an alternate > > path to solving this? > > > > Does the creation of a new api for cinder have any implications for > > being able to backport this patch for nova? > > > > I don't think backporting it in Nova would do us much good, since the > Cinder code would not be backportable (it's a new feature, not a bug > fix). > > > > What would be the process for me to kick off this work? > > > > You should propose a Cinder spec [1] for your work and you may provide a > WIP patch to Cinder at the same time, though depending on the review you > may have to completely change that code later. > > Cheers, > Gorka. > > [1]: https://opendev.org/openstack/cinder-specs > > > Thanks for your help, > > Brett (bcm) > > -- > > ❯ brett > > > Cheers Matt, Groka, I thought this might be the case. -- ❯ brett

2 2

[election][nova] PTL Candidacy for Victoria
by Balázs Gibizer 18 Apr '20

18 Apr '20

Hi, I'd like to announce my candidacy for Nova PTL for Train. I'm a long time Nova contributor and Nova core since 2017. I'm currently serving as temporary Nova PTL since we lost Eric during the Ussuri cycle. We are facing multiple challenges for Victoria. We have to organize the first fully virtual PTG to plan the Victoria cycle due to sever travel restrictions around the word. I have already proposed ways [1] to the Nova team and I will continue working on making the best out of the current situation. In the past months we have lost Matt and Eric, two key contributors and Nova maintainers. So I will work on regrowing the Nova core team to restore our review bandwidth. As a first step I proposed two new Nova core members [2][3]. But I will also look for more possible candidates and I will try to mentor them to further increase the size of the core team. We have many unfinished business from the Ussuri cycle. So as a PTL I have the intention to make the team focus on finishing these work before we start any new big change in Nova. Thanks, Balazs Gibizer (gibi) [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013555.ht… [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013511.ht… [3] http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013519.ht…

3 2

OpenStack should be buildable with sphinx 2.4: FTBFS in many packages
by Thomas Goirand 16 Apr '20

16 Apr '20

Hi, Sphinx 2.4 has been uploaded to Experimental. I just received bug reports against the OpenStack packages because some of them cannot be built with Sphinx 2.4. Here's the list: - ironic - os-api-ref - aodh - cloudkitty - kombu - panko - ceilometerclient - ceilometermiddleware - doc8 - dracclient - glareclient - murano-pkg-check - os-xenapi - scciclient - sahara - sphinxcontrib-programoutput - tempest-horizon It'd be nice if those affected could be fixed for Ussuri. IMO, it's just the doc, so fixing it wouldn't hurt code quality. In other words: help would be more than welcome writing patch or reviewing them, and testing Sphinx 2.4 compatibility. Cheers, Thomas Goirand (zigo)

7 13

[cinder][ops][extended-maintenance-sig][public-cloud-sig][enterprise-wg] Cinder to EOL some branches
by Brian Rosmaita 13 Apr '20

13 Apr '20

This is a courtesy notice that having received no responses to my email of 28 October [0] proposing to EOL some currently open Cinder branches, and following the policy articulated in [1], at today's Virtual PTG meeting the Cinder project team has decided to put the following stable branches into the End of Life state: driverfixes/mitaka driverfixes/newton stable/ocata stable/pike I will submit the paperwork to get this process moving one week from today (2 December 2019). cheers, brian [0] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010385.… [1] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l…

4 8

[octavia] dropping support for stable/queens
by Adam Harwell 10 Apr '20

10 Apr '20

I'm officially proposing dropping support for the queens stable branch of Octavia. Given a quorum of cores, we will move forward with marking it EOL and all that entails. Thanks, --Adam

5 5

[all][PTL][release] Call for Ussuri Cycle Highlights
by Kendall Nelson 08 Apr '20

08 Apr '20

Hello Everyone! Its time to start thinking about calling out 'cycle-highlights' in your deliverables! As PTLs, you probably get many pings towards the end of every release cycle by various parties (marketing, management, journalists, etc) asking for highlights of what is new and what significant changes are coming in the new release. By putting them all in the same place it makes them easy to reference because they get compiled into a pretty website like this from the last few releases: Rocky[1], Stein[2], Train[3]. We don't need a fully fledged marketing message, just a few highlights (3-4 ideally), from each project team. Looking through your release notes might be a good place to start. *The deadline for cycle highlights is the end of the R-5 week [4] on April 10th.* How To Reminder: ------------------------- Simply add them to the deliverables/train/$PROJECT.yaml in the openstack/releases repo like this: cycle-highlights: - Introduced new service to use unused host to mine bitcoin. The formatting options for this tag are the same as what you are probably used to with Reno release notes. Also, you can check on the formatting of the output by either running locally: tox -e docs And then checking the resulting doc/build/html/train/highlights.html file or the output of the build-openstack-sphinx-docs job under html/train/ highlights.html. Can't wait to see you all have accomplished this release! Thanks :) -Kendall Nelson (diablo_rojo) [1] https://releases.openstack.org/rocky/highlights.html [2] https://releases.openstack.org/stein/highlights.html [3] https://releases.openstack.org/train/highlights.html [4] htt <https://releases.openstack.org/train/schedule.html#t-cycle-highlights> https://releases.openstack.org/ussuri/schedule.html

2 5

[nova][gate] status of some gate bugs
by melanie witt 07 Apr '20

07 Apr '20

Hey all, We've been having a tough time lately in the gate hitting various bugs while our patches go through CI. I just wanted to mention a few of them that I've seen often in my gerrit notifications and give a brief status on fix efforts. * http://status.openstack.org/elastic-recheck/#1813789 This one is where the nova-live-migration job fails a server evacuate test with: "Timeout waiting for [('network-vif-plugged', 'e3d3db3f-bce4-4889-b161-4b73648f79be')] for instance with vm_state error and task_state rebuild_spawning.: eventlet.timeout.Timeout: 300 seconds" in the screen-n-cpu.txt log. lyarwood has a WIP patch here: https://review.opendev.org/713674 and sean-k-mooney has a WIP patch here: https://review.opendev.org/713342 * https://launchpad.net/bugs/1867380 This one is where the nova-live-migration or nova-grenade-multinode job fail due to n-cpu restarting slowly after being reconfigured for ceph. The server will fail to build and it's because the test begins before nova-compute has fully come up and we see this error: "Instance spawn was interrupted before instance_claim, setting instance to ERROR state {{(pid=3783) _error_out_instances_whose_build_was_interrupted" in the screen-n-cpu.txt log. lyarwood has a patch approved here that we've been rechecking the heck out of that has yet to merge: https://review.opendev.org/713035 * https://launchpad.net/bugs/1844568 This one is where a job fails with: "Body: b'{"conflictingRequest": {"code": 409, "message": "Multiple possible networks found, use a Network ID to be more specific."}}'" gmann has a patch proposed to fix some of these here: https://review.opendev.org/711049 There might be more test classes that need create_default_network = True. * http://status.openstack.org/elastic-recheck/#1844929 This one is where a job fails and the following error is seen one of the logs, usually screen-n-sch.txt: "Timed out waiting for response from cell 8acfb79b-2e40-4e1c-bc3d-d404dac6db90". The TL;DR on this one is there's no immediate clue why it's happening. This bug used to hit more occasionally on "slow" nodes like nodes from the OVH or INAP providers (and OVH restricts disk iops [1]). Now, it seems like it's hitting much more often (still mostly on OVH nodes). I've been looking at it for about a week now and I've been using a DNM patch to add debug logging, look at dstat --disk-wait output, try mysqld my.cnf settings, etc: https://review.opendev.org/701478 So far, what I find is that when we get into the fail state, we get no rows back from the database server when we query for nova 'services' and 'compute_nodes' records, and we fail with the "Timed out waiting for response" error. Haven't figured out why yet, so far. The disk wait doesn't look high when this happens (or at any time during a run) so it's not seeming like it's related to disk IO. I'm continuing to look into it. Cheers, -melanie [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505…

5 8

[k8s][zun] Introduce a new feature in Ussuri - CRI integration
by Hongbin Lu 07 Apr '20

07 Apr '20

Hi all, As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1]. As known by people, Zun is an OpenStack Container service. It provides API for users to create and manage application containers in an OpenStack cloud. The main concepts in Zun are "Container" and "Capsule". A container is a single container, while a capsule is a group of co-located and co-scheduled containers (basically the same as k8s pod). Probably, the "Container" concept is more widely used. People can use the /containers API endpoint to create and manage a single container. Under the hook, a container is a Docker container in a compute node. What is special is that each Docker container is given a Neutron port so the container is connected to a tenant network in Neutron. Kuryr-libnetwork is the Docker network plugin we use to perform the Neutron port binding which basically connects the container to the virtual switch managed by Neutron. As mentioned before, the concept of "Capsule" in Zun is basically the same as pod in k8s. We introduced this concept mainly for k8s integration. Roughly speaking, the Zun-k8s integration is achieved by (i) registering a special node in k8s, (ii) watching the k8s API for pods being scheduled to this node and (iii) invoking Zun's /capsules API endpoint to create a capsule for each incoming pod. The (i) and (ii) is done by a CNCF sandbox project called Virtual Kubelet [2]. The (iii) is achieved by providing an OpenStack provider [3] for Virtual Kubelet. The special node registered by Virtual Kubelet is called a virtual node because the node doesn't physically exist. Pods being scheduled to the virtual node is basically offloaded from the current k8s cluster, eventually landed on an external platform such as an OpenStack cloud. In high level, what is offered to end-users is a "serverless kubernetes pod" [4]. This term basically means the ability to run pods on demand without planing the capacity (i.e. nodes) upfront. An example of that is AWS EKS on Fargate [5]. In comparison, the traditional approach is to create an entire k8s cluster upfront in order to run the workload. Let's give a simple example. Suppose you want to run a pod, the traditional approach is to provision a k8s cluster with a worker node. Then, run the pod on the worker node. In contract, the "serverless" approach is to create a k8s cluster without any worker node and the pod is offloaded to a cloud provider that provisions the pods at runtime. This approach works well for applications who have fluctuated workloads so it is hard to provision a cluster with the right size for them. Furthermore, from cloud provider's perspective, if all tenant users offloads their pods to the cloud, the cloud provider might be able to pack the workload better (i.e. with fewer physical nodes) thus saving cost. Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun-cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below: +--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+ Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below: +----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+ In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun-cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun-cni-daemon will look up the Neutron port information from DB and perform the port binding. In conclusion, starting from Ussuri, Zun adds support for CRI-compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate. [1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime <https://github.com/virtual-kubelet/openstack-zun> [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernete… [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-av…

3 8

[all][TC][PTL][election] Nominations Close & Campaigning Begins
by Kendall Nelson 07 Apr '20

07 Apr '20

The PTL and TC Nomination periods are now over. The official candidate lists are available on the election website[0][1]. --PTL Election Details-- There are 16 projects without candidates, so according to this resolution[2], the TC will have to decide how the following projects will proceed: Adjutant Barbican Cloudkitty Congress I18n Infrastructure Loci Masakari Oslo Packaging_Rpm Placement Rally Swift Tacker Tricircle Zaqar There are no projects with more than one candidate so we won't need to hold any runoffs! Congratulations to our new and returning PTLs! [0] --TC Election Details-- The official candidate list is available on the election website[1]. There are will be a TC election following the campaigning period that has now begun and runs till Apr 07, 2020 23:45 UTC when the polling will begin. You are encouraged to ask questions to the candidates on the ML to help inform your decisions when polling begins. Thank you, -Kendall (diablo_rojo) & the Election Officials [0] https://governance.openstack.org/election/#victoria-ptl-candidates [1] https://governance.openstack.org/election/#victoria-tc-candidates [2] https://governance.openstack.org/resolutions/20141128-elections-process-for…

8 10

[neutron][OVN] Multiple mechanism drivers
by Sam Morrison 07 Apr '20

07 Apr '20

We are looking at using OVN and are having some issues with it in our ML2 environment. We currently have 2 mechanism drivers in use: linuxbridge and midonet and these work well (midonet is the default tenant network driver for when users create a network) Adding OVN as a third mechanism driver causes the linuxbridge and midonet networks to stop working in terms of CRUD operations etc. It looks as if the OVN driver thinks it’s the only player and is trying to do things on ports that are in linuxbridge or midonet networks. Am I missing something here? (We’re using Stein version) Thanks, Sam

5 6

[openstack-discuss] [puppet] Nominating Takashi Kajinami for core of the Puppet OpenStack modules
by Shengping Zhong 07 Apr '20

07 Apr '20

Hey Puppet Cores, I would like to nominate Takashi Kajinami as a Core reviewer for the Puppet OpenStack modules. He is an excellent contributor to our modules over the last several cycles. His stats for the last 90 days can be viewed here[0]. Please response with your +1 or any objections. If there are no objections by April 6 I will add him to the core list. As there were no objections, I have added Takashi to the core list. Keep up the good work. Thanks, Shengping. [0] https://www.stackalytics.com/report/contribution/puppet%20openstack-group/90

5 4

[qa] Proposing Martin Kopec to Tempest core
by Ghanshyam Mann 06 Apr '20

06 Apr '20

Hello Everyone, Martin Kopec (IRC: kopecmartin) has been doing great contribution in Tempest since a long time. He has been doing bugs Triage also in Ussuri cycle and has good understanding of Tempest. I would like to propose him for Tempest Core. You can vote/feedback on this email. If no objection by the end of this week, I will add him to the list. -gmann

9 9