March 2020 - openstack-discuss - lists.openstack.org

[all] A call for consolidation and simplification
by Thierry Carrez 01 May '20

01 May '20

Hi all, I'd like to issue a call for consolidation and simplification for OpenStack development. In the early years of the project, we faced a lot of challenges. We had to spread the development load across manageable-size groups, so we encouraged the creation of a lot of project teams. We wanted to capture all the energy that was sent towards the project, so we passed project structure reforms (like the big tent) that would aggressively include new community groups in the "official" OpenStack community. We needed to remove bottlenecks, so we encouraged decentralized decision making. And we had to answer unique challenges, so we created software to match them (Zuul). In summary, we had a lot of people, and not enough systems to organize them, so we created those. Fast-forward to 2020, and our challenges are different. The many systems that we created in the early days have created silos, with very small groups of people working in isolation, making cross-project work more difficult than it should be. The many systems that we created generate a lot of fragmentation. Like we have too many meetings (76, in case you were wondering), too much energy spent running them, too much frustration when nobody joins. Finally, the many systems that we created represent a lot of complexity for newcomers to handle. We have 180 IRC channels, most of them ghost towns where by the time someone answers, the person asking the question is long gone. So I think it's time to generally think about simplifying and consolidating things. It's not as easy as it sounds. Our successful decentralization efforts make it difficult to make the centralized decision to regroup. It's hard to justify time and energy spent to /remove/ things, especially those that we spent time creating in the first place. But we now have too many systems and not enough people, so we need to consolidate and simplify. Back around Havana, when we had around the same number of active contributors as today, we used to have 36 meetings and 20 teams. Do we really need 180 IRC channels, 76 meetings, 63 project teams (not even counting SIGs)? Yes, we all specialized over time, so it's hard to merge for example Oslo + Requirements, or QA + Infrastructure, or Stable + Release Management, or Monasca + Telemetry. We are all overextended so it's hard to learn new tricks or codebases. And yet, while I'm not really sure what the best approach is, I think it's necessary. Comments, thoughts? -- Thierry Carrez (ttx)

9 16

[manila][ptg] Victoria Cycle PTG in Vancouver - Planning
by Goutham Pacha Ravi 30 Apr '20

30 Apr '20

Hello Zorillas and other interested Stackers, We've requested space to gather at the Open Infrastructure PTG event in Vancouver, BC between June 8th-11th 2020. This event gives us the opportunity to catch up in person (and via telepresence) and discuss the next set of goals for manila and its ecosystem of projects. As has been the norm, there's now a planning etherpad for us to collect topics and vote upon [1]. Depending on how much space/time we are budgeted, we will select and schedule these topics. Please take a look and add your topics to the etherpad. Thanks, Goutham [1] https://etherpad.openstack.org/p/vancouver-ptg-manila-planning

1 1

Virtual OpenDev Update
by Ashlee Ferguson 22 Apr '20

22 Apr '20

Hi everyone, As we continue to gather feedback and discuss what a virtual OpenDev will look like, we wanted to provide you with an update around the event. While we are still attempting to solve some of the riddles of virtual event world, we did come up with some decisions that give us something to build on, which are based on input and experience from members of the Programming Committee. Decisions: - OpenDev will happen after the Virtual PTG (exact format and timing TBD) - Will not be simultaneous/parallel Tracks so everyone has the opportunity to participate in more than one - 3 Tracks, each on different date: - Hardware Automation - Large Scale Ops - Containers in Production We also discussed the original intent and goals of OpenDev to make sure we’re still heading in a direction that’s beneficial for the community, especially since it will no longer directly precede the upcoming PTG. Similar to past OpenDev events, the goal is to identify the questions we don’t have the answers for, and to use that time to dive in and determine the work ahead of us, working together, sharing use cases, and learning from other people asking those same questions. High-level decisions, goals, and some areas we’re looking for feedback are in the etherpad below. As you continue to experience this new amazing world of virtual events, please add any feedback or ideas to the etherpad below. We’re looking forward to creating virtual OpenDev with you all! https://etherpad.openstack.org/p/Virtual_OpenDev_Planning <https://etherpad.openstack.org/p/Virtual_OpenDev_Planning> Ashlee Ashlee Ferguson Community & Events Coordinator OpenStack Foundation

1 2

Re: [nova][cinder][ops] question/confirmation of legacy vol attachment migration
by Brett Milford 22 Apr '20

22 Apr '20

On Tue, Dec 10, 2019 at 11:15 PM Gorka Eguileor <geguileo(a)redhat.com> wrote: > > On 06/12, Brett Milford wrote: > > On Fri, Nov 22, 2019 at 3:54 AM Matt Riedemann <mriedemos(a)gmail.com> wrote: > > > > > > On 10/17/2019 5:24 AM, Gorka Eguileor wrote: > > > > I stand by my initial recommendation, being able to update the existing > > > > attachment to add the connection information from Nova. > > > > > > OK, thanks for the input and thoughtfulness on this. I've abandoned my > > > change since I'm not going to be pushing this boulder anymore but left > > > notes in the change in case someone else wants to pick it up some day. > > > > > > Note to nova cores: this means we'll have legacy volume attachment > > > compat code around forever. > > > > > > -- > > > > > > Thanks, > > > > > > Matt > > > > > > > Hi Groka, Cinder & Nova devs, > > > > Following up this thread from the context of > > https://review.opendev.org/#/c/579004/ > > > > To summarise the discussion: > > - initialize_connection shouldn't be called more than once, as it > > may mess up some drivers. > > - To (safely) refresh connection_info a new api for cinder is required. > > Hi, > > It may be a new API or a modification of one of the existing ones. > > > > - a patch to nova, such as #579004 could make a call to this new api > > to refresh connection info on events such as reboot. > > > > Is there any other context to this issue I've missed or an alternate > > path to solving this? > > > > Does the creation of a new api for cinder have any implications for > > being able to backport this patch for nova? > > > > I don't think backporting it in Nova would do us much good, since the > Cinder code would not be backportable (it's a new feature, not a bug > fix). > > > > What would be the process for me to kick off this work? > > > > You should propose a Cinder spec [1] for your work and you may provide a > WIP patch to Cinder at the same time, though depending on the review you > may have to completely change that code later. > > Cheers, > Gorka. > > [1]: https://opendev.org/openstack/cinder-specs > > > Thanks for your help, > > Brett (bcm) > > -- > > ❯ brett > > > Cheers Matt, Groka, I thought this might be the case. -- ❯ brett

2 2

[election][nova] PTL Candidacy for Victoria
by Balázs Gibizer 18 Apr '20

18 Apr '20

Hi, I'd like to announce my candidacy for Nova PTL for Train. I'm a long time Nova contributor and Nova core since 2017. I'm currently serving as temporary Nova PTL since we lost Eric during the Ussuri cycle. We are facing multiple challenges for Victoria. We have to organize the first fully virtual PTG to plan the Victoria cycle due to sever travel restrictions around the word. I have already proposed ways [1] to the Nova team and I will continue working on making the best out of the current situation. In the past months we have lost Matt and Eric, two key contributors and Nova maintainers. So I will work on regrowing the Nova core team to restore our review bandwidth. As a first step I proposed two new Nova core members [2][3]. But I will also look for more possible candidates and I will try to mentor them to further increase the size of the core team. We have many unfinished business from the Ussuri cycle. So as a PTL I have the intention to make the team focus on finishing these work before we start any new big change in Nova. Thanks, Balazs Gibizer (gibi) [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013555.ht… [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013511.ht… [3] http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013519.ht…

3 2

OpenStack should be buildable with sphinx 2.4: FTBFS in many packages
by Thomas Goirand 16 Apr '20

16 Apr '20

Hi, Sphinx 2.4 has been uploaded to Experimental. I just received bug reports against the OpenStack packages because some of them cannot be built with Sphinx 2.4. Here's the list: - ironic - os-api-ref - aodh - cloudkitty - kombu - panko - ceilometerclient - ceilometermiddleware - doc8 - dracclient - glareclient - murano-pkg-check - os-xenapi - scciclient - sahara - sphinxcontrib-programoutput - tempest-horizon It'd be nice if those affected could be fixed for Ussuri. IMO, it's just the doc, so fixing it wouldn't hurt code quality. In other words: help would be more than welcome writing patch or reviewing them, and testing Sphinx 2.4 compatibility. Cheers, Thomas Goirand (zigo)

7 13

[cinder][ops][extended-maintenance-sig][public-cloud-sig][enterprise-wg] Cinder to EOL some branches
by Brian Rosmaita 13 Apr '20

13 Apr '20

This is a courtesy notice that having received no responses to my email of 28 October [0] proposing to EOL some currently open Cinder branches, and following the policy articulated in [1], at today's Virtual PTG meeting the Cinder project team has decided to put the following stable branches into the End of Life state: driverfixes/mitaka driverfixes/newton stable/ocata stable/pike I will submit the paperwork to get this process moving one week from today (2 December 2019). cheers, brian [0] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010385.… [1] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l…

4 8

[octavia] dropping support for stable/queens
by Adam Harwell 10 Apr '20

10 Apr '20

I'm officially proposing dropping support for the queens stable branch of Octavia. Given a quorum of cores, we will move forward with marking it EOL and all that entails. Thanks, --Adam

5 5

[all][PTL][release] Call for Ussuri Cycle Highlights
by Kendall Nelson 08 Apr '20

08 Apr '20

Hello Everyone! Its time to start thinking about calling out 'cycle-highlights' in your deliverables! As PTLs, you probably get many pings towards the end of every release cycle by various parties (marketing, management, journalists, etc) asking for highlights of what is new and what significant changes are coming in the new release. By putting them all in the same place it makes them easy to reference because they get compiled into a pretty website like this from the last few releases: Rocky[1], Stein[2], Train[3]. We don't need a fully fledged marketing message, just a few highlights (3-4 ideally), from each project team. Looking through your release notes might be a good place to start. *The deadline for cycle highlights is the end of the R-5 week [4] on April 10th.* How To Reminder: ------------------------- Simply add them to the deliverables/train/$PROJECT.yaml in the openstack/releases repo like this: cycle-highlights: - Introduced new service to use unused host to mine bitcoin. The formatting options for this tag are the same as what you are probably used to with Reno release notes. Also, you can check on the formatting of the output by either running locally: tox -e docs And then checking the resulting doc/build/html/train/highlights.html file or the output of the build-openstack-sphinx-docs job under html/train/ highlights.html. Can't wait to see you all have accomplished this release! Thanks :) -Kendall Nelson (diablo_rojo) [1] https://releases.openstack.org/rocky/highlights.html [2] https://releases.openstack.org/stein/highlights.html [3] https://releases.openstack.org/train/highlights.html [4] htt <https://releases.openstack.org/train/schedule.html#t-cycle-highlights> https://releases.openstack.org/ussuri/schedule.html

2 5

[nova][gate] status of some gate bugs
by melanie witt 07 Apr '20

07 Apr '20

Hey all, We've been having a tough time lately in the gate hitting various bugs while our patches go through CI. I just wanted to mention a few of them that I've seen often in my gerrit notifications and give a brief status on fix efforts. * http://status.openstack.org/elastic-recheck/#1813789 This one is where the nova-live-migration job fails a server evacuate test with: "Timeout waiting for [('network-vif-plugged', 'e3d3db3f-bce4-4889-b161-4b73648f79be')] for instance with vm_state error and task_state rebuild_spawning.: eventlet.timeout.Timeout: 300 seconds" in the screen-n-cpu.txt log. lyarwood has a WIP patch here: https://review.opendev.org/713674 and sean-k-mooney has a WIP patch here: https://review.opendev.org/713342 * https://launchpad.net/bugs/1867380 This one is where the nova-live-migration or nova-grenade-multinode job fail due to n-cpu restarting slowly after being reconfigured for ceph. The server will fail to build and it's because the test begins before nova-compute has fully come up and we see this error: "Instance spawn was interrupted before instance_claim, setting instance to ERROR state {{(pid=3783) _error_out_instances_whose_build_was_interrupted" in the screen-n-cpu.txt log. lyarwood has a patch approved here that we've been rechecking the heck out of that has yet to merge: https://review.opendev.org/713035 * https://launchpad.net/bugs/1844568 This one is where a job fails with: "Body: b'{"conflictingRequest": {"code": 409, "message": "Multiple possible networks found, use a Network ID to be more specific."}}'" gmann has a patch proposed to fix some of these here: https://review.opendev.org/711049 There might be more test classes that need create_default_network = True. * http://status.openstack.org/elastic-recheck/#1844929 This one is where a job fails and the following error is seen one of the logs, usually screen-n-sch.txt: "Timed out waiting for response from cell 8acfb79b-2e40-4e1c-bc3d-d404dac6db90". The TL;DR on this one is there's no immediate clue why it's happening. This bug used to hit more occasionally on "slow" nodes like nodes from the OVH or INAP providers (and OVH restricts disk iops [1]). Now, it seems like it's hitting much more often (still mostly on OVH nodes). I've been looking at it for about a week now and I've been using a DNM patch to add debug logging, look at dstat --disk-wait output, try mysqld my.cnf settings, etc: https://review.opendev.org/701478 So far, what I find is that when we get into the fail state, we get no rows back from the database server when we query for nova 'services' and 'compute_nodes' records, and we fail with the "Timed out waiting for response" error. Haven't figured out why yet, so far. The disk wait doesn't look high when this happens (or at any time during a run) so it's not seeming like it's related to disk IO. I'm continuing to look into it. Cheers, -melanie [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505…

5 8