April 2020 - openstack-discuss - lists.openstack.org

[octavia] dropping support for stable/queens
by Adam Harwell 10 Apr '20

10 Apr '20

I'm officially proposing dropping support for the queens stable branch of Octavia. Given a quorum of cores, we will move forward with marking it EOL and all that entails. Thanks, --Adam

5 5

[all][PTL][TC] U Community Goal: Project PTL & Contrib Docs Update #6
by Kendall Nelson 10 Apr '20

10 Apr '20

Hello Everyone! While we may be past feature freeze there is still time to complete this goal given that it is documentation focused. I urge those that have not started to do so immediately. If you need help, please let me know ASAP. For those projects that have not started yet, I would start with filling in the template[2] and then go take a look at what the others have done for more inspiration[1]. If you have any questions, please let me know or refer to the goal[3]. Once you get started, please make sure to keep the story up to date[4]. I look forward to reviewing your patches!! If there is anything I can do to help, please let me know. And also, I want to give a HUGE thank you to those that have already started/completed the goal! Nice work :) -Kendall (diablo_rojo) [1] Progress: https://review.opendev.org/#/q/topic:project-ptl-and-contrib-docs+(status:o… [2] Cookiecutter Template: https://opendev.org/openstack/cookiecutter/src/branch/master/%7b%7bcookiecu… [3] Accepted Goal: https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-c… [4] Task Tracking: https://storyboard.openstack.org/#!/story/2007236

1 0

[release] Release countdown for week R-4, April 13 - 17
by Sean McGinnis 10 Apr '20

10 Apr '20

Development Focus ----------------- We just passed feature freeze! Until release branches are cut, you should stop accepting featureful changes to deliverables following the cycle-with-rc release model, or to libraries. Exceptions should be discussed on separate threads on the mailing-list, and feature freeze exceptions approved by the team's PTL. Focus should be on finding and fixing release-critical bugs, so that release candidates and final versions of the ussuri deliverables can be proposed, well ahead of the final ussuri release date. General Information ------------------- We are still finishing up processing a few release requests, but the ussuri release requirements are now frozen. If new library releases are needed to fix release-critical bugs in ussuri, you must request a Requirements Freeze Exception (RFE) from the requirements team before we can do a new release to avoid having something released in ussuri that is not actually usable. This is done by posting to the openstack-discuss mailing list with a subject line similar to: [$PROJECT][requirements] RFE requested for $PROJECT_LIB Include justification/reasoning for why a RFE is needed for this lib. If/when the requirements team OKs the post-freeze update, we can then process a new release. A soft String freeze is now in effect, in order to let the I18N team do the translation work in good conditions. In Horizon and the various dashboard plugins, you should stop accepting changes that modify user-visible strings. Exceptions should be discussed on the mailing-list. By April 23 this will become a hard string freeze, with no changes in user-visible strings allowed. Actions ------- stable/ussuri branches should be created soon for all not-already-branched libraries. You should expect 3-4 changes to be proposed for each: a .gitreview update, a reno update (skipped for projects not using reno), a tox.ini constraints URL update, and a patch updating any cycle-specific job templates to the victoria version. Please review those in priority so that the branch can be functional ASAP. Note that the victoria job templates may not be quite ready yet, so the patch may initially fail. A simple recheck once they are in place should allow the patch to pass. The Prelude section of reno release notes is rendered as the top level overview for the release. Any important overall messaging for ussuri changes should be added there to make sure the consumers of your release notes see them. Finally, if you haven't proposed ussuri cycle-highlights yet, you are already late to the party. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-April/013823.ht… for details. Upcoming Deadlines & Dates -------------------------- RC1 deadline: April 23 (R-3 week) Final RC deadline: May 7 (R-1 week) Final Ussuri release: May 13

1 0

[release][nova] FFE request for Nova Policy Defaults Refresh BP
by Ghanshyam Mann 10 Apr '20

10 Apr '20

Hello Everyone, I would like to ask the FFE for policy default refresh BP[1]. Most of the work is merged for this[2] but not 100%. Few policies not yet merged (~7) and 3 policies code is not yet up which I am working on. With that, I need more time to finish this work and require the FFE. [1] https://blueprints.launchpad.net/nova/+spec/policy-defaults-refresh [2] https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+(status:ope… -gmann

3 3

[election][tacker][ptl] Announcing my PTL candidacy for Tacker (Hyunsik Yang)
by hyunsikYang 10 Apr '20

10 Apr '20

Hello all Tacker members! I am Hyunsik Yang from the IISTRC(Internet Infra System Technology Research Center), South Korea. I wish to nominate myself for consideration as a Tacker Project PTL for the Victoria Cycle. I've previously contributed code, bug fixes and several reviews for Tacker from the Queens cycle. My goal is to improve the Tacker project by making it more standardized and precise. If you have any questions, please feel free to ask me. Thank you. Best regards, Hyunsik Yang *----------------------------------------------------------------* *Hyunsik Yang (Ph. D candidate)* *DCN laboratory / School of Electronic Engineering* *Soongsil University * *511 Sangdo-dong Dongjak-gu* *Seoul, 156-743 Korea* *TEL : (+82)-2-820-0841 * *M.P: (+82)-10-9005-7439*

2 2

[all][ideas][releases] Introducing Project Stopwatch: Another proposal to change the release cycles
by Jean-Philippe Evrard 09 Apr '20

09 Apr '20

Hello, I would like us to discuss how our release cycles are defined. We've been discussing in the past about longer or shorter cycles. This is only recorded in the mailing lists, which are hard to search. We should try to record conversations in this new "ideas" framework if we can. I think it's time to record something about the release cycles in our ideas. However, before I start the 2020 version of the eternal war "longer cycles" vs "shorter cycles", I would like us to discuss about something a little bit different, that you can find in review [1]: project Stopwatch. I have discussed with a few of you already about Stopwatch and how we could do cycles differently. This is to write the idea down. For me, it's time to change the cycles, by introducing more time to cool down. To make this happen, I need your opinions, as it quite an impactful change for the whole OpenStack. Regards, Jean-Philippe Evrard [1]: https://review.opendev.org/#/c/718929/

1 0

Etherpad and Gerrit maintenance on 2020-04-10
by corvus＠inaugust.com 09 Apr '20

09 Apr '20

Hi, Tomorrow, April 10, 2020, the OpenDev team will be performing minor updates to two services. * Gerrit restart for https://review.opendev.org/ at 16:00 UTC. This should be a momentary outage to restart the service in a container. We will maintain the same version in the same configuration; this is a preparatory step in order to perform future version upgrades more easily. Other than a short outage of about a minute, there should be no service impact or visible changes, but since the environment is changing, we wanted to announce this in case any problems are detected. * Etherpad migration and domain change at 17:00 UTC. We will migrate the data currently hosted on etherpad.openstack.org to a new server, also running in a container, on etherpad.opendev.org. We will set up redirects so that on completion of the migration, all etherpad.openstack.org URLs will redirect to the same pages on etherpad.opendev.org. https://etherpad.opendev.org/ will become the new canonical service URL. This also includes an upgrade to the latest release of Etherpad Lite (1.8.0). This involves a database migration as well. The service will be unavailable for the duration of this migration (expected to be about 30 minutes). Please report any issues in #opendev on Freenode or to the service-discuss(a)lists.opendev.org mailing list. Note that this message is being sent to service-announce and several other lists. Future announcements of this type will only be sent to service-announce, so please sign up[1] for that list if you haven't already. [1] http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -Jim

1 0

[release][oslo] FFE request for Oslo Policy
by Ghanshyam Mann 09 Apr '20

09 Apr '20

Hi, This is regarding the FFE request for policy for - https://review.opendev.org/#/c/717879/ This is part of introducing scope_type and new defaults in policies. During Nova implemtnation[1], we came across two things to solve before we release the new policies changes: 1. A lot of warnings as policy default are changing which end up filling the logs (226 MB n-api logs today). 2. Give operators an option to switch to the new system (scope and new defaults) at the same time without overwriting the policy file. We discussed those two things to solve via oslo_policy[2] which need FFE approval. Sorry for being late on this which should be planned early. Please let me know if it is ok to include these changes in the Ussuri release. [1] https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+(status:ope… [2] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.… -gmann

4 5

[kolla] Kolla Klub first meeting tomorrow
by Mark Goddard 09 Apr '20

09 Apr '20

Hi Kollaborators, Just a reminder that we will meet tomorrow at 15:00 UTC. Join here: https://meet.google.com/hph-pynx-vsy I have created a Google doc for the meeting agenda and notes: https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_…. See you tomorrow, Mark

1 1

[neutron] Drivers meeting - 10.04.2020
by Slawek Kaplonski 09 Apr '20

09 Apr '20

Hi, I will be on PTO on Good Friday so lets cancel our drivers meeting at that day. See You all on drivers meeting next week. And have a great holidays :) — Slawek Kaplonski Senior software engineer Red Hat

1 0

[tc][election] Simple question for the TC candidates
by Jean-Philippe Evrard 09 Apr '20

09 Apr '20

Hello, I read your nominations, and as usual I will ask what do you _technically_ will do during your mandate, what do you _actively_ want to change in OpenStack? This can be a change in governance, in the projects, in the current structure... it can be really anything. I am just hoping to see practical OpenStack-wide changes here. It doesn't need to be a fully refined idea, but something that can be worked upon. Thanks for your time. Regards, Jean-Philippe Evrard

12 26

[manila][release] Ussuri release preparation
by Goutham Pacha Ravi 09 Apr '20

09 Apr '20

Hi Stackers, (Following gibi's lead here on relaying release status to the ML) We're at feature freeze, and I'm providing the current state of Ussuri release deliverables from the manila team: openstack/python-manilaclient is waiting on a couple of changes: - Release patch: https://review.opendev.org/#/c/718254/ - Pending changes: bp/create-share-from-snapshot-in-another-pool-or-backend: - https://review.opendev.org/#/c/712543/ bp/graduate-share-groups-feature - https://review.opendev.org/#/c/708282/ openstack/manila-ui has a few open bug-fixes, we'll target a release in the next week or so: - https://review.opendev.org/#/q/topic:bug/1823078 openstack/manila has the following pending changes concerning Feature Freeze - bp/create-share-from-snapshot-in-another-pool-or-backend: - https://review.opendev.org/#/c/709697/ (Manila server side changes) - https://review.opendev.org/#/c/711611/ (ZFSOnLinux implementation) - https://review.opendev.org/#/c/712642/ (NetApp driver implementation) - bp/graduate-share-groups-feature - https://review.opendev.org/#/c/706959/ Ussuri Release goals status: - Drop Python 2.7 Support - Complete - Project Specific PTL and Contributor Documentation - In-Progress Thanks, Goutham

1 0

[StoryBoard][Security] The process to report a security bug
by Lingxian Kong 09 Apr '20

09 Apr '20

As most of the projects have migrated to storyboard for bug tracking, after reading https://security.openstack.org/vmt-process.html, I have two questions: 1. I didn't find openstack/ossa or ossa project exists in storyboard. 2. I didn't find a place in storyboard to attach a patch. Am I missing something? - Best regards, Lingxian Kong Catalyst Cloud

2 3

[all] Please remove all external resources from docs
by Thomas Goirand 09 Apr '20

09 Apr '20

Hi, I've wrote about this earlier I guess, but I believe I need to do it once more. Often I see in the docs things like this: .. image:: https://governance.openstack.org/tc/badges/<FOO>.svg :target: https://governance.openstack.org/tc/reference/tags/index.html [note: this is only one simple example, there's many of other types linking to external images, I've even seen stuff linking to CDNs...] I'd like to see these go away. Indeed, distributions like mine (ie: Debian) are packaging the doc. And it's *very* annoying when one browses the doc to have such an external link, and the browser going on the internet to fetch an external resource. So, in Debian, I'm patching these away. Often, the most ugly way where I don't care about the resulting document (as long as the doc is there). This is both annoying, and a loss of my time. The solution is simple: have the resource being *LOCAL* (ie: stored in the project's doc), not stored on an external site. Thanks to anyone who will follow this advice. Cheers, Thomas Goirand (zigo)

7 9

[nova] Ussuri release preparation
by Balázs Gibizer 09 Apr '20

09 Apr '20

Hi, Today is Feature Freeze for Ussuri and the final release of novaclient for Ussuri. Release TODOs are tracked in [1]. Any feature code in nova that is approved before end of today is OK to be rechecked or rebased if needed after the freeze. Features that are close to land but haven't approved until the freeze, needs a Feature Freeze Exception on the ML or they will be postponed to the Victoria cycle. When you request such FFE please try to find cores who support your request. Bug fixes, test coverage and documentation enhancements are still OK to be proposed after the freeze. However please be careful approving risky patches. Any release blocker issue should have a bug tagged with ussuri-rc-potential in launchpad. Please also add them the the etherpad [1] Cheers, gibi [1] https://etherpad.openstack.org/p/nova-ussuri-rc-potential

1 0

Re: [Release-job-failures] release-post job for openstack/releases for ref refs/heads/master failed
by Sean McGinnis 08 Apr '20

08 Apr '20

On 4/8/20 4:53 PM, zuul(a)openstack.org wrote: > Build failed. > > - tag-releases https://zuul.opendev.org/t/openstack/build/28b78b5235894878b4cb92822b95bbff : FAILURE in 5m 53s > - publish-tox-docs-releases https://zuul.opendev.org/t/openstack/build/None : SKIPPED This failure was due to a new operation we added to the branching process. Once working, the new code will propose patches to branched repos to update them to use the job template defined for the next release cycle, similar to how the patches are proposed to update reno. I think this was due to an error in a sed command that is used to perform that update, and will hopefully be fixed after https://review.opendev.org/718564 merges. No further action is needed here. The only thing missed was the publishing of the releases docs, but that will be corrected with the next release patch that goes through. Sean

1 0

[all][PTL][release] Call for Ussuri Cycle Highlights
by Kendall Nelson 08 Apr '20

08 Apr '20

Hello Everyone! Its time to start thinking about calling out 'cycle-highlights' in your deliverables! As PTLs, you probably get many pings towards the end of every release cycle by various parties (marketing, management, journalists, etc) asking for highlights of what is new and what significant changes are coming in the new release. By putting them all in the same place it makes them easy to reference because they get compiled into a pretty website like this from the last few releases: Rocky[1], Stein[2], Train[3]. We don't need a fully fledged marketing message, just a few highlights (3-4 ideally), from each project team. Looking through your release notes might be a good place to start. *The deadline for cycle highlights is the end of the R-5 week [4] on April 10th.* How To Reminder: ------------------------- Simply add them to the deliverables/train/$PROJECT.yaml in the openstack/releases repo like this: cycle-highlights: - Introduced new service to use unused host to mine bitcoin. The formatting options for this tag are the same as what you are probably used to with Reno release notes. Also, you can check on the formatting of the output by either running locally: tox -e docs And then checking the resulting doc/build/html/train/highlights.html file or the output of the build-openstack-sphinx-docs job under html/train/ highlights.html. Can't wait to see you all have accomplished this release! Thanks :) -Kendall Nelson (diablo_rojo) [1] https://releases.openstack.org/rocky/highlights.html [2] https://releases.openstack.org/stein/highlights.html [3] https://releases.openstack.org/train/highlights.html [4] htt <https://releases.openstack.org/train/schedule.html#t-cycle-highlights> https://releases.openstack.org/ussuri/schedule.html

2 5

No UC meeting this week
by Mohamed Elsakhawy 08 Apr '20

08 Apr '20

Hi All, There will be no UC meeting tomorrow. Our next meeting is on Thursday April 16th in our regular time. Thanks Mohamed

1 0

[all] Gate status: tempest-full broken on stein
by Ghanshyam Mann 08 Apr '20

08 Apr '20

Hello Everyone, tempest-full (py2 job) started failing on stable/stein gate because devstack try to install master Tempest on system-wide and fail for min require py version by Tempest. We did not set the INSTALL_TEMPEST flag to false for stable branches since stable/pike. I am going to add this as one of thing to do while cutting devstack stable branch. More details on Bug: https://bugs.launchpad.net/tempest/+bug/1871327 Here are the fixes, I am backporting from stable/train: Please hold the recheck until they are merged. https://review.opendev.org/#/q/I60949fb735c82959fb2cfcb6aeef9e33fb0445b6 -gmann

1 2

[tc][oslo] Looking for a Victoria Oslo PTL volunteer
by Thierry Carrez 08 Apr '20

08 Apr '20

Hello everyone, Nobody stepped up yet to continue the great work of Ben Nemec as PTL of the Oslo project team. Oslo handles common code between our various projects, and is key to reduce duplication of code and waste of effort. Oslo is very critical for OpenStack, as all other projects depend on its libraries, but can be relatively low-maintenance as most libraries are feature-complete. Being the Oslo PTL gives you a great horizontal perspective on OpenStack, so they are a great opportunity for smaller organizations that depend on OpenStack to keep track of what's happening and contribute back, without having to specialize in one of the larger projects. I suspect Ben can help answer any question on the workload it represents to be the Oslo PTL. Getting a new volunteer for Oslo PTL is our preferred solution. In case the PTL position is just too intimidating and nobody volunteers, the TC is considering experimenting with a PTL-less form of governance that would involve only designating two liaisons (a release liaison(s) and a security liaison(s)). That is the minimal accountability that we require of an openstack project team (someone(s) signing off releases, and someone(s) to contact in case of embargoed vulnerabilities). I say "someone(s)" as ideally multiple people sign up to be a liaison. In that form of governance, if there is any conflict in the team, the TC can ultimately arbitrate the conflict, instead of the PTL. That would be our plan B. So please let us know if you'd be interested in the Oslo PTL position, or if you'd be OK just for filling the duties of a specific liaison. Thanks for considering it! -- Thierry Carrez (ttx)

3 3

[tripleo] older packages in promoted repos
by Wesley Hayutin 08 Apr '20

08 Apr '20

Greetings, Please hold off any major changes in TripleO for 24 - 48 hours. As discussed in the #tripleo meeting this morning, old rpms from December 2019 have found their way into promoted RDO yum repos for CentOS-8 master [1]. The CI team and RDO team working together to resolve the issue. This is causing some jobs to fail, especially 3rd party jobs at the moment like featureset001. [2,3] We are working through each component yum repo to ensure the latest content is available and will have to repromote content to the upstream. Sorry for the delay and the bugs this is causing in getting your patches validated. [1] https://review.rdoproject.org/r/#/c/26294/ [2] https://bugs.launchpad.net/tripleo/+bug/1871010 [3] https://bugs.launchpad.net/tripleo/+bug/1871181

1 1

[largescale-sig] Next meeting: Apr 8, 8utc
by Thierry Carrez 08 Apr '20

08 Apr '20

Hi everyone, The Large Scale SIG will have a meeting this week on Wednesday, April 8 at 8 UTC[1] in #openstack-meeting on IRC. The meeting time was changed at the last meeting to be more friendly to our Asian participants, so please doublecheck how that translates for you at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200408T08 Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting A reminder of the TODOs we had from last meeting, in case you have time to make progress on them: - ttx to push for oslo.metric spec approval - oneswig to contribute a scaling story on bare metal cluster scaling - amorin to create a wiki page for large scale documentation - amorin to propose patch against Nova doc Talk to you all on Wednesday, -- Thierry Carrez

1 1

[ironic][release] List cycle-with-intermediary deliverables that have not been released yet
by Herve Beraud 08 Apr '20

08 Apr '20

Hello ironic team, Quick reminder that we'll need a release very soon for a number of deliverables following a cycle-with-intermediary release model but which have not done *any* release yet in the Ussuri cycle: - bifrost - ironic-prometheus-exporter - ironic-ui Those should be released ASAP, and in all cases before R-3 week (RC1 deadline Apr 20 - Apr 24) , so that we have a release to include in the final Ussuri release. Cheers, -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

6 10

[stackalytics] x/stackalytics repo still used?
by Andreas Jaeger 07 Apr '20

07 Apr '20

Hi Sergey and stackalytics experts, the wiki at https://wiki.openstack.org/wiki/Stackalytics#Company_affiliation points to using the github site for stackalytics - but I see that there are still submissions to the opendev repo at https://review.opendev.org/#/q/project:x/stackalytics so, how are those repos used? Is it time to retire the x/stackalytics repository so that nobody can submit there? Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

2 1

[nova][gate] status of some gate bugs
by melanie witt 07 Apr '20

07 Apr '20

Hey all, We've been having a tough time lately in the gate hitting various bugs while our patches go through CI. I just wanted to mention a few of them that I've seen often in my gerrit notifications and give a brief status on fix efforts. * http://status.openstack.org/elastic-recheck/#1813789 This one is where the nova-live-migration job fails a server evacuate test with: "Timeout waiting for [('network-vif-plugged', 'e3d3db3f-bce4-4889-b161-4b73648f79be')] for instance with vm_state error and task_state rebuild_spawning.: eventlet.timeout.Timeout: 300 seconds" in the screen-n-cpu.txt log. lyarwood has a WIP patch here: https://review.opendev.org/713674 and sean-k-mooney has a WIP patch here: https://review.opendev.org/713342 * https://launchpad.net/bugs/1867380 This one is where the nova-live-migration or nova-grenade-multinode job fail due to n-cpu restarting slowly after being reconfigured for ceph. The server will fail to build and it's because the test begins before nova-compute has fully come up and we see this error: "Instance spawn was interrupted before instance_claim, setting instance to ERROR state {{(pid=3783) _error_out_instances_whose_build_was_interrupted" in the screen-n-cpu.txt log. lyarwood has a patch approved here that we've been rechecking the heck out of that has yet to merge: https://review.opendev.org/713035 * https://launchpad.net/bugs/1844568 This one is where a job fails with: "Body: b'{"conflictingRequest": {"code": 409, "message": "Multiple possible networks found, use a Network ID to be more specific."}}'" gmann has a patch proposed to fix some of these here: https://review.opendev.org/711049 There might be more test classes that need create_default_network = True. * http://status.openstack.org/elastic-recheck/#1844929 This one is where a job fails and the following error is seen one of the logs, usually screen-n-sch.txt: "Timed out waiting for response from cell 8acfb79b-2e40-4e1c-bc3d-d404dac6db90". The TL;DR on this one is there's no immediate clue why it's happening. This bug used to hit more occasionally on "slow" nodes like nodes from the OVH or INAP providers (and OVH restricts disk iops [1]). Now, it seems like it's hitting much more often (still mostly on OVH nodes). I've been looking at it for about a week now and I've been using a DNM patch to add debug logging, look at dstat --disk-wait output, try mysqld my.cnf settings, etc: https://review.opendev.org/701478 So far, what I find is that when we get into the fail state, we get no rows back from the database server when we query for nova 'services' and 'compute_nodes' records, and we fail with the "Timed out waiting for response" error. Haven't figured out why yet, so far. The disk wait doesn't look high when this happens (or at any time during a run) so it's not seeming like it's related to disk IO. I'm continuing to look into it. Cheers, -melanie [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505…

5 8