May 2019 - openstack-discuss - lists.openstack.org

[dev][keystone] Keystone Team Update - Week of 13 May 2019
by Colleen Murphy 17 May '19

17 May '19

# Keystone Team Update - Week of 13 May 2019 ## News Another quiet week as people gear up for summer holidays. ## Open Specs Train specs: https://bit.ly/2uZ2tRl Ongoing specs: https://bit.ly/2OyDLTh As we're gearing up for feature work for the cycle, please prioritize reviewing specs. ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 17 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 32 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ## Bugs This week we opened 10 new bugs and closed 1. Two of the bugs opened this week are for tracking deprecations and removals and will be closed at the end of the cycle, some others are for tracking work that was discussed at the PTG. Bugs opened (10) Bug #1829453 (keystone:Low) opened by Colleen Murphy https://bugs.launchpad.net/keystone/+bug/1829453 Bug #1829454 (keystone:Low) opened by Colleen Murphy https://bugs.launchpad.net/keystone/+bug/1829454 Bug #1829573 (keystone:Low) opened by Colleen Murphy https://bugs.launchpad.net/keystone/+bug/1829573 Bug #1829574 (keystone:Low) opened by Colleen Murphy https://bugs.launchpad.net/keystone/+bug/1829574 Bug #1829455 (keystone:Wishlist) opened by Vishakha Agarwal https://bugs.launchpad.net/keystone/+bug/1829455 Bug #1828783 (keystone:Undecided) opened by Akihiro Motoki https://bugs.launchpad.net/keystone/+bug/1828783 Bug #1829296 (keystone:Undecided) opened by Douglas Mendizábal https://bugs.launchpad.net/keystone/+bug/1829296 Bug #1829575 (keystonemiddleware:Low) opened by Colleen Murphy https://bugs.launchpad.net/keystonemiddleware/+bug/1829575 Bug #1828737 (oslo.policy:Undecided) opened by Doug Hellmann https://bugs.launchpad.net/oslo.policy/+bug/1828737 Bug #1828739 (oslo.policy:Undecided) opened by Doug Hellmann https://bugs.launchpad.net/oslo.policy/+bug/1828739 Bugs closed (1) Bug #1828783 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1828783 ## Milestone Outlook https://releases.openstack.org/train/schedule.html Our spec proposal freeze is three weeks away. If you want to get a major initiative done in keystone this cycle, now is the time to propose it. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter

1 0

[self-healing] live-migrate instance in response to fault signals
by Eric K 17 May '19

17 May '19

Hi dasp, Follow up on the discussion today at self-healing BoF. I think you said on the etherpad [1]: ==== Ability to drain (live migrate away) instances automatically in response to any failure/soft-fail/early failure indication (e.g. dropped packets, SMART disk status, issues with RBD connections, repeated build failures, etc) Then quarantine, rebuild, self-test compute host (or hold for hardware fix) Context: generally no clue what is running inside VMs (like public cloud) ==== I just want to follow up to get more info on the context; specifically, which of the following pieces are the main difficulties? - detecting the failure/soft-fail/early failure indication - codifying how to respond to each failure scenario - triggering/executing the desired workflow - something else [1] https://etherpad.openstack.org/p/DEN-self-healing-SIG

2 2

[nova][all][ptg] Summary: Same-Company Approvals
by Eric Fried 17 May '19

17 May '19

(NB: I tagged [all] because it would be interesting to know where other teams stand on this issue.) Etherpad: https://etherpad.openstack.org/p/nova-ptg-train-governance Summary: - There is a (currently unwritten? at least for Nova) rule that a patch should not be approved exclusively by cores from the same company. This is rife with nuance, including but not limited to: - Usually (but not always) relevant when the patch was proposed by member of same company - N/A for trivial things like typo fixes - The issue is: - Should the rule be abolished? and/or - Should the rule be written down? Consensus (not unanimous): - The rule should not be abolished. There are cases where both the impetus and the subject matter expertise for a patch all reside within one company. In such cases, at least one core from another company should still be engaged and provide a "procedural +2" - much like cores proxy SME +1s when there's no core with deep expertise. - If there is reasonable justification for bending the rules (e.g. typo fixes as noted above, some piece of work clearly not related to the company's interest, unwedging the gate, etc.) said justification should be clearly documented in review commentary. - The rule should not be documented (this email notwithstanding). This would either encourage loopholing or turn into a huge detailed legal tome that nobody will read. It would also *require* enforcement, which is difficult and awkward. Overall, we should be able to trust cores to act in good faith and in the appropriate spirit. efried .

11 15

It’s OpenStack User Survey Time!
by Matt Van Winkle 17 May '19

17 May '19

Hi everyone, If you operate OpenStack, it’s time to participate in the annual OpenStack User Survey <https://www.openstack.org/user-survey>. This is your opportunity to provide direct feedback to the OpenStack community, so we can better understand your environment and needs. We send all anonymous feedback directly to the project teams who work to improve the software and documentation. The survey will take less than 20 minutes. All participants who complete the OpenStack deployment survey will receive AUC status and discounted registration to the Shanghai Summit. Please let me or any members of the User Committee know if you have any questions. Cheers, VW -- Matt Van Winkle Senior Manager, Software Engineering | Salesforce

1 0

[placement] update 19-19
by Chris Dent 17 May '19

17 May '19

HTML: https://anticdent.org/placement-update-19-19.html Woo! Placement update 19-19. First one post PTG and Summit. Thanks to everyone who helped make it a useful event for Placement. Having the pre-PTG meant that we had addressed most issues prior to getting there meaning that people were freed up to work in other areas and the discussions we did have were highly coherent. Thanks, also, to everyone involved in getting placement deleted from nova. We did that while at the PTG and had a little [celebration](https://tank-binaries.s3.amazonaws.com/8e922a32c7ff4116a68d730…. # Most Important We're still working on narrowing priorities and focusing the details of those priorities. There's an [etherpad](https://etherpad.openstack.org/p/placement-ptg-train-rfe-voter) where we're taking votes on what's important. There are three specs in progress from that that need review and refinement. There are two others which have been put on the back burner (see specs section below). # What's Changed * We're now [running a subset](https://review.opendev.org/657077) of nova's functional tests in placement's gate. * osc-placement is using the PlacementFixture to run its functional tests making them _much_ faster. * There's a set of StoryBoard [worklists](https://docs.openstack.org/placement/latest/contributor/contribu… that can be used to help find in progress work and new bugs. That section also describes how tags are used. * There's a [summary of summaries](http://lists.openstack.org/pipermail/openstack-discuss/2019-May/… email message that summarizes and links to various results from the PTG. # Specs/Features As the summary of summaries points out, we have two major features this cycle, one of which is _large_: getting consumer types going and getting a whole suite of features going to support nested providers in a more effective fashion. * <https://review.opendev.org/654799> Support Consumer Types. This is very close with a few details to work out on what we're willing and able to query on. It only has reviews from me so far. * <https://review.opendev.org/658510> Spec for Nested Magic. This is associated with a [lengthy story](https://storyboard.openstack.org/#!/story/2005575) that includes visual artifacts from the PTG. It covers several related features to enable nested-related requirements from nova and neutron. It is a work in progress, with several unanswered questions. It is also something that efried started but will be unable to finish so the rest of us will need to finish it up as the questions get answered. And it also mostly subsumes a previous spec on [subtree affinity](https://review.opendev.org/#/c/650476/). (Eric, please correct me if I'm wrong on that.) * <https://review.opendev.org/657582> Resource provider - request group mapping in allocation candidate. This spec was copied over from nova. It is a requirement of the overall nested magic theme. While it has a well-defined and refined design, there's currently no one on the hook implement it. There are also two specs that are still live but de-prioritized: * <https://review.openstack.org/649992> support any trait in allocation candidates * <https://review.openstack.org/649368> support mixing required traits with any traits These and other features being considered can be found on the [feature worklist](https://storyboard.openstack.org/#!/worklist/594). Some non-placement specs are listed in the Other section below. # Stories/Bugs There are 23 stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 are [untagged](https://storyboard.openstack.org/#!/worklist/580). 4 are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 12 are [rfes](https://storyboard.openstack.org/#!/worklist/594). 2 are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. On launchpad: * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 16. +3 * [In progress placement bugs](https://goo.gl/vzGGDQ) on launchpad: 6. +2. These are placement-related, in nova. Of those there two interesting ones to note: * <https://bugs.launchpad.net/nova/+bug/1829062> nova placement api non-responsive due to eventlet error. When using placement-in-nova in stein, recent eventlet changes can cause issues. As I've mentioned on the bug the best way out of this problem is to use placement-in-placement but there are other solutions. * <https://bugs.launchpad.net/nova/+bug/1829479> The allocation table has residual records when instance is evacuated and the source physical node is removed. This appears to be yet another issue related to orphaned allocations during one of the several move operations. The impact they are most concerned with, though, seems to be the common "When I bring up a new compute node with the same name there's an existing resource provider in the way" that happens because of the unique constrain on the rp name column. I'm still not sure that constraint is the right thing unless we want to make people's lives hard when they leave behind allocations. We may want to make it hard because it will impact quota... # osc-placement osc-placement is currently behind by 11 microversions. No change since the last report. Pending changes: _Note: a few of these having been sitting for some time with my +2 awaiting review by some other placement core. Please remember osc-placement when reviewing._ * <https://review.openstack.org/#/c/640898/> Add 'resource provider inventory update' command (that helps with aggregate allocation ratios). * <https://review.openstack.org/#/c/651783/> Add support for 1.22 microversion * <https://review.openstack.org/586056> Provide a useful message in the case of 500-error * <https://review.openstack.org/650257> Remove unused cruft from doc and releasenotes config * <https://review.openstack.org/652100> Improve aggregate version check error messages with min_version * <https://review.opendev.org/653285> Expose version error message generically # Main Themes Now that the PTG has passed some themes have emerged. Since the Nested Magic one is rather all encompassing and Cleanup is a catchall, I think we can consider three enough. If there's some theme that you think is critical that is being missed, let me know. For people coming from the nova-side of the world who need or want something like review runways to know where they should be focusing their review energy, consider these themes and the links within them as a runway. But don't forget bugs and everything else. ## Nested Magic At the PTG we decided that it was worth the effort, in both Nova and Placement, to make the push to make better use of nested providers — things like NUMA layouts, multiple devices, networks — while keeping the "simple" case working well. The general ideas for this are described in a [story](https://storyboard.openstack.org/#!/story/2005575) and an evolving [spec](https://review.opendev.org/658510). Some code has started, mostly to reveal issues: * <https://review.opendev.org/657419> Changing request group suffix to string * <https://review.opendev.org/657510> WIP: Allow RequestGroups without resources * <https://review.opendev.org/657463> Add NUMANetworkFixture for gabbits * <https://review.opendev.org/658192> Gabbi test cases for can_split ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. A [spec](https://review.opendev.org/654799) has started. There are some questions about request and response details that need to be resolved, but the overall concept is sound. ## Cleanup As we explore and extend nested functionality we'll need to do some work to make sure that the code is maintainable and has suitable performance. There's some work in progress for this that's important enough to call out as a theme: * <https://storyboard.openstack.org/#!/story/2005712> Some work from Tetsuro exploring ways to remove redundancies in the code. There's a [related WIP](https://review.opendev.org/658778) * <https://review.opendev.org/659522> Enhance debug logging in allocation candidate handling * <https://review.opendev.org/658164> Start of a stack that will allow us to remove the protections against null root providers (which turns out is a pretty significant performance hit). * <https://review.opendev.org/643269> WIP: Optionally run a wsgi profiler when asked. This was used to find some of the above issues. Should we make it generally available or is it better as a thing to base off when exploring? Ed Leafe has also been doing some intriguing work on using graph databases with placement. It's not yet clear if or how it could be integrated with mainline placement, but there are likely many things to be learned from the experiment. # Other Placement * <https://review.opendev.org/#/q/topic:refactor-classmethod-diaf> A suite of refactorings that given their lack of attention perhaps we don't need or want, but let's be explicit about that rather than ignoring the patches if that is indeed the case. * <https://review.opendev.org/645255> A start at some unit tests for the PlacementFixture which got lost in the run up to the PTG. They may be less of a requirement now that placement is running nova's functional tests. But again, we should be explicit about that decision. # Other Service Users New discoveries are added to the end. Merged stuff is removed. * <https://review.openstack.org/552924> Nova: Spec: Proposes NUMA topology with RPs * <https://review.openstack.org/622893> Nova: Spec: Virtual persistent memory libvirt driver implementation * <https://review.openstack.org/641899> Nova: Check compute_node existence in when nova-compute reports info to placement * <https://review.openstack.org/601596> Nova: spec: support virtual persistent memory * <https://review.openstack.org/#/q/topic:bug/1790204> Workaround doubling allocations on resize * <https://review.openstack.org/645316> Nova: Pre-filter hosts based on multiattach volume support * <https://review.openstack.org/647396> Nova: Add flavor to requested_resources in RequestSpec * <https://review.openstack.org/633204> Blazar: Retry on inventory update conflict * <https://review.openstack.org/#/q/topic:bp/count-quota-usage-from-placement> Nova: count quota usage from placement * <https://review.openstack.org/#/q/topic:bug/1819923> Nova: nova-manage: heal port allocations * <https://review.openstack.org/648665> Nova: Spec for a new nova virt driver to manage an RSD * <https://review.openstack.org/625284> Cyborg: Initial readme for nova pilot * <https://review.openstack.org/629142> Tempest: Add QoS policies and minimum bandwidth rule client * <https://review.openstack.org/648687> Nova-spec: Add PENDING vm state * <https://review.openstack.org/650188> nova-spec: Allow compute nodes to use DISK_GB from shared storage RP * <https://review.openstack.org/651024> nova-spec: RMD Plugin: Energy Efficiency using CPU Core P-State control * <https://review.openstack.org/651455> puppet: Debian: Add support for placement-api over uwsgi * <https://review.openstack.org/650963> nova-spec: Proposes NUMA affinity for vGPUs. This describes a legacy way of doing things because affinity in placement may be a ways off. But it also [may not be](https://review.openstack.org/650476). * <https://review.openstack.org/#/q/topic:heal_allocations_dry_run> Nova: heal allocations, --dry-run * <https://review.openstack.org/642527> Neutron: Fullstack test for placement sync * <https://review.opendev.org/656448> Watcher spec: Add Placement helper * <https://review.opendev.org/659233> Cyborg: Placement report * <https://review.opendev.org/657884> Nova: Spec to pre-filter disabled computes with placement * <https://review.opendev.org/657801> rpm-packaging: placement service * <https://review.opendev.org/657016> Delete resource providers for all nodes when deleting compute service # End I'm out of practice on these things. This one took a long time. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

2 1

[ptg][kolla][openstack-ansible][tripleo] PTG cross-project summary
by Mark Goddard 17 May '19

17 May '19

Hi, This is a summary of the ad-hoc cross project session between the OpenStack Ansible and Kolla teams. It occurred to me that our two teams likely face similar challenges, and there are areas we could collaborate on. I've tagged TripleO also since the same applies there. [Collaboration on approach to features] This was my main reason for proposing the session - there are features and changes that all deployment tools need to make. Examples coming up include support for upgrade checkers and IPv6. Rather than work in isolation and solve the same problem in different ways, perhaps we could share our experiences. The implementations will differ, but providing a reasonably consistent feel between deployment tools can't be a bad thing. As a test case, we briefly discussed our experience with the upgrade checker support added in Stein, and found that our expectation of how it would work was fairly aligned in the room, but not aligned with how I understand it to actually work (it's more of a post-upgrade check than a pre-upgrade check). I was also able to point the OSA team at the placement migration code added to Kolla in the Stein release, which should save them some time, and provide more eyes on our code. I'd like to pursue this more collaborative approach during the Train release where it fits. Upgrade checkers seems a good place to start, but am open to other ideas such as IPv6 or Python 3. [OSA in Kayobe] This was my wildcard - add support for deploying OpenStack via OSA in Kayobe as an alternative to Kolla Ansible. It could be a good fit for those users who want to use OSA but don't have a provisioning system. This wasn't true of anyone in the room, and lack of resources deters from 'build it and they will come'. Still, the seed is planted, it may yet grow. [Sharing Ansible roles] mnaser had an interesting idea: add support for deploying kolla containers to the OSA Ansible service roles. We could then use those roles within Kolla Ansible to avoid duplication of code. There is definitely some appeal to this in theory. In practice however I feel that the two deployment models are sufficiently different that it would add significantly complexity to both projects. Part of the (relative) simplicity and regularity of Kolla Ansible is enabled by handing off installation and other tasks to Kolla. One option that might work however is sharing some of the lower level building blocks. mnaser offered to make a PoC for using https://github.com/openstack/ansible-config_template to generate configuration in Kolla Ansible in place of merge_config and merge_yaml. It requires some changes to that role to support merging a list of source template files. We'd also need to add an external dependency to our 'monorepo', or 'vendor' the module - trade offs to make in complexity vs. maintaining our own module. I'd like to thank the OSA team for hosting the discussion - it was great to meet the team and share experience. Cheers, Mark

4 6

[Cinder][Ceph] Migrating backend host
by Gary Molenkamp 17 May '19

17 May '19

I am moving my cinder-volume service from one controller to another, and I'm trying to determine the correct means to update all existing volume's back-end host reference. I now have two cinder volume services running in front of the same ceph cluster, and I would like to retire the old cinder-volume service. For example, on my test cloud, "openstack volume service list": +------------------+----------------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+----------------------------------+------+---------+-------+----------------------------+ | cinder-scheduler | osdev-ctrl1 | nova | enabled | up | 2019-05-15T17:44:05.000000 | | cinder-volume | osdev-ctrl2@rbd | nova | disabled | up | 2019-05-15T17:44:04.000000 | | cinder-volume | osdev-ctrl1@rbd | nova | enabled | up | 2019-05-15T17:44:00.000000 | +------------------+----------------------------------+------+---------+-------+----------------------------+ Now, an existing volume has a reference to the disabled cinder-volume: "os-vol-host-attr:host | osdev-ctrl2@rbd#rbd" but this needs to be changed to: "os-vol-host-attr:host | osdev-ctrl1@rbd#rbd" As both controllers are members of the same ceph cluster, an "openstack volume migrate" is not appropriate. If it is appropriate, my testing has shown that it errors out and deletes the source volume from ceph. I can alter this field manually in the cinder database, but in following the "don't mess with the data model" mantra, is there a means to do this from the cli? Thanks, Gary. Openstack release: Queens. Distro: Redhat (Centos) 7.6 -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566

3 2

[tripleo][ci] rdo cloud outage - maintenance
by Marios Andreou 16 May '19

16 May '19

There is ongoing maintenance outage in rdo cloud... this means there are currently NO 3rd party rdo jobs running on any tripleo project reviews. We also stopped the promoter about an hour ago in preparation so there are no promotions. The outage should be resolved today we will update when things are back to normal thanks (marios|ruck and panda|rover on #tripleo and #oooq if you want to ping us)

1 1

Re: [OSC][PTG] Summary: many things to do!
by Matt Riedemann 16 May '19

16 May '19

On 5/16/2019 1:48 AM, Artem Goncharov wrote: > Artom signed up for this but I don't see patches yet. > > Here you are: <snip> > > Since I am not a full-time contributor it takes a bit more time, > otherwise I would have created patches for migration of all osc services > to sdk already ;-) I'm hoping there wasn't a misunderstanding with my comment. I literally meant "Artom" and not a typo for "Artem", because I'm talking about Artom Lifshitz specifically signing up [1] for the boot-from-volume stuff I mentioned. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005793.html -- Thanks, Matt

1 0

[nova] cross-cell resize review guide
by Matt Riedemann 16 May '19

16 May '19

At the Train PTG people said it would be useful to have a review guide in the ML like what gibi did for the bandwidth provider series in Stein, so this is my attempt. First let me start by saying if I were you, I'd read the spec [1] and watch the presentation from the summit [2] since those should give you the high-level overview of what's going on. **Notes** * The current bottom of the series starts here [3]. That bottom change is actually a bug fix for a long-standing latent race issue in the resource tracker and a functional recreate test patch has already been merged. I ran into this during my functional testing later in the series when auditing resource tracker stuff. * I'm pushing as much non-cross-cell related stuff to the bottom of the series as possible, like refactoring utility methods and test fixture enhancements, and some bug fixes. For anything that would otherwise be unused outside of the series I don't move to the bottom. * The rest of the series works from the bottom up in that nothing "turns it all on" externally in the API until the end of the series. This means we can land code as we go with little risk of impacting existing behavior. Even then the new policy rule that enables this is disabled for all users by default. * The instance task_state flow, instance action records and notifications should be consistent to normal resize throughout to make sure it is transparent to the end user. * We do hard deletes of database records on rollback, confirm and revert. We have a copy of the instance in each DB while resized, but need to hard delete the one we aren't going to keep at our terminal state (otherwise you can't try to resize the server back to the source cell without archiving that DB). * Rather than have the computes RPC cast back and forth to each other, conductor will be orchestrating the compute service work using synchronous RPC calls with the long_rpc_timeout option. This is because we assume the computes can't communicate with each other over RPC, SSH or shared storage. **Testing** * About halfway through the series [4], once the code is plumbed to get the server to VERIFY_RESIZE state, I start a series of functional tests because for a lot of this I'm trying to rely on functional rather than unit tests. I do have unit tests in the majority of the patches but not all of them toward the end because frankly I don't want to spend a lot of time writing unit tests before the code is reviewed - the functional tests are much better at flushing out issues. * At the very end of the series I have a patch [5] which enables cross-cell resize and cold migration using the new nova-multi-cell job. I'm easing into this by only enabling a few tests at a time, flushing issues, and iterating that way since it makes debugging one failing test rather than 10 much easier (to trace the logs and such). So far I've gotten resize + confirm, cold migrate + confirm, and volume-backed resize + confirm passing in that job. I'm currently working on a cold migrate + revert test. **The main idea** We want to keep the new cross-cell resize flow as similar to traditional resize as possible, so you'll see a similar flow of: a) prep for resize on dest host This is just a resource claim and to create the migration context in the target cell DB. But otherwise it's very similar to prep_resize. b) resize from source host Power off the guest, disconnect volumes and interfaces from the source host, and do disk stuff (in this case create a temporary snapshot of the root disk for a non-volume-backed server). c) finish on the dest host Connect volumes, plug VIFs, and spawn a guest (using the snapshot created for a non-volume-backed server). Super easy right?! **Conductor** The big change with cross-cell resize is that (super)conductor is going to be doing the heavy lifting of orchestrating the resize across the two cells. For this I've created a main task (CrossCellMigrationTask) which then drives the sequential execution of sub-tasks, which for the most part mirror the steps above for the operations in the compute services. Also note that each task has a rollback method and as such if any task fails, we run the rollback of all the other tasks back to the beginning which means each task can just care about cleaning up anything it needs to rather than a giant single try/except in the main task. The first thing we have to do is copy the instance and its related records to the target cell database (TargetDBSetupTask). The copy of the instance created in the target cell DB will be marked with a new "hidden" field so that when listing servers in the API we'll filter it out. For the DB copy stuff there needed to be the DB schema change for that hidden column and then just some new versioned object DB API type methods to aid with the data transfer. After that we run the sub-tasks that mirror the compute service operations (prep on dest, resize from source, finish on dest). In between the compute services doing stuff we sometimes need to copy data around, e.g. after PrepResizeAtDestTask we need to copy the migration context from the target cell DB to the source cell DB so the API can handle the network-vif-plugged even when the guest is spawned on the dest host. At the very end once we've finished on the dest host, we update the instance mapping in the API DB to point at the target cell and swap the hidden values on the source/target cell instance so when listing servers we'll be showing the target cell instance in VERIFY_RESIZE state. **Scheduler** All move operations are currently restricted to the same cell (there is code in conductor that does this). If the request passes the new policy check, the RequestSpec is modified so the scheduler will not restrict to just hosts in the source cell. There is a new CrossCellWeigher added which by default will prefer hosts in the same cell if there is a choice, but there might not be a choice, e.g. an admin could target a host in another cell for a cold migration. Also, the weigher can be configured to prefer to move instances to other cells or be overridden on a per-aggregate basis. I talk about this in the video a bit, but the idea is leave the default to prefer hosts in the source cell but override via aggregates to drain instances out of old cells using the weigher. Also, a reminder that when the scheduler picks a primary host, any alternates are in the same cell. This isn't new but is good to remember. **Operation flow** 1. The API will check the new policy rule and if the request passes the policy check, the RequestSpec is modified to say cross-cell migration is OK (conductor and scheduler will look for this) and RPC cast to conductor (rather than RPC call like it does today). 2. The MigrationTask in conductor will RPC call the scheduler's select_destinations method as normal. a) If the selected target host is in another cell, kick off the CrossCellMigrationTask. b) If the selected target host is in the same cell as the instance, we do traditional resize as normal. 3. CrossCellMigrationTask will check that neutron and cinder are new enough for port bindings and volume attachment resources and then execute TargetDBSetupTask. 4. TargetDBSetupTask does the DB copy stuff mentioned above. 5. CrossCellMigrationTask then executes PrepResizeAtDestTask which will create (inactive) port bindings for the target host and empty volume attachments, then RPC call the dest compute to do the resize claim (this is needed for instances with PCI devices and NUMA topology). a) If all of that works, cool, continue. b) If any of that fails, cleanup and try an alternate host (note the alternate host processing is not implemented yet but shouldn't be hard). 6. Next PrepResizeAtSourceTask gets executed which creates a snapshot image for a non-volume-backed server, powers off the guest (but does not destroy the root disks), disconnects volumes and interfaces. At this step on the source compute we also activate the inactive dest host port bindings created earlier. 7. Next FinishResizeAtDestTask calls the dest compute to setup networking on the dest host (most just for PCI stuff), connect volumes, and spawn the guest (from root volume or temp snapshot). Once that is done the task will swap the hidden value on the instances and update the instance mapping in the API DB. At this point the instance is in VERIFY_RESIZE status. **Confirm/Revert** Confirm and revert are implemented similarly with new conductor tasks. Confirm is pretty simple in that it just cleans up the guest from the source compute and deletes the records from the source cell DB. Revert is a bit more work since we have to cleanup the dest compute host, adjust BDMs in the source cell DB if any volumes were attached to or detached from the instance while it was in VERIFY_RESIZE state (yes the API allows that), spawn the guest in the source compute and then twiddle the DB bits so the instance mapping is pointing at the source cell and the source instance is no longer hidden, and then cleanup the target cell DB. **Patch series structure** As noted I'm writing this from the bottom up, and even within sub-operations I'm following a pattern of (1) write the compute methods first and then (2) write the conductor task that will use them and integrate it into the main CrossCellMigrationTask. This means there will be multiple compute RPC API version bumps as new methods get added. The scheduler stuff and some API things happen before the functional testing starts so I can actually start functional testing with stubbing out as little as possible (I have to stub out the policy check in the functional test long before it's enabled in the API at the end of the series). The CrossCellWeigher comes late in the series because by default all in-tree weighers are enabled so I don't want to land that and have it running if there is no chance it will be used yet. It would be a no-op but still not much point in landing that early. Also towards the end of the series is the API code to handle routing external events to multiple cells [6]. That's actually pretty lightweight and we could move it up to merge earlier, it's just not really something that benefits the in-tree functional tests (but is required to make the tempest tests work). **Hairy things** * The patch that adds the Instance.hidden field [7] is a bit complicated just because we have to account for it in a few places (not only listing instances but counting quotas). I have functional testing for all of that though to make people feel better. * The DB copy stuff in the TargetDBSetupTask is not really complicated but there are a lot of instance-related records to account for so I've tried to do my best there and written a pretty extensive set of tests for it. There are things we don't copy over like pci_devices and services since those are only relevant to other resources, like the compute node, in the source cell DB. Also, I messed around with doing the inserts in a single transaction like was done here [8] but couldn't really get it to work so I just left a TODO for now (maybe melwitt can figure it out). The single transaction insert is mostly low priority for me since if anything in there fails we hard destroy the target cell instance which will cascade delete any related records in the target cell DB. * The functional tests themselves can be a little daunting at first but I've tried to fully document those to make it clear what is being setup, what is happening, and what is being asserted. There are still TODOs for more testing to be done as I get more review (and time). * I have not written a docs patch yet since it could change as the series starts getting reviewed, and there is likely a lot to document (the spec is big, this email is big). When I do I will likely try to add a sequence diagram as well (and write one for normal resize since we don't have one of those in our docs today - we can do that whenever). If you have questions please reply to this email, ping me on IRC, or just ask in the code review - if you find I'm not replying in the review and rebasing over your comment, drop a -1 to make me notice (I'm currently up to 43 open changes so noticing random comments in a change in the middle is not obvious without a -1). If you made it this far, thanks, and remember that you asked for this review guide. :) [1] https://specs.openstack.org/openstack/nova-specs/specs/train/approved/cross… [2] https://youtu.be/OyNFIOSGjac?t=143 [3] https://review.opendev.org/#/c/641806/ [4] https://review.opendev.org/#/c/636253/ [5] https://review.opendev.org/#/c/656656/ [6] https://review.opendev.org/#/c/658478/ [7] https://review.opendev.org/#/c/631123/ [8] https://review.opendev.org/#/c/586742/ -- Thanks, Matt

1 0