Nova Train PTG - Denver 2019 For forum session brainstorming use https://etherpad.openstack.org/p/DEN-train-nova-brainstorming Attendance: efried sean-k-mooney,sean-k-mooney aspiers stephenfin takashin helenafm gmann Sundar mriedem gibi melwitt alex_xu mdbooth lyarwood tssurya kashyap (first two days will be sparsely available; be present fully on the last day) artom egallen dakshina-ilangov (joining post 11:30AM on Thur, Fri) jaypipes adrianc IvensZambrano johnthetubaguy (afternoon thursday, onwards) amodi gryf cfriesen (bouncing around rooms a bit) med_ mnestratov shuquan bauzas dklyle jichenjc sorrison jgasparakis tetsuro Team photo Friday 11:50-1200 https://ethercalc.openstack.org/3qd1fj5f3tt3 Topics - Please include your IRC nick next to your topic so we know who to talk to about that topic. NUMA Topology with placement Spec: https://review.openstack.org/#/c/552924/ XPROJ see https://etherpad.openstack.org/p/ptg-train-xproj-nova-placement Subtree affinity with placement [efried] XPROJ: see https://etherpad.openstack.org/p/ptg-train-xproj-nova-placement completing numa affinity polices for neutron sriov interfaces. ==> neutron XPROJ (efried 20190422) https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/share-pci-between-numa-nodes.html ^ dose not work for nutron port as the flavor extra specs and image properties were removed during the implematnion and the spec was retroativly updated to document what was implemented. we should fix that by supportin numa polices TODO sean-k-mooney to write blueprint/spec either reporpose original spec or explore using the new port requestes/traits mechanisium. cpu modeling in placement jays spec https://review.openstack.org/#/c/555081/ #agree approve the spec more or less as is and get moving how to make it work with numa affinity and cache RMD - Resource Management Daemon (dakshina-ilangov/IvensZambrano) Base enablement - https://review.openstack.org/#/c/651130/ The following blueprints reference the base enablement blueprint above Power management using CPU core P state control - https://review.openstack.org/#/c/651024/ Last-level cache - https://review.openstack.org/#/c/651233/ #agree Generic file (inventory.yaml?) allowing $daemon (RMD) to dictate inventory to report, which can be scheduled via extra_specs #agree RMD to monitor (by subscribing to nova and/or libvirt notifications) and effect assignments/changes out-of-band - no communication from virt to RMD resource provider yaml (or how we whitelist/model host resources via config in general) https://review.openstack.org/#/c/612497/ Code: https://review.openstack.org/#/c/622622/ [efried 20190418] scrubbing from agenda for general lack of interest AMD SEV support efried 20190424 - removing from agenda because approved Any matters arising (if we're lucky, there won't be any) Train spec: https://review.opendev.org/#/c/641994/ Note this is significantly different from the... Stein spec: https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/amd-sev-libvirt-support.html ...in that we're now using a resource class and making SEV contexts a quantifiable resource. "Guest CPU selection with hypervisor consideration" (kashyap) efried 20190425 -- removing from agenda because spec/bp approved Blueprint: https://blueprints.launchpad.net/nova/+spec/cpu-selection-with-hypervisor-consideration Spec: https://review.openstack.org/#/c/645814/ tl;dr: Re-work (for the better) the way Nova's libvirt driver chooses CPU models. Problem: Currently the CPU configuration APIs that Nova's libvirt driver uses — baselineCPU() and compareCPU() — ignore the host hypervisor's (QEMU + KVM) capabilities when determining guest CPU model To solve that, libvirt has introduced two new APIs that are "hypervisor-literate" — baselineHypervisorCPU() and compareHypervisorCPU(). These newer APIs (requires: libvirt-4.0.0; and QEMU-2.9 — both for x86_64) take into account the hypervisor's capabilities, and are therefore much more useful. This addresses several problems (along multiple TODOs item the libvirt driver code (refer to _get_guest_cpu_model_config() and _get_cpu_traits() methods in libvirt/driver.py) Reference: Slide-28 here: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf Making extra specs less of a forgotten child (stephenfin) spec: https://review.openstack.org/#/c/638734/ Unlike config options, we have no central reference point for flavour extra specs. There are a *lot* of them and I frequently see typos, people setting them wrong etc. We don't? https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs Do you intend to make this exhaustive / comprehensive / exclusive on the first pass (i.e. unknown/unregistered items trigger an error rather than being allowed)? I'd like this to be configurable, though I'm not sure if that's allowed (no to configurable APIs) so maybe warning-only first Glance supports metadata definitions but the definitions look old https://github.com/openstack/glance/blob/master/etc/metadefs/compute-cpu-pinning.json i think the way that the glance metadata refence the type of resouce they refer to (flavor,image,volume,host aggragate) is also tied into how heat references them. the glance metadef my be old but they are used to generate ui and validation logic in horizon too. they are availabe via a glance api endpoint which is how they are consumed by other services. There are also several missing metadefs and documented image properties. https://developer.openstack.org/api-ref/image/v2/metadefs-index.html Do we want to start validating these on the API side (microversion) and before an instance boots? Rough PoC for flavour extra spec definition here: http://paste.openstack.org/show/B9unIL8e2KpeSMBGaINe/ extracting the metadef into an external lib that is importable by several service may be useful Not entirely sure if this is necessary. They change very rarely and it could be just as easy to have a "nova metadef" -> "glance metadef" translation tool. Worth discussing though We have json schema for scheduler hints but still allow undefined out of tree scheduler hints, do something like that? https://github.com/openstack/nova/blob/c7f4190/nova/api/openstack/compute/schemas/servers.py#L93 Seems like trying to build a new metadefs type API for flavor extra specs in nova would take a lot longer than simply doing json schema validation of the syntax for known extra specs (like scheduler hints). (Sundar) +1 to two ideas: (a) Keep a standard list of keys which operators cannot modify and allow operators to add more to the schema (b) Use a new microversion to enforce strict extra specs key checking. #agree do it as part of flavor property set #agree first do value side validation for known keys, then validate keys, allowing admin ability to augment the schema of valid keys/values #agree validate the values without a microversion - doesn't fix the fat finger key issue, but helps with value validation (and doesn't need to all land at once) probably need something later for validating keys (strict mode or whatever) Persistent memory (alex_xu) spec https://review.openstack.org/601596, https://review.openstack.org/622893 patches: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/virtual-persistent-memory NUMA rears its ugly head again #agree can "ignore NUMA" initially #agree Document lifecycle ops that are and aren't supported New virt driver for rsd: For management of a composable infrastructure bp: https://blueprints.launchpad.net/nova/+spec/rsd-virt-for-nova-implementation rsd-virt-for-nova virt driver: https://github.com/openstack/rsd-virt-for-nova spec review: https://review.openstack.org/#/c/648665/ Questions Can this be done with nova + ironic? Why can't the driver live out of tree? What's the 3rd party CI story? Nova governance (stephenfin) In light of the placement split and massively reduced nova contributor base, it would be good to take the time to examine some of the reasons we do what we do and why...on a day that isn't the last day [efried] how about right after the retrospective on Thursday morning? Do we need more than 15 minutes for this? Right after the retro is fine. It's very subjective and I don't expect to make any decisions on the day. More of a sharing session. Ideas cores vs. spec cores This is done: https://review.openstack.org/#/admin/groups/302,members nova cores vs. neutron cores (the subsystem argument) two +2s from same company (for a change from someone in the same company?) (mriedem): I'm still a -1 on this. Multiple problems with this e.g. reviewing with the blinders and pressure to deliver for your downstream product roadmap ("we'll fix it downstream later") and I as a nova maintainer don't want to be responsible for maintaining technical debt rushed in by a single vendor. (kashyap) While I see where yuo're coming from, your comment implies mistrust and that people will intentionally "rush things in". As long as a particular change is publicly advertized well-enough, gave sufficient time for others to catch up, all necessary assmuptions are described clearly, and respond in _every_ detail that isn't clear to a community reviewer, then it is absolutely reasonable for reviewers from a comapny to merge a change by a contributor from the same company. This happens _all_ time in other mature open source communities (kernel, QEMU, et al). (adrianc) -1 on that, diversty is more likely to ensure community goals. (adrianc) perhaps a happy middle e.g bugfixes ? (mdbooth): The concern above is completely valid, but I don’t believe it will happen in practise, and in the meantime we’re making it harder on ourselves. I would like to trust our core reviewers, and we can address this if it actually happens. (+1)+1 +1 separate specs repo vs. in-tree specs directory more effort than it's worth (anything else that nova does differently to other OpenStack projects and other large, multi-vendor, open source projects) Compute capabilities traits placement request filter (mriedem) Solution for https://bugs.launchpad.net/nova/+bug/1817927 and other things like booting from a multi-attach volume, we need the scheduler to pick a host with a virt driver that supports those types of requests. Think we agreed in https://etherpad.openstack.org/p/ptg-train-xproj-nova-placement toward the bottom that we're OK with this. Here's some code: https://review.opendev.org/#/c/645316/ (gibi): I'm OK to modify the flavor extra_spec for now. I think we agreed yestarday to allow numbered group without resources in placement as a final solution. It is also OK to me. However we have a comment about storing the unnumbered request group in RequestSpec.requested_resources list. (https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L93) I tried to do that to give a place where the capability traits can be stored, but failed: https://review.opendev.org/#/c/647396/5//COMMIT_MSG . Is there any reason to still try to store the unnumbered group in the RequestSpec.requested_resources? How do you like this hack? https://review.opendev.org/#/c/656885/3/nova/scheduler/manager.py (not traits related) [dtroyer][sean-k-mooney] 3rd party CI for NUMA/PCI/SRIOV (mriedem) moved to https://etherpad.openstack.org/p/nova-ptg-train-ci Corner case issues with root volume detach/attach (mriedem/Kevin_Zheng) Go over the stuff that came up late in the Stein cycle: tags and multiattach volumes: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003376.html https://etherpad.openstack.org/p/detach-attach-root-volume-corner-cases When rebuilding with a new image we reset the stashed image_* system_metadata on the instance and some other fields on the instance based on new image metadata. When attaching a new root volume, the underlying image (and its metadata) could change, so presumably we need to do the same type of updates to the instance record when attaching a new root volume with a new image - agree? The state of nova's documentation (stephenfin) There are numerous issues with our docs Many features aren't documented or are barely documented in nova. I've cleaned up some but there's much more to do metadata (the metadata service, config drives, vendordata etc.), console proxy services, man pages, cross_az_attach: https://review.opendev.org/#/c/650456/ Flow diagram for resize like for live migration https://docs.openstack.org/nova/latest/reference/live-migration.html Loads of stuff is out-of-date If you're a mere user of nova, the docs are essentially useless as admin'y stuff is scattered everywhere Other stuff Testing guides down cells: https://review.opendev.org/#/c/650167/ Before there's serious time sunk into this, does anyone really care and should it be a priority? (kashyap) I enjoy improving documentation, so, FWIW, in my "copious free time" I am willing to help chip in with areas that I know a thing or two about. Broader topic: how can we get our respective downstream documentation teams to engage upstream?+∞ (kashyap) A potential first step is to agree on a "system" (and consistently stick to it). E.g. the "Django" project's (IIRC, Stephen even mentioned this in Berlin) documentation model (described here: https://www.divio.com/blog/documentation/) Tutorials — learning oriented How-To guides — problem-oriented Explanation — understanding-oriented Reference — information-oriented (mriedem): I try to push patches to fix busted / incorrect stuff or add missing things when I have the context (I'm looking through our docs for some specific reason). If I don't have the time, I'll report a bug and sometimes those can be marked as low-hanging-fruit for part time contributors to work on those. e.g. https://bugs.launchpad.net/nova/+bug/1820283 TODO(stephenfin): Consider making this a mini cycle goal Tech debt: Removing cells v1 Already in progress \o/ Removing nova-network CERN are moving off this entirely (as discussed on the mailing list). We can kill it now? (melwitt): We have had the go ahead [from CERN] since Stein to remove nova-network entirely. \o/ 🎉 Can we remove the nova-console, nova-consoleauth, nova-xvpxvncproxy service? (mriedem, stephenfin) The nova-console service is xenapi-specific and was deprecated in stein: https://review.openstack.org/#/c/610075/ There are, however, REST APIs for it: https://developer.openstack.org/api-ref/compute/#server-consoles-servers-os-consoles-os-console-auth-tokens (stephenfin): Maybe I misunderstood you, but I thought these APIs could also talk to the DB stuff Mel did? So if we drop the nova-console service, the APIs would no longer work. It seems our options are: Delete the nova-console service and obsolete the REST APIs (410 response on all microversions like what we're doing with nova-cells and nova-network APIs) Deprecate the REST APIs on a new microversion but continue to support older microversions - this means the nova-console service would live forever. Are people still using the nova-console service? Are there alternatives/replacements for xen users? BobBall seemed to suggest there was http://lists.openstack.org/pipermail/openstack-dev/2018-October/135422.html but it's not clear to me. Matt DePorter (Rackspace) said this week that they rely on it - but they are on Queens and not sure if there are alternatives (as Bob suggested in the ML). But they're also on cellsv1, so they have work to do to upgrade anyway, so they might as well move to whatever isn't these things we want to delete. What is that? So upgrade and migrate from xen to kvm? #agree: Do it Migrating rootwrap to privsep #agree: Continue getting an MVP since it's not any worse than what we have and mikal is doing the work Bumping the minimum microversion Did this ever progress in ironic? #agree: nope imagebackend/image cache another go? (mdbooth) (mriedem): How could we have real integration testing for this before refactoring it? Not tempest, just a minimal devstack with (for lack of a better word) exercises. Why not tempest? Too heavy? Tempest tests the API, you need low-level testing of the cache to see if it's doing what you expect. Whitebox? I refactored the unit tests to be somewhat functional-y at the time for exactly this reason. This was before functional was a thing. Clean up rest of the needless libvirt driver version constants and compat code(kashyap) Mostly an FYI (as this is a recurring item) WIP: https://review.opendev.org/#/q/topic:Bump_min_libvirt_and_QEMU_for_Stein+(status:open+OR+status:merged) (kashyap) Some more compat code to be cleaned up, it's noted at the end of this (merge) change: https://review.opendev.org/#/c/632507/ Remove mox (takashin) https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/mox-removal-train Remove explicit eventlet usage (preferably entirely), specifically to allow wsgi to no longer require it. (mdbooth) Removing virt drivers that no longer has third-party CI (cough XenAPI cough) Removing fake libvirt driver (we use that! mdbooth) "we" who? It's basically a hack that's in place only because mocking wasn't done properly. mikal has a series to do the mocking properly and remove that driver. Link: https://review.opendev.org/#/q/topic:fake-cleanup+(status:open+OR+status:merged) Ah, fake libvirt *driver*. The fake libvirt module is used in functional (by the real libvirt driver). Fixing OSC's live migrate interface (mriedem) OSC CLI is not like nova and defaults to 2.1 unless the user overrides on the CLI or uses an environment variable. The "openstack server migrate --live " CLI therefore essentially makes all live migrations by default forced live migrations, bypassing the scheduler, which is very dangerous. Changing the interface is likely going to have to be a breaking change and major version bump, but it needs to happen and has been put off too long. Let's agree on the desired interface, taking into account that you can also specify a host with cold migration now too, using the same CLI (openstack server migrate). See https://review.openstack.org/#/c/627801/ and the referenced changes for attempts at fixing this. (dtroyer) IIRC at least part of this can be done without breaking changes and should move forward. But yeah, its a mess and time to fix it... More details in the Forum session etherpad: https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps See the ML summary: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005783.html Let's plan the next steps of the bandwidth support feature (gibi) [efried: XPROJ? does this need to involve neutron folks? If so, please add to https://etherpad.openstack.org/p/ptg-train-xproj-nova-neutron] (gibi): these items are mostly nova only things but I added a XPROJ item to the Neutron pad about multisegement support. Obvious next step is supporting server move operations with bandwidth: https://blueprints.launchpad.net/nova/+spec/support-server-move-operations-with-ports-having-resource-request spec: https://review.opendev.org/#/c/652608 Question: Do we want to add support for server move operations with port having resource request as a new API microversion or as bug fixes? (gibi) background from ML http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001881.html Server delete and detach port works without require a specific microversion Server create works since microversion 2.72 Server move operations rejected since https://review.openstack.org/#/c/630725 #agree : no microversion (will be proposed in the spec, and allow people to object there) Question: Can the live migration support depend on the existence of multiple portbinding or we have to support the old codepath as well when the port binding is created by the nova-compute on the destination host? #agree: Yes, this extension cannot be turned off But there are various smaller and bigger enchancements. Tracked in https://blueprints.launchpad.net/nova/+spec/enhance-support-for-ports-having-resource-request Which one seems the most imporant to focus on in Train? Use placement to figure out which RP fulfills a port resource request (currently it is done by Nova). This requires the Placement bp https://blueprints.launchpad.net/nova/+spec/placement-resource-provider-request-group-mapping-in-allocation-candidates to be implemented first. A consensus is emerging to do the '"mappings" dict next to "allocations"' solution+1+1 Supporting SRIOV ports with resource request requires virt driver support (currently supported by libvirt driver) to include the parent interface name to the descriptor of the PCI device represents VFs. Introduce a new TRAIT based capability to the virt drivers to report if they support SRIOV port with resource request to be able to drive the scheduling of server using such ports. Today only the pci_claim stops a boot if the virt driver does not support the feature and that leads to reschedule. #agree: add a new capablity as a trait Automatically set group_policy if more than one RequestGroup is generated for an allocation_candidate query in nova. This first needs an agreement what is a good default value for such policy. #agree: 'none' seems to be a sensible policy state that default on the nova side, not placement (which wants explicit) # agree: priority order: 1) group_policy, 2) capability trait, 3) port mapping (gibi) The rest is a long shot in Train but I added them for completness: Support attaching a port to a server where the port has resource request. This needs a way to increase the allocation of the running servers. So this requires the in_tree allocation candidate support from placement that was implemented in Stein https://blueprints.launchpad.net/nova/+spec/alloc-candidates-in-tree Also this operation can only be supported if the new, increased allocation still fits to the current compute the server is running on. Support attaching a network to a server where the network has a (default) QoS minimum bandwidth rule. Support creating a server with a network that has a (default) QoS minimum bandwidth rule. This requires to move the port create from the nova-compute to the nova-conductor first. Changing how server create force_hosts/nodes works (mriedem) Spec: https://review.openstack.org/#/c/645458/ (merged) Blueprint https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server approved Code https://review.openstack.org/#/c/645520/ started See discussion in the mailing list: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003813.html API change option: Add a new parameter (or couple of parameters) to the server create API which would deprecate the weird az:host:node format for forcing a host/node and if used, would run the requested destination through the scheduler filters. This would be like how cold migrate with a target host works today. If users wanted to continue forcing the host and bypass the scheduler, they could still use an older microversion with the az:host:node format. Other options: config option or policy rule Integrating openstacksdk and replacing use of python-*client blueprint: https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova code: openstacksdk patch to support ksa-conf-based connection construction: https://review.openstack.org/#/c/643601/ Introduce SDK framework to nova (get_sdk_adapter): https://review.opendev.org/#/c/643664/ WIP use openstacksdk instead of ksa for placement: https://review.opendev.org/#/c/656023/ WIP/PoC start using openstacksdk instead of python-ironicclient: https://review.openstack.org/#/c/642899/ questions: Community and/or project goal? Move from one-conf-per-service to unified conf and/or clouds.yaml How does the operator tell us to do this? Config options for location of clouds.yaml This should be in [DEFAULT] since it'll apply to all services (that support it) Which cloud region (is that the right term?) from clouds.yaml to use specifying this option would take precedence, ignore the ksa opts, and trigger use of clouds.yaml or perhaps a [DEFAULT] use_sdk_for_every_service_that_supports_it_and_use_this_cloud_region Process (blueprints/specs required?) API inconsistency cleanup (gmann) There are multiple API cleanup were found which seems worth to fix. These cleanups are API change so need microversion bump. Instead of increasing microversion separatly for each cleanup, I propose to be fix them under single microversion bump. Current list of cleanup - https://etherpad.openstack.org/p/nova-api-cleanup #. 400 for unknown param for query param and for request body.http://lists.openstack.org/pipermail/openstack-discuss/2019-May/ Consensus is sure do this. #. Remove OS-* prefix from request and response field. Alternative: return both in response, accept either in request Dan and John are -1 on removing the old fields If you're using an SDK it should hide this for you anyway. Consensus in the room is to just not do this. #. Making server representation always consistent among all APIs returning the complete server representation. GET /servers/detail GET /servers/{server_id} PUT /servers/{server_id} POST /servers/{server_id} (rebuild) Consensus in the room is this is fine, it's just more fields in the PUT and rebuild responses. #. Return ``servers`` field always in response of GET /os-hypervisors this was nacked/deferred (i.e. not to be included in same microversion as above) Consensus: do it in the same microversion as the above #. Consistent error codes on quota exceeded this was nacked/deferred Spec - https://review.openstack.org/#/c/603969/ Do we want to also lump https://review.opendev.org/#/c/648919/ (change flavors.swap default from '' [string] to 0 [int] in the response) into gmann's spec? It's a relatively small change. +1 https://github.com/openstack/nova/blob/11de108daaab4a70e11f13c8adca3f5926aeb540/nova/api/openstack/compute/views/flavors.py#L55 Consensus is yeah sure let's do this, clients already have to handle the empty string today for older clouds. This just fixes it for newer clouds. Libvirt + block migration + config drive + iso9660 Would like another option besides enabling rsync or ssh across all compute nodes due to security concerns In our specific case we dont use any of the --files option when booting vm's. We would like to be able to just regenerate the config drive contents on the destination side. Instead of copying the existing config drive. This is for live migration, cold migration, or both? (mriedem): Who is "we"? GoDaddy? Secure Boot support for QEMU- and KVM-based Nova instances (kashyap) Blueprint: https://blueprints.launchpad.net/nova/+spec/allow-secure-boot-for-qemu-kvm-guests Spec (needs to be refreshed): https://review.openstack.org/#/c/506720/ (Add UEFI Secure Boot support for QEMU/KVM guests, using OVMF) Use case: Prevent guests from running untrusted code ("malware") at boot time. Refer to the periodic updates I posted in the Nova specification over the last year, as various pieces of work in lower layers got completed Upstream libvirt recently (13 Mar 2019) merged support for auto-selecting guest firmware: https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=1dd24167b ("news: Document firmware autoselection for QEMU driver") NOTE: With the above libvirt work in place, Nova should have all the pieces ready (OVMF, QEMU, and libvirt) to integrate this. PS: Nova already has Secure Boot support for HyperV in-tree (http://git.openstack.org/cgit/openstack/nova/commit/?id=29dab997b4e) based on this: https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/hyper-v-uefi-secureboot.html Action: Kashyap to write a summary brief to the mailing list John Garbutt will look at the spec Securing privsep (mdbooth) Privsep isn't currently providing any security, and is arguably worse than rootwrap: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004358.html Support filtering of allocation_candidates by forbidden aggregates (tpatil) Specs: https://review.opendev.org/#/c/609960/ #action tpatil to answer questions on spec Allow compute nodes to use DISK_GB from shared storage RP (tpatil) [efried 20190422 XPROJ placement] Specs: https://review.opendev.org/#/c/650188/ Disabled compute service request filter (mriedem) https://bugs.launchpad.net/nova/+bug/1805984 related bug on using affinity with limits: https://bugs.launchpad.net/nova/+bug/1827628, we could just do a more generic solution for both. Modeling server groups in placement is a longer-term thing that requires some thought. Could pre-filter using in_tree filter in the strict affinity case, but that only works if there are already members of the group on a host (I think). Move affinity to given host in tree anti-affinity, retry a few times to see if you get lucky https://www.youtube.com/watch?v=mBluR6cLxJ8 PoC using forbidden trait and a request filter: https://review.opendev.org/#/c/654596/ Should we even do this? It would mean we'd have two source of truth about a disabled compute and they could get out of sync (we could heal in a periodic on the compute but still). Using a trait for this sort of violates the "traits should only be capabilities" thing "capable of hosting instances" seems like a fundamental capability to me https://review.opendev.org/#/c/623558/ attempts to address the reason why CERN limits allocation candidates to a small number (20) (should be fixed regardless) If one of the goals with nova's use of placement is to move as many python scheduler filters into placement filtering-in-sql, this would seem to align with that goal. How to deal with rolling upgrades while there are older computes? Should the API attempt to set the trait if the compute is too old to have the new code (and remove that in U Alternatives: Use a placement aggregate for all disabled computes and filter using negative member_of: https://docs.openstack.org/placement/latest/placement-api-microversion-history.html#support-forbidden-aggregates We'd have to hard-code the aggregate UUID in nova somewhere. This could be hard to debug for an operator since we can't put a meaningful name into a hex UUID. I liked your suggestion: d15ab1ed-dead-dead-dead-000000000000 Update all resource class inventory on the compute node provider and set reserved=total (like the ironic driver does when a node is in maintenance): https://github.com/openstack/nova/blob/fc3890667e4971e3f0f35ac921c2a6c25f72adec/nova/virt/ironic/driver.py#L882 Might be a pain if we have to update all inventory classes. Baremetal providers are a bit easier since they have one custom resource class. What about nested provider inventory? Unable to configure this behavior like a request filter since it's messing with inventory. The compute update_provider_tree code would have to be aware of the disabled status to avoid changing the reserved value. #agree: Create a standard trait like COMPUTE_DISABLED and add required=!COMPUTE_DISABLED to every request TODO: mriedem to push a spec #agree: tssurya to push up the CERN downstream fix for bug 1827628 as the backportable fix for now (affinity problem) TODO: tssurya will push a spec for the feature Clean up orphan instances: https://review.opendev.org/#/c/627765/ (yonglihe, alex_xu) Problem: Even though splited, still long and boring. Need fully review. Last time discussion link: https://etherpad.openstack.org/p/nova-ptg-stein L931 Who could help on libvirt module? (mriedem): I've reviewed the big change before the split, and am still committed to reviewing this, just haven't thought about it lately - just ping me to remind me about reviews (Huawei also needs this). I don't think we really need this as a PTG item. (melwitt): I can also help continue to review. I don't think it's a bug (defect) but the launchpad bug has been changed to Wishlist so I guess that's OK. Will need a release note for it. The change is naturally a bit complex -- I haven't gotten around to reviewing it again lately. (johnthetubaguy) having attempted this manually recently, I want to review this too, if I can (gibi): I feel that this periodic cleanup papers over some bugs in nova results in orphans. Can we fix try to fix the root cause / original bug? TODO: https://review.opendev.org/#/c/556751/ so you can archive everything before a given time (not recent stuff). Might help with the case that you archived while a compute was down so the compute wasn't able to delete the guest on compute while it was still in the DB. Question: can this be integrated into the existing _cleanup_running_deleted_instances periodic task with new values for the config option, e.g. reap_with_orphans? Rather than mostly duplicating that entire periodic for orphans. StarlingX Add server sub-resource topology API https://review.opendev.org/#/c/621476/ (yonglihe, alex_xu) Problem: Inernal NUMA information of NOVA kind of too complex for end user, need to expose a clear well defined, understandable information. How we define the infomation is Open: a) Starting from current bp, elimated the fuzzy one, keep the clear one b) Come up with a new set of data, if we have a clear model for all that stuff. discussion link: https://docs.google.com/document/d/1kRRZFq_ha0T9mFDOEzv0PMvXgtnjGm5ii9mSzdqt1VM/edit?usp=sharing (alex) remove the cpu topology from the proposal or just move that out of numa topology? only using the cpupinning info instead of cpuset? hugepage is per Numa node or not? bp: https://blueprints.launchpad.net/nova/+spec/show-server-numa-topology Last time discussion link: https://etherpad.openstack.org/p/nova-ptg-stein L901 Who could help on NUMA module? StarlingX Briefly discuss idea for transferring ownership of nova resources (melwitt) Want to run the idea by the team and get a sanity check or yea/nay From the "Change ownership of resources - followup" session from Monday: https://etherpad.openstack.org/p/DEN-change-ownership-of-resources Idea: build upon the implementation in https://github.com/kk7ds/oschown/tree/master/oschown Each project (nova, cinder, neutron) has its own dir containing the code related to transferring ownership of its resources, to be available as a plugin This way, each project is responsible for providing test coverage, regression testing, upgrade testing (?) of their ownership transfer code. This is meant to address concerns around testing and maintenance of transfer code over time and across releases Then, something (micro service that has DB creds to nova/cinder/neutron or maybe an adjuntant workflow) will load plugins from all the projects and be able to carry out ownership changes based on a call to its REST API AGREE: sounds reasonable, melwitt to talk to tobberydberg and send summary to ML and figure out next steps Reduce RAM & CPU quota usage for shelved servers (mnestratov) - this is actually superseded by https://review.opendev.org/#/c/638073/ Spec https://review.opendev.org/#/c/656806/ there was a bug closed as invalid https://bugs.launchpad.net/nova/+bug/1630454 proposing to create spec StarlingX Reviews RBD: https://review.opendev.org/#/c/640271/, https://review.opendev.org/#/c/642667/ auto-converge spec: https://review.opendev.org/#/c/651681/ vCPU model:spec: https://review.openstack.org/#/c/642030/ NUMA aware live migration This needs fixing first :( #action: Prioritize these for review somehow (runway, gerrit priorities, ...) Thursday: 0900-0915: Settle, greet, caffeinate 0915-0945: Retrospective https://etherpad.openstack.org/p/nova-ptg-train-retrospective 0945-1000: Nova governance (stephenfin) 1000-1030: cpu modeling in placement 1030-1100: Persistent memory (alex_xu, rui zang) 1100-1130: Support filtering of allocation_candidates by forbidden aggregates (tpatil) 1115-1145: Corner case issues with root volume detach/attach (mriedem/Kevin_Zheng) 1145-1215: Making extra specs less of a forgotten child (stephenfin) 1215-1230: The state of nova's documentation (stephenfin) 1230-1330: Lunch 1330-1400: Let's plan the next steps of the bandwidth support feature (gibi) 1400-1430: RMD - Resource Management Daemon Part I (dakshina-ilangov/IvensZambrano) 1430-1445: Integrating openstacksdk and replacing use of python-*client (efried, dustinc, mordred) 1445-1500: RMD - Resource Management Daemon Part II (dakshina-ilangov/IvensZambrano) 1500-beer: Placement XPROJ: https://etherpad.openstack.org/p/ptg-train-xproj-nova-placement (ordered as shown in etherpad) Friday: 0900-1000: Cyborg XPROJ (Ballroom 4!): https://etherpad.openstack.org/p/ptg-train-xproj-nova-cyborg 1015-1115: Ironic XPROJ: https://etherpad.openstack.org/p/ptg-train-xproj-nova-ironic 1115-1150: Cinder XPROJ: https://etherpad.openstack.org/p/ptg-train-xproj-nova-cinder in the Cinder room 203 they broadcast via microphones so not so portable. 1150-1200: Team Photo https://ethercalc.openstack.org/3qd1fj5f3tt3 1200-1230: API inconsistency cleanup (gmann) 1230-1330: Lunch 1330-1400: Glance topics - todo: dansmith to summarize the idea in the ML 1400-1515: Neutron XPROJ: https://etherpad.openstack.org/p/ptg-train-xproj-nova-neutron *1430-1440: Placement team picture 1515-1615: Keystone XPROJ: https://etherpad.openstack.org/p/ptg-train-xproj-nova-keystone 1615-1630: Compute capabilities traits placement request filter (mriedem) (aspiers sched) 1630-beer: Disabled compute service request filter (mriedem) (aspiers sched) Saturday: 0900-1000: [dtroyer][sean-k-mooney] 3rd party CI for NUMA/PCI/SRIOV (mriedem) https://etherpad.openstack.org/p/nova-ptg-train-ci 1000-1030: Clean up orphan instances, Add server sub-resource topology API (yonglihe, alex_xu) 1030-1045: Tech debt 1115-1130: Securing privsep (mdbooth) 1100-1115: StarlingX patches 1115-1130: Secure Boot support for QEMU- and KVM-based Nova instances (kashyap sched) 1130-1200: New virt driver for rsd 1200-1230: Train Theme setting https://etherpad.openstack.org/p/nova-train-themes 1230-1330: Lunch 1330-1345: Governance (single-company patch+approval, trusting cores) cont'd 1345-beer: Continue deferred discussions Deferred Governance: two cores from same company Mdbooth proposed words: https://etherpad.openstack.org/p/nova-ptg-train-governance mini-cores: can we trust them to be SMEs and not just shove in new code? (Isn't that the same question we ask for other cores?)