November 2019 - openstack-discuss

[nova] Thoughts on exposing exception type to non-admins in instance action event
by Matt Riedemann 14 Nov '19

14 Nov '19

tl;dr: What do people think about storing and showing the *type* of exception that is recorded with a failed instance action event (like a fault) to the owner of the server who may not be an admin? Details: As noted here [1] and recreated here [2] the instance action event details that a non-admin owner of a server sees do not contain any useful information about what caused the failure of the action. Here is an example of a failed resize from that paste (this is what the non-admin owner of the server would see): $ openstack --os-compute-api-version 2.51 server event show vm2 req-11487504-da59-411b-b3b8-267bebe9b0d2 -f json -c events { "events": [ { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "cold_migrate", "result": "Error" }, { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "conductor_migrate_server", "result": "Error" } ] } Super useful, right? In this case scheduling failed for the resize so the instance is not in ERROR status which means the user cannot see a fault message with the NoValidHost error either. The admin can see the traceback in the failed action event list: $ openstack --os-compute-api-version 2.51 server event show 3ef043ea-e2d7-4565-a401-5c758e149f23 req-11487504-da59-411b-b3b8-267bebe9b0d2 -f json -c events { "events": [ { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "traceback": " File \"/opt/stack/nova/nova/conductor/manager.py\", line 301, in migrate_server\n host_list)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 367, in _cold_migrate\n raise exception.NoValidHost(reason=msg)\n", "event": "cold_migrate", "result": "Error" }, { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "traceback": " File \"/opt/stack/nova/nova/compute/utils.py\", line 1411, in decorated_function\n return function(self, context, *args, **kwargs)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 301, in migrate_server\n host_list)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 367, in _cold_migrate\n raise exception.NoValidHost(reason=msg)\n", "event": "conductor_migrate_server", "result": "Error" } ] } So when the admin gets the support ticket they can at least tell that scheduling failed and then dig into why. My idea is to store the exception *type* with the action event, similar to the recorded instance fault message for non-NovaExceptions [3] which will show to the non-admin owner of the server if the server status is ERROR or DELETED [4]. We should record the exc_val to get a prettier message like "No valid host was found." but that could leak details in the error message that we don't want non-admins to see [5]. With what I'm thinking, the non-admin owner of the server could see something like this for a failed event: { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "cold_migrate", "result": "Error", "details": "NoValidHost" } That's pretty simple, doesn't leak details, and at least indicates to the user that maybe they can retry the resize with another flavor or something. It's just an example. This would require a microversion so before writing a spec I wanted to get general feelings about this in the mailing list. I accept that it might not really be worth the effort so that's good feedback if it's how you feel (I'll only cry a little). [1] https://review.opendev.org/#/c/693937/2/nova/objects/instance_action.py [2] http://paste.openstack.org/show/786054/ [3] https://github.com/openstack/nova/blob/20.0.0/nova/compute/utils.py#L101 [4] https://github.com/openstack/nova/blob/20.0.0/nova/api/openstack/compute/vi… [5] https://bugs.launchpad.net/nova/+bug/1851587 -- Thanks, Matt

5 11

Order filters by cost
by Albert Braden 14 Nov '19

14 Nov '19

I'm working on a list of filters ordered by cost. This is what I have so far. Does this look reasonably correct for Rocky? Cheap Filters: AllHostsFilter - does no filtering. It passes all the available hosts. AvailabilityZoneFilter - filters hosts by availability zone. AggregateInstanceExtraSpecsFilter - checks aggregate metadata set with aggregate_instance_extra_specs. All hosts are passed if no extra_specs are specified. AggregateCoreFilter - filters hosts by CPU core number with per-aggregate cpu_allocation_ratio setting. AggregateRamFilter - filters hosts by RAM with per-aggregate ram_allocation_ratio setting. AggregateDiskFilter - filters hosts by disk allocation with per-aggregate disk_allocation_ratio setting. AggregateNumInstancesFilter - filters hosts by number of instances with per-aggregate max_instances_per_host setting. AggregateIoOpsFilter - filters hosts by I/O operations with per-aggregate max_io_ops_per_host setting. AggregateMultiTenancyIsolation - isolate tenants in specific aggregates. AggregateTypeAffinityFilter - limits instance_type by aggregate. AggregateImagePropertiesIsolation - isolates hosts based on image properties and aggregate metadata. DifferentHostFilter - allows the instance on a different host from a set of instances. SameHostFilter - puts the instance on the same host as another instance in a set of instances. ComputeFilter - passes all hosts that are operational and enabled. NumInstancesFilter - filters compute nodes by number of running instances. IoOpsFilter - filters hosts by concurrent I/O operations. More Expensive Filters: ServerGroupAntiAffinityFilter - This filter implements anti-affinity for a server group. ServerGroupAffinityFilter - This filter works the same way as ServerGroupAntiAffinityFilter. The difference is that when you create the server group, you should specify a policy of 'affinity'. ImagePropertiesFilter - filters hosts based on properties defined on the instance's image. Doc on setting image properties is here: https://docs.openstack.org/glance/rocky/admin/useful-image-properties.html IsolatedHostsFilter - filter based on isolated_images, isolated_hosts and restrict_isolated_hosts_to_isolated_images flags. SimpleCIDRAffinityFilter - allows a new instance on a host within the same IP block. MetricsFilter - filters hosts based on metrics weight_setting. Most Expensive Filters: PciPassthroughFilter - Filter that schedules instances on a host if the host has devices to meet the device requests in the 'extra_specs' for the flavor. ComputeCapabilitiesFilter - checks that the capabilities provided by the host compute service satisfy any extra specifications associated with the instance type. NUMATopologyFilter - filters hosts based on the NUMA topology requested by the instance, if any. JsonFilter - allows simple JSON-based grammar for selecting hosts. Deprecated Filters: Please don't use any of these; they are obsolete in Rocky: RetryFilter - filters hosts that have already been attempted for scheduling. Obsolete since Queens. RamFilter - filters hosts by their RAM. Obsolete since Pike. CoreFilter - filters based on CPU core utilization. Obsolete since Pike. DiskFilter - filters hosts by their disk allocation. Obsolete since Pike.

2 1

[all] requirements-check failures
by Sean McGinnis 14 Nov '19

14 Nov '19

Hey everyone, You may have noticed some odd failures with the requirements-check job on your patches lately. The requirements team is aware of this issue and are working to get it resolved ASAP. I believe things should be good again once https://review.opendev.org/694248 lands. So for the time being, please hold off on doing rechecks on these patches. This job should only be running for patches that touch any of the requirements files. As a workaround for now, if your patch can make a change without modifying requirements, that should bypass the need to run this job. Another alternative would be to add: Depends-on: https://review.opendev.org/694248 to your commit message, but hopefully that will not be necessary once this patch makes it through and the test is fixed. Sorry for the inconvenience this has caused. Sean

2 2

[nova][ptg] Resource provider delete at service delete
by Balázs Gibizer 14 Nov '19

14 Nov '19

ML thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007135.html Agreements in the room: * Check ongoing migration and reject the delete if migration with this compute having the source node exists. Let operator confirm the migrations * Cascade delete providers and allocations in placement. * in case of evacuated instances this is the right thing to do * in any other dangling allocation case nova has the final thrut so nova has the authority to delete them. * Document possible ways to reconcile Placement with Nova using heal_allocations and eventually the audit command once it's merged. Cheers, gibi

2 4

[nova][ironic] nova docs bug for ironic looking for an owner
by Matt Riedemann 14 Nov '19

14 Nov '19

While discussing some tribal knowledge about how ironic is the black sheep of nova compute drivers I realized that we (nova) have no docs about the ironic driver like we do for other drivers, so we don't mention anything about the weird cardinality rules around compute service : node : instance and host vs nodename things, how to configure the service for HA mode, how to configure baremetal flavors with custom resource classes, how to partition for conductor groups, how to deal with scaling issues, missing features (migrate), etc. I've opened a bug in case someone wants to get started on some of that information: https://bugs.launchpad.net/nova/+bug/1852446 -- Thanks, Matt

2 1

[tripleo] Adding Alex Schultz as OVB core
by Ben Nemec 14 Nov '19

14 Nov '19

Hi, After a discussion with Wes in Shanghai about how to make me less of a SPOF for OVB, one of the outcomes was that we should try to grow the OVB core team. Alex has been reviewing a lot of the patches to OVB lately and obviously has a good handle on how all of this stuff fits together, so I've added him to the OVB core team. Thanks and congratulations(?) Alex! :-) -Ben

2 1

[nova][oot] Adding `context` to ComputeDriver.unrescue
by Eric Fried 14 Nov '19

14 Nov '19

Though still very much WIP, emulated TPM [1] is looking like it will need to add the RequestContext to the ``unrescue`` ComputeDriver method [2]. This is a heads up to out-of-tree virt driver maintainers to keep an eye on this patch, as you will need to update your overrides accordingly once it merges. Thanks, efried [1] https://review.opendev.org/#/c/631363/ [2] https://review.opendev.org/#/c/631363/31/nova/virt/driver.py

1 0

[neutron] Review priorities
by Slawek Kaplonski 14 Nov '19

14 Nov '19

Hi neutrinos, According to our discussion during Train retrospective in Shanghai, I added "review-priority" label for neutron projects. It can be set by every core team member to values like: -1 - Branch Freeze +1 - Important Change +2 - Gate Blocker Fix / Urgent Change You can use dashboard like [1] to track such high priority patches and review them. I will also add some note about this to our docs this week to make it clear and visible for everyone. [1] https://tinyurl.com/vezk6n6 -- Slawek Kaplonski Senior software engineer Red Hat

3 2

[oslo] Adding Michael Johnson as Taskflow core
by Ben Nemec 14 Nov '19

14 Nov '19

Hi, After discussion with the Oslo team, we (and he) have agreed to add Michael as a Taskflow core. He's done more work on the project than anyone else still active in Oslo and also works on a project that consumes it so he likely understands it better than anyone else at this point. Welcome Michael and thanks for your contributions! -Ben

4 3

[stable][glance] Proposal to add Abhishek Kekane to glance-stable-maint
by Brian Rosmaita 14 Nov '19

14 Nov '19

I'd like to propose adding Abhishek Kekane to the Glance stable maintenance team. He's been a glance core for a few years now, and we are currently understaffed in glance-stable-maint. Plus, he's the current Glance PTL. cheers, brian

3 3