August 2019 - openstack-discuss - lists.openstack.org

[placement] update 19-32
by Chris Dent 16 Aug '19

16 Aug '19

HTML: https://anticdent.org/placement-update-19-32.html Here's placement update 19-32. There will be no update 33; I'm going to take next week off. If there are Placement-related issues that need immediate attention please speak with any of Eric Fried (efried), Balazs Gibizer (gibi), or Tetsuro Nakamura (tetsuro). # Most Important Same as last week: The main things on the Placement radar are implementing Consumer Types and cleanups, performance analysis, and documentation related to nested resource providers. A thing we should place on the "important" list is bringing the osc placement plugin up to date. We also need to discuss what would we would like the plugin to be. Is it required that it have ways to perform all the functionality of the API, or is it about providing ways to do what humans need to do with the placement API? Is there a difference? We decided that consumer types is medium priority: The nova-side use of the functionality is not going to happen in Train, but it would be nice to have the placement-side ready when U opens. The primary person working on it, tssurya, is spread pretty thin so it might not happen unless someone else has the cycles to give it some attention. On the documentation front, we realized during some performance work [last](https://review.opendev.org/675606) [week](https://review.opendev.org/#/c/676204/4/placement/tests/functional/ga… that it easy to have an incorrect grasp of how `same_subtree` works when there are more than two groups involved. It is critical that we create good "how to use" documentation for this and other advanced placement features. Not only can it be easy to get wrong, it can be challenge to see that you've got it wrong (the failure mode is "more results, only some of which you actually wanted"). # What's Changed * Yet more [performance fixes](https://review.opendev.org/#/q/topic:optimize-_build_provider_summar… are in the process of merging. Most of these are related to getting `_merge_candidates` and `_build_provider_summaries` to have less impact. The fixes are generally associated with avoiding duplicate work by generating dicts of reusable objects earlier in the request. This is possible because of the relatively new `RequestWideSearchContext`. In a request that returns many provider summaries `_build_provider_summaries` continues to have a significant impact because it has to create many objects but overall everything is much less heavyweight. More on performance in Themes, below. * The combination of all these performance fixes, and because of microversions, makes it reasonable for anyone running placement in a resource constrained environment (or simply wanting things to be faster) to consider running Train placement with _any_ release of OpenStack. Obviously you should test it first, but it is worth investigating. More information on how to achieve this can be found in the [upgrade to stein docs](https://docs.openstack.org/placement/latest/upgrade/to-stein.html) # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 23 (1) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 4 (1) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 4 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 11 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 18 (1). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (-1). # osc-placement osc-placement is currently behind by 12 microversions. * <https://review.opendev.org/666542> Add support for multiple member_of. There's been some useful discussion about how to achieve this, and a consensus has emerged on how to get the best results. * <https://review.opendev.org/640898> Adds a new '--amend' option which can update resource provider inventory without requiring the user to pass a full replacement for inventory. This has been broken up into three patches to help with review. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * <https://review.opendev.org/#/q/topic:bp/support-consumer-types> A WIP, as microversion 1.37, has started. As mentioned above, this is currently paused while other things take priority. If you have time that you could spend on this please respond here expressing that interest. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. As said above, there's lots of performance work in progress. We'll need to make a similar effort with regard to docs. For example, all of the coders involved in the creation and review of the `same_subtree` functionality struggle to explain, clearly and simply, how it will work in a variety of situations. We need to enumerate the situations and the outcomes, in documentation. One outcome of this work will be something like a _Deployment Considerations_ document to help people choose how to tweak their placement deployment to match their needs. The simple answer is use more web servers and more database servers, but that's often very wasteful. On the performance front, there is one major area of impact which has not received much attention yet. When requesting allocation candidates (or resource providers) that will return many results the cost of JSON serialization is just under one quarter of the processing time. This is to be expected when the response body is `2379k` big, and 154000 lines long (when pretty printed) for 7000 provider summaries and 2000 allocation requests. But there are ways to fix it. One is to ask more focused questions (so fewer results are expected). Another is to `limit=N` the results (but this can lead to issues with migrations). Another is to [use a different JSON serializer](https://review.opendev.org/674661). Should we do that? It make a _big_ difference with large result sets (which will be common in big and sparse clouds). # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:op…. There are two [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:… being discussed. And zero [os-resource-classes changes](https://review.opendev.org/#/q/project:openstack/os-resource-class…. # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * <https://review.openstack.org/#/q/topic:bug/1819923> Nova: nova-manage: heal port allocations * <https://review.opendev.org/659233> Cyborg: Placement report * <https://review.opendev.org/662229> helm: add placement chart * <https://review.opendev.org/634551> libvirt: report pmem namespaces resources by provider tree * <https://review.opendev.org/660852> Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * <https://review.opendev.org/670112> Nova: WIP: Add a placement audit command * <https://review.opendev.org/671312> blazar: Fix placement operations in multi-region deployments * <https://review.opendev.org/671793> Nova: libvirt: Start reporting PCPU inventory to placement A part of <https://review.opendev.org/#/q/topic:bp/cpu-resources * <https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports> Nova: support move ops with qos ports * <https://review.opendev.org/666202> Blazar: Create placement client for each request * <https://review.opendev.org/667952> nova: Support filtering of hosts by forbidden aggregates * <https://review.opendev.org/669079> blazar: Send global_request_id for tracing calls * <https://review.opendev.org/670696> tempest: Add placement API methods for testing routed provider nets * <https://review.opendev.org/672678> openstack-helm: Build placement in OSH-images * <https://review.opendev.org/674129> Correct global_request_id sent to Placement * <https://review.opendev.org/#/q/topic:bp/cross-cell-resize> Nova: cross cell resize * <https://review.opendev.org/674524> Nova: Scheduler translate properties to traits * <https://review.opendev.org/623558> Nova: single pass instance info fetch in host manager * <https://review.opendev.org/674708> Zun: [WIP] Claim container allocation in placement # End Have a good next week. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

1 0

kayobe 6.0.0.0rc1 (stein)
by no-reply＠openstack.org 16 Aug '19

16 Aug '19

Hello everyone, A new release candidate for kayobe for the end of the Stein cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/kayobe/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Stein release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/stein release branch at: https://opendev.org/openstack/kayobe/log/?h=stable/stein Release notes for kayobe can be found at: https://docs.openstack.org/releasenotes/kayobe/

1 0

[ironic] [stable] Proposal to add Riccardo Pittau to ironic-stable-maint
by Dmitry Tantsur 16 Aug '19

16 Aug '19

Hi folks! I'd like to propose adding Riccardo to our stable team. He's been consistently checking stable patches [1], and we're clearly understaffed when it comes to stable reviews. Thoughts? Dmitry [1] https://review.opendev.org/#/q/reviewer:%22Riccardo+Pittau+%253Celfosardo%2…

6 6

Re: [lists.openstack.org]Re: [nova] The race for 2.76
by Brin Zhang(张百林) 16 Aug '19

16 Aug '19

> On 8/13/2019 7:25 AM, Matt Riedemann wrote: > > There are several compute API microversion changes that are > > conflicting and will be fighting for 2.76, but I think we're trying to > > prioritize this one [1] for the ironic power sync external event > > handling since (1) Surya is going to be on vacation soon, (2) there is > > an ironic change that depends on it which has had review [2] and (3) > > the nova change has had quite a bit of review already. > > > > As such I think others waiting to rebase from 2.75 to 2.76 should > > probably hold off until [1] is approved which should happen today or > > tomorrow. > > > > [1] https://review.opendev.org/#/c/645611/ > > [2] https://review.opendev.org/#/c/664842/ > These are both approved now so let the rebasing begin! "Specifying az when restore shelved server" have been updated, and now it's in the nova-runway, please review, thanks. Links: https://review.opendev.org/#/q/topic:bp/support-specifying-az-when-restore-… -- Thanks, Matt

1 0

[nova] NUMA live migration is ready for review and testing
by Artom Lifshitz 16 Aug '19

16 Aug '19

tl;dr If you care about NUMA live migration, check out [1] and test in in your env(s), or review it. Over the months that I've worked on NUMA LM, I've been pinged by various folks that were interested in helping out. At this point I've addressed all the issues that were found at the end of the Stein cycle, and the series is ready for review and testing, with the aim of getting it merged in Train (for real this time). So if you care about NUMA-aware live migration and have some spare time and hardware (if you're in the former category I don't think I need to explain what kind of hardware - though I'll try to answer questions as best I can), I would greatly appreciate it if you deployed the patches and tested them. I've done that myself, of course, but, as at the end of Stein, I'm sure there are edge cases that I didn't think of (though I'm selfishly hoping that there aren't). I believe the series is also ready for review, though I haven't put it in the runway queue just yet because the last functional test patch is still a WIP, as I need to fiddle with it to assert more things. Thanks in advance, cheers! [1] https://review.opendev.org/#/c/672595/8

4 3

[qa][openstackclient] Debugging devstack slowness
by Clark Boylan 15 Aug '19

15 Aug '19

Today I have been digging into devstack runtime costs to help Donny Davis understand why tempest jobs sometimes timeout on the FortNebula cloud. One thing I discovered was that the keystone user, group, project, role, and domain setup [0] can take many minutes [1][2] (in the examples here almost 5). I've rewritten create_keystone_accounts to be a python tool [3] and get the runtime for that subset of setup from ~100s to ~9s [4]. I imagine that if we applied this to the other create_X_accounts functions we would see similar results. I think this is so much faster because we avoid repeated costs in openstack client including: python process startup, pkg_resource disk scanning to find entrypoints, and needing to convert names to IDs via the API every time osc is run. Given my change shows this can be so much quicker is there any interest in modifying devstack to be faster here? And if so what do we think an appropriate approach would be? [0] https://opendev.org/openstack/devstack/src/commit/6aeaceb0c4ef078d028fb6605… [1] http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output… [2] http://logs.openstack.org/05/672805/4/check/tempest-full/14f3211/job-output… [3] https://review.opendev.org/#/c/673108/ [4] http://logs.openstack.org/08/673108/6/check/devstack-xenial/a4107d0/job-out… Note the jobs compared above all ran on rax-dfw. Clark

11 23

[question][placement][rest api][403]
by 崔恒香 15 Aug '19

15 Aug '19

This is my problem with placement rest api.(Token and endpoint were OK) {"errors": [{"status": 403, "title": "Forbidden", "detail": "Access was denied to this resource.\n\n Policy does not allow placement:resource_providers:list to be performed. ", "request_id": "req-5b409f22-7741-4948-be6f-ea28c2896a3f" }]} Regards, Cuihx

2 1

[nova] The race for 2.76
by Matt Riedemann 15 Aug '19

15 Aug '19

There are several compute API microversion changes that are conflicting and will be fighting for 2.76, but I think we're trying to prioritize this one [1] for the ironic power sync external event handling since (1) Surya is going to be on vacation soon, (2) there is an ironic change that depends on it which has had review [2] and (3) the nova change has had quite a bit of review already. As such I think others waiting to rebase from 2.75 to 2.76 should probably hold off until [1] is approved which should happen today or tomorrow. [1] https://review.opendev.org/#/c/645611/ [2] https://review.opendev.org/#/c/664842/ -- Thanks, Matt

1 1

[glance] worker, thread, taskflow interplay
by Gregory Orange 15 Aug '19

15 Aug '19

We are trying to figure out how these two settings interplay in: [DEFAULT]/workers [taskflow_executor]/max_workers [oslo_messaging_zmq]/rpc_thread_pool_size Just setting workers makes a bit of sense, and based on our testing: =0 creates one process =1 creates 1 plus 1 child =n creates 1 plus n children Are there green threads (i.e. coroutines, per https://eventlet.net/doc/basic_usage.html) within every process, regardless of the value of workers? Does max_workers affect that? We have read some Glance doco, hunted about various bug reports[1] and other discussions online to get some insight, but I think we're not clear on it. Can anyone explain this a bit better to me? This is all in Rocky. Thank you, Greg. [1] https://bugs.launchpad.net/glance/+bug/1748916

2 1

[keystone] Feature proposal freeze exception for refreshable group membership
by Colleen Murphy 15 Aug '19

15 Aug '19

Work is in progress to implement refreshable group membership in keystone[1]. In order to allow for some breathing room for thorough discussion on the implementation details, we're proposing a 1-week extension to our scheduled feature proposal freeze deadline (this week)[2]. Please let me or the team know if you have any concerns about this. Colleen [1] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/train/ex… [2] https://releases.openstack.org/train/schedule.html

2 1