May 2019 - openstack-discuss - lists.openstack.org

[nova][ironic] Lock-related performance issue with update_resources periodic job
by Jason Anderson 09 Jul '19

09 Jul '19

Hey OpenStackers, I work on a cloud that allows users to reserve and provision bare metal instances with Ironic. We recently performed a long-overdue upgrade of our core components, all the way from Ocata up through Rocky. During this, we noticed that instance build requests were taking 4-5x (!!) as long as before. We have two deployments, one with ~150 bare metal nodes, and another with ~300. These are each managed by one nova-compute process running the Ironic driver. After investigation, the root cause appeared to be contention between the update_resources periodic task and the instance claim step. There is one semaphore "compute_resources" that is used to control every access within the resource_tracker. In our case, what was happening was the update_resources job, which runs every minute by default, was constantly queuing up accesses to this semaphore, because each hypervisor is updated independently, in series. This meant that, for us, each Ironic node was being processed and was holding the semaphore during its update (which took about 2-5 seconds in practice.) Multiply this by 150 and our update task was running constantly. Because an instance claim also needs to access this semaphore, this led to instances getting stuck in the "Build" state, after scheduling, for tens of minutes on average. There seemed to be some probabilistic effect here, which I hypothesize is related to the locking mechanism not using a "fair" lock (first-come, first-served) by default. Our fix was to drastically increase the interval this task runs at--from every 1 minute to every 12 hours. We only provision bare metal, so my rationale was that the periodic full resource sync was less important and mostly helpful for fixing weird things where somehow Placement's state got out of sync with Nova's somehow. I'm wondering, after all this, if it makes sense to rethink this one-semaphore thing, and instead create a per-hypervisor semaphore when doing the resource syncing. I can't think of a reason why the entire set of hypervisors needs to be considered as a whole when doing this task, but I could very well be missing something. TL;DR: if you have one nova-compute process managing lots of Ironic hypervisors, consider tweaking the update_resources_interval to a higher value, especially if you're seeing instances stuck in the Build state for a while. Cheers, Jason Anderson Cloud Computing Software Developer Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

5 11

[qa][ptg][nova][cinder][keystone][neutron][glance][swift][placement] How to make integrated-gate testing (tempest-full) more stable and fast
by Ghanshyam Mann 07 Jul '19

07 Jul '19

Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We tried to improve it via filtering the slow tests in the separate tempest-slow job but the situation has not been improved much. We talked about the Ideas to make it more stable and fast for projects especially when failure is not related to each project. We are planning to split the integrated-gate template (only tempest-full job as first step) per related services. Idea: - Run only dependent service tests on project gate. - Tempest gate will keep running all the services tests as the integrated gate at a centeralized place without any change in the current job. - Each project can run the below mentioned template. - All below template will be defined and maintained by QA team. I would like to know each 6 services which run integrated-gate jobs 1."Integrated-gate-networking" (job to run on neutron gate) Tests to run in this template: neutron APIs , nova APIs, keystone APIs ? All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for neutron gate: exlcude the cinder API tests, glance API tests, swift API tests, 2."Integrated-gate-storage" (job to run on cinder gate, glance gate) Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs, Nova APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for cinder, glance gate: excluded the neutron APIs tests, Keystone APIs tests 3. "Integrated-gate-object-storage" (job to run on swift gate) Tests to run in this template: Cinder APIs , Glance APIs, Swift APIs and All scenario currently running in tempest-full in the same way ( means non-slow and in serial) Improvement for swift gate: excluded the neutron APIs tests, - Keystone APIs tests, - Nova APIs tests. Note: swift does not run integrated-gate as of now. 4. "Integrated-gate-compute" (job to run on Nova gate) tests to run is : Nova APIs, Cinder APIs , Glance APIs ?, neutron APIs and All scenario currently running in tempest-full in same way ( means non-slow and in serial) Improvement for Nova gate: excluded the swift APIs tests(not running in current job but in future, it might), Keystone API tests. 5. "Integrated-gate-identity" (job to run on keystone gate) Tests to run is : all as all project use keystone, we might need to run all tests as it is running in integrated-gate. But does keystone is being unsed differently by all services? if no then, is it enough to run only single service tests say Nova or neutron ? 6. "Integrated-gate-placement" (job to run on placement gate) Tests to run in this template: Nova APIs tests, Neutron APIs tests + scenario tests + any new service depends on placement APIs Improvement for placement gate: excluded the glance APIs tests, cinder APIs tests, swift APIs tests, keystone APIs tests Thoughts on this approach? The important point is we must not lose the coverage of integrated testing per project. So I would like to get each project view if we are missing any dependency (proposed tests removal) in above proposed templates. - https://etherpad.openstack.org/p/qa-train-ptg -gmann

7 11

[tc][ops] reviving osops- repos
by Mohammed Naser 02 Jul '19

02 Jul '19

Hi everyone, So we've realized that we have a few operational tools that we keep around that live all over the place and it would probably be the best for the community to have access to them. I did a bit of research and it looks like the openstack/osops-* repos used to exist between operators to maintain things like this, however, they've gone pretty abandoned over time. I'd like to pick up this effort and as part of doing that, I'd like to move them into a GitHub organization and rename them out of x/ to make them not seem so .. 'dead'. I've requested to move x/osops-* to openstack-operators/* in GitHub so that I can setup the appropriate mirroring in post pipeline (and then propose a patch to rename things inside Gerrit as well). There was concern around the idea of using 'openstack' as it's a copyrighted term and a TC discussion was brought up as an idea which makes sense. I'd like to ask if everyone else feels like it's okay so move things into that org and host them there. Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

9 15

[dev][requirements] Upcoming changes to constraints handling in tox.ini
by Tony Breeds 27 Jun '19

27 Jun '19

Hi folks, This is a heads-up to describe 3 sets of changes you'll start seeing starting next week. 1) lower-constraints.txt handling TL;DR: Make sure projects do not specify a constraint file in install_command 2) Switch to the new canonical constraints URL on master TR;DR: Make sure you use https://releases.openstack.org/constraints/upper/master 3) Switch to the new canonical constraints URL on stable branches TR;DR: Make sure you use https://releases.openstack.org/constraints/upper/$series These will be generated from a member of the requirements team[1], and will be on the gerrit topic constraints-updates. We'll start next week to give y'all a few days to digest this email Now for slightly more details: 1) lower-constraints.txt handling If a projects has an install_command that includes a constraint file *and* the also have a lower-constraints test env what may happen is something like: given a tox.ini that contains: --- [testenv] install_command = pip install -U -c upper-constraints.txt {opts} {deps} deps = -r requirements.txt -r test-requirements.txt [testenv:lower-constraints] deps = -c lower-constrtaints.txt -r requirements.txt -r test-requirements.txt --- pyXX: virtualenv -p $python_version .tox/$testenv_name .tox/$testenv_name/bin/pip install -U -c upper-constratints.txt -r requirements.txt -rtest-requirements.txt .tox/$testenv_name/bin/pip install . lower-constraints: virtualenv -p $python_version .tox/$testenv_name .tox/$testenv_name/bin/pip install -U -c upper-constratints.txt -c lower-constrtaints.txt -r requirements.txt -rtest-requirements.txt .tox/$testenv_name/bin/pip install . pip will constrain a library to the first item it encounters, which pretty much means that the lower-constraints testenv is still testing against upper-constraints.txt[3] The fix is to move all constraints into deps rather than install_command. This impacts ~40 projects and will be fixed by hand, rather than a tool so we can preserve each projects style for tox.ini. There's a pretty good chance that this change will break the lower-constraints testenv, project teams are encouraged to help identify the incorrect values in lower-constraints, the requirements team will pick the current value in upper-constraints which almost certainly raises a value well past the minimum. https://review.opendev.org/#/c/601188/ is the change we did for keystone way back when Some projects have upper-constraints.txt in install_command but explicitly override it in lower-constratints.txt this works and wont be changed at this point but ideally all projects will use the same format to reduce the impact in switching between projects. 2) Switch to the new canonical constraints URL on master Way back at the first Denver PTG thr requirements, release and infrastructure teams discussed avoiding using gitweb urls for constraints urls. This was discussed[4] in more detail after the liberty EOL broke consumers still wanting to use tox as: https://git.openstack.org/cgit/openstack/requirements/plain/upper-constrain… 'vanished' The idea was re-raised[5] in Feb. this year where we devised a plan to create URLs that abstract the git hosting and allow us to switch between branches and tags at the right times in the OpenStack life-cycle. Other, more dynamic, ideas were discussed but over complicate things and make OpenStack slightly harder to understand or create more work. for example we could: a. Add an install_script that looks up $metadata and grab the right constraints We're trying avoid scripts like this as it very rapidly gets us to the point where we're doing "managed copy-and-paste" between projects *or* projects install scripts diverge b. Add to tox itself the ability to derive the URL from $metadata This is doable but frankly requires more time than we have right now. So what we have is: redirect 301 /constraints/upper/master http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/train http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/stein http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/rocky http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/queens http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/pike http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/ocata http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/newton http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/mitaka http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/liberty http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/kilo http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… redirect 301 /constraints/upper/juno http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraint… [Yes I know we need to use opendev that's tomorrows problem] so that means that: http{s,}://releases.openstack.org/constraints/upper/master http{s,}://releases.openstack.org/constraints/upper/$series will always do the right thing based on the state of each OpenStack release. So we're going to generate changes that point to the right place. For master we have 2 options: a. Use .../train If we use train in the constraints URLs now things will work and when we branch for 'U' we'd need to update master to point to the right series name. This is very doable. The release team already generates a change on master at this point in time to update reno. We also need to update the new stable branch so that .gitreview points to the new branch. b. Use .../master If we use 'master' in the constratints URLs master will always work and when we branch for 'U' we'd need to update the stable/train branch to point to train. The release team already does this today along with the .gitreview change. So the bottom line is that using master (over train) is *slighty* less work for the release team and *slightly* less work for each project team. as such that's what we're going to do. There are several changes in the works that set the constraints url to either https://opendev.org/openstack/requirements/raw/branch/master/upper-constrai… or https://releases.openstack.org/constraints/upper/train These will work and will not break anything when the project branches, however for the sake of OpenStack wide consistency, and avoiding merge conflicts project teams are discouraged from merging them, or if the change also fixes other references to git.openstack.org just have the tox.ini updates removed. Once tox.ini is done the requirements team will look for additional places where the 'old style' urls are used and follow-up but right now there's a lot of noise to sift through 3) Switch to the new canonical constraints URL on stable branches Hopefully with all the background from 2 above this should be pretty self-explanatory. This is however much lower priority and probably wont start until the bulk of 1 and 2 is complete Yours Tony. [1] https://review.opendev.org/#/admin/groups/131,members [2] [2] Probably me actually ;P [3] There is some nuance here but this is the general rule [4] In the thread that starts with: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122333.ht… [5] In the thread that starts with: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002682…

6 7

[stackalytics] Reported numbers seem inaccurate
by Mark Goddard 26 Jun '19

26 Jun '19

Hi, I've heard a couple of people say they don't feel the numbers reported by Stackalytics accurately reflect reality. I've been trying to gather a few stats for my Kolla update session in Denver, and am finding the same. I'll try to give some concrete examples. Reviews for all kolla deliverables in Stein [1]. Here the company stats don't reflect the individual stats. Also, the total reviews in the 'Kolla Official' module does not equal the sum of the reviews of its submodules (kolla, kolla-ansible, kolla-cli). If I look at the contribution summary for Kolla Official in the last 90 days [2], they are actually greater than those for the last 180 days [3]! There are also similar issues with commit metrics, and none seem to match what I see in git. Thanks, Mark [1] https://www.stackalytics.com/?metric=marks&module=kolla-group [2] https://www.stackalytics.com/report/contribution/kolla-group/90 [3] https://www.stackalytics.com/report/contribution/kolla-group/180

10 11

[cinder] ceph multiattach details?
by Bernd Bausch 24 Jun '19

24 Jun '19

The Stein release notes mention that the RBD driver now supports multiattach, but i have not found any details. Are there limitations? Is there a need to configure anything? In the RBD driver <https://opendev.org/openstack/cinder/src/branch/master/cinder/volume/driver…>, I find this: def _enable_multiattach(self, volume): multipath_feature_exclusions = [ self.rbd.RBD_FEATURE_JOURNALING, self.rbd.RBD_FEATURE_FAST_DIFF, self.rbd.RBD_FEATURE_OBJECT_MAP, self.rbd.RBD_FEATURE_EXCLUSIVE_LOCK, ] This seems to mean that journaling and other features (to me, it's not quite clear what they are) will be automatically disabled when switching on multiattachment. Further down in the code I see that replication and multiattach are mutually exclusive. Is there some documentation about the Ceph multiattach feature, even an email thread? Thanks, Bernd ||

6 8

Re: [nova][placement] Need for a nova-manage placement audit command
by Matt Riedemann 19 Jun '19

19 Jun '19

Re-posting this since it came up again in IRC today. On 9/20/2018 6:09 PM, Matt Riedemann wrote: > mnaser wrote a simple placement audit tool today and dumped his script > into a bug report [1]. It turns out several operators have already > written a variant of this same tool. The idea is to compare what's in > placement for allocations and (compute node) resource providers against > what's in nova for instances and compute nodes, look for orphans in > placement and then report them, or maybe also clean them up. > > I think this could go into a "nova-manage placement audit" command and > should be pretty easy to write for minimum functionality (start with > just the report). > > I'm advertising the need here in case someone wants to work on this. I'd > like to myself, but just can't justify the time right now. > > [1] https://bugs.launchpad.net/nova/+bug/1793569 > -- Thanks, Matt

2 1

[nova][dev][ops] server status when compute host is down
by melanie witt 18 Jun '19

18 Jun '19

Hey all, I'm looking for feedback around whether we can improve how we show server status in server list and server show when the compute host it resides on is down. When a compute host goes down while a server on it was previously running, the server status continues to show as ACTIVE in a server list. This is because the power state and status is adjusted by a periodic task run by nova-compute, so if nova-compute is down, it cannot update those states. So, for an end user, when they do a server list, they see their server as ACTIVE when it's actually powered off. We have another field called 'host_status' available since API microversion 2.16 [1] which is controlled by policy and defaults to admin, which is capable of showing the server status as UNKNOWN if the field is specified, for example: nova list --fields id,name,status,task_state,power_state,networks,host_status This is cool, but it is only available to admin by default, and it requires that the end user adds the field to their CLI command in the --fields option. Question: do people think we should make the server status field reflect UNKNOWN as well, if the 'host_status' is UNKNOWN? And if so, should it be controlled by policy or no? Normally, we do not expose compute host details to non-admin in the API by default, but I noticed recently that our "down cells" support will show server status as UNKNOWN if a server is in a down cell [2]. So I wondered if it would be considered OK to show UNKNOWN if a host is down we well, without defaulting it to admin-only. I would really appreciate if people could share their opinion here and if consensus is in support, I will move forward with proposing a change accordingly. Cheers, -melanie [1] https://docs.openstack.org/nova/latest/reference/api-microversion-history.h… [2] https://github.com/openstack/nova/blob/66a77f2fb75bbb9daebdca1cad0255ecafe4…

8 14

[oslo] Courtesy ping changes
by Ben Nemec 18 Jun '19

18 Jun '19

Important: Action is required if you want to continue receiving courtesy pings. Read on for details. This is an oslo-specific followup to [0]. There's a lot of good discussion there if you're interested in the background for this email. The TLDR version is that we're going to keep the Oslo courtesy ping list because a number of Oslo contributors have expressed their preference for it. However, we are making some changes. First, the ping list will be cleared at the start of each cycle. This should prevent us from pinging people who no longer work on Oslo (which is almost certainly happening right now). Everyone who wants a courtesy ping will need to re-opt-in each cycle. We'll work out a transition process so people don't just stop receiving pings. Second, the ping list is going to move from the script in oslo.tools to the meeting agenda[1] on the wiki. There's no need for Oslo core signoff on ping list changes and that makes it a waste of time for both the cores and the people looking to make changes to the list. This does mean we'll lose the automatic wrapping of the list, but once we clean up the stale entries I suspect we won't need to wrap it as much anyway. I will continue to use the existing ping list for the next two weeks to give everyone a chance to add their name to the new list. I've seeded the new list with a couple of people who had expressed interest in continuing to receive pings, but if anyone else wants to continue getting them please add yourself to the list in [1] (see the Agenda Template section). I'm intentionally _not_ adding all of the active Oslo cores on the assumption that you may prefer to set up your own notification method. I might automatically carry over cores from cycle to cycle since they are presumably still interested in Oslo and chose that notification method, but we'll worry about that at the start of next cycle. I think that covers the current plan for courtesy pings in Oslo. If you have any comments or concerns please let me know. Otherwise expect this new system to take effect in 3 weeks. Thanks. -Ben 0: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006235.html 1: https://wiki.openstack.org/wiki/Meetings/Oslo

1 1

[all][qinling] Please check your README files
by Marcin Juszkiewicz 17 Jun '19

17 Jun '19

Train cycle is supposed to be "we really go for Python 3" cycle. Contrary to previous "we do not have to care about Python 3" ones. I am working on switching Kolla images into Python 3 by default [1]. It is a job I would not recommend even to my potential enemies. Misc projects fail into random ways (sent a bunch of patches). 1. https://review.opendev.org/#/c/642375 Quite common issue is related to README files. I know that we have XXI century and UTF-8 is encoding for most of people here. But Python tools are not so advanced so far and loves to explode when see characters outside of US-ASCII encoding. Please check your README files. And move them to US-ASCII if needed. I already fixed few projects [2][3][4][5] but would love to see developers fixing their code too. 2. https://review.opendev.org/644531 3. https://review.opendev.org/644533 4. https://review.opendev.org/644535 5. https://review.opendev.org/644536

5 8