July 2019 - openstack-discuss - lists.openstack.org

[osc] Very slow openstack commands in Stein Devstack running on VM
by Bernd Bausch 29 Jul '19

29 Jul '19

I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication. Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed): Traceback (most recent call last): File "/usr/local/bin/openstack", line 7, in <module> from openstackclient.shell import main File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in <module> from osc_lib.api import auth File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in <module> from keystoneauth1.loading import base File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in <module> from keystoneauth1.loading import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in <module> from keystoneauth1 import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in <module> from keystoneauth1 import session File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in <module> import requests File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in <module> from urllib3.contrib import pyopenssl File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in <module> from cryptography.hazmat.backends.openssl import backend as openssl_backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in <module> backend = Backend() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__ self.activate_osrandom_engine() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine with self._get_osurandom_engine() as e: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine res = self._lib.ENGINE_init(e) KeyboardInterrupt My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI? Thus my questions: *Is the delay really caused by my running the cloud on a VM? What can I do to improve this?* Thanks, Bernd

4 4

[goals][IPv6-Only Deployments and Testing] Week R-12 Update
by Ghanshyam Mann 29 Jul '19

29 Jul '19

Hello Everyone, I am starting this status email for 'IPv6-Only Deployments and Testing ' community goal for Train. Storyboard: ========= - https://storyboard.openstack.org/#!/story/2005477 Current status: ============ 1. Base job 'devstack-tempest-ipv6' has been prepared on Tempest (patch yet to merge)[1] 2. 'tempest-ipv6-only' is defined in Tempest (patch yet to merge)[1] 3. 'tempest-ipv6-only' has been proposed to run on 5 services project date side [2]. 4. I have proposed few more projects jobs like congress etc and will keep doing for other projects also. 5. Neutron stadium projects jobs. I talked to slaweq on IRC [3]on what all neutron stadium falls under the IPv6 address communication criteria so that we can add ipv6-only job for those project only. He is going to prepare the list of those project and accordingly we will add jobs Everything related to this goal can be found under this topic: Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(stat… How to define and run new IPv6 Job on project side: ======================================= - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing - I am adding this wiki info in goal doc also[4]. Review suggestion: ============== - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with that point of view. If anything missing, comment on patch. - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var setting. But if your project needs more specific varification then it can be added in project side job as post-run playbooks as described in wiki page[5]. [1] https://review.opendev.org/#/c/671231/ [2] https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(stat… [3] http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-ne… [4] https://review.opendev.org/#/c/671898/ [5] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing -gmann

2 2

[tripleo] Roadmap to simplify TripleO
by Emilien Macchi 29 Jul '19

29 Jul '19

Even though TripleO is well known for its maturity, it also has a reputation of being complex when it comes to the number of tools that it uses. Somewhat related to the efforts that James is leading with "Scaling TripleO" [1] [2], I would like to formalize our joint efforts to make TripleO simpler in the future. Some work has been done over the last releases already and yet we have seen net benefits; however we still have challenges ahead of us. - With no UI anymore, do we really need an API? - How can we reduce the number of languages in TripleO? ... and make Python + Ansible the main ones. - How can we reduce our dependencies? I created a document which explains the problems and propose some solutions: https://docs.google.com/document/d/1vY9rsccgp7NHFXpLtCFTHQm14F15Tij7lhn5X_P… For those who can't or don't want Google Doc, I've put together the notes into etherpad [3] and I'll take care of making sure it's updated at last at the beginning until we sort things out. Feel free to be involved: - comment or suggest edits if you have feedback / ideas - sign-up if you're interested to contribute For now I expect this document to be a clear place what our plan is but I wouldn't be surprised if in the future we break it down into specs / blueprints / etc for better tracking. Thanks, [1] https://docs.google.com/document/d/12tPc4NC5fo8ytGuFZ4DSZXXyzes1x3U7oYz9nea… [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007638.html [3] https://etherpad.openstack.org/p/tripleo-simplification -- Emilien Macchi

3 2

[requirements] docutils-0.15.1 pypi missing wheel
by Matthew Thode 28 Jul '19

28 Jul '19

Hi, In trying to update openstack past the broken 0.15 release to the 0.15.1 bugfix release I ran into a problem when trying to install it in python3 (we are currently testing on ppython 3.6). It seems like the missing wheel is the issue. When do you plan on publishing it? The following demonstrates the issue. The logs are not kept forever but it's a standard error. ERROR: No matching distribution found for docutils===0.15.1 http://logs.openstack.org/73/672873/3/check/requirements-tox-py36-check-uc/… It seems like other devs/users have reported the problem as well but there hasn't been any activity to address it. See the following messages: https://sourceforge.net/p/docutils/mailman/message/36726543/ https://sourceforge.net/p/docutils/mailman/message/36726888/ Please let me know if you need anything else from me. -- Matthew Thode

2 2

[keystone] Keystone Team Update - Week of 22 July 2019
by Colleen Murphy 27 Jul '19

27 Jul '19

# Keystone Team Update - Week of 22 July 2019 ## News ### Midcycle held this week We held our first midcycle since the PTGs became a thing, possibly also the team's first virtual midcycle (I'm not sure). It went quite well and it was great to sync up with everyone. Recap posted on the mailing list[1]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007987.html ## Action Items As discussed during the midcycle retrospective, we'll start calling out action items from the last meeting (or last midcycle, or last PTG, or perhaps any other commitment worth mentioning in a weekly update). * Remove the keystone socket timeout option (IN PROGRESS: https://bugs.launchpad.net/keystone/+bug/1837407) * Seek feedback from operators on caching guide (guide draft IN PROGRESS: https://review.opendev.org/672120) ## Office Hours When there are topics to cover, the keystone team holds office hours on Tuesdays at 17:00 UTC. The topic for next week's office hour will be: bug triage! The location for next week's office hour will be: https://meet.jit.si/keystone-office-hours (fallback to #openstack-keystone if needed) Add topics you would like to see covered during office hours to the etherpad: https://etherpad.openstack.org/p/keystone-office-hours-topics ## Open Specs Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 16 changes this week, which included progress made on system scope/default roles[2] and the last of the changes needed for the python3-train goal[3][4]. [2] https://review.opendev.org/670926 [3] https://review.opendev.org/667755 [4] https://review.opendev.org/667756 ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 35 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ### Priority Reviews * Train Roadmap Stories System scope/default roles (https://trello.com/c/ERo50T7r , https://trello.com/c/RlYyb4DU) - https://review.opendev.org/645968 (Implement domain reader support for grants) - https://review.opendev.org/667730 (Implement domain admin support for grants) Application credential access rules (https://trello.com/c/XyBGhKrE) - https://review.opendev.org/668238 (Expose access rules as its own API) - https://review.opendev.org/663440 (Add user_id, external_id to access rules table) Caching Guide (https://trello.com/c/UCFt3mfF) - https://review.opendev.org/672120 (Update the caching guide) Predictable IDs (https://trello.com/c/MVuu6DbU) - https://review.opendev.org/651655 (Predictable IDs for Roles) Federation Improvements (https://trello.com/c/GEaIpbpm) - https://review.opendev.org/637305 (Add new attribute to the federation protocol API) Oslo.limit (https://trello.com/c/KGGkNijR) - https://review.opendev.org/667242 (Add usage example) - https://review.opendev.org/666444 (Flush out basic enforcer and model relationship) - https://review.opendev.org/666085 (Add ksa connection logic) YAML Catalog (https://trello.com/c/Qv14G0xp) - https://review.opendev.org/483514 (Add yaml-loaded filesystem catalog backend) * Needs Discussion - https://review.opendev.org/669959 (discourage using X.509 with external auth) - https://review.opendev.org/655166 (Allows to use application credentials through group membership) * Oldest - https://review.opendev.org/448755 (Add federated support for creating a user) ## Bugs This week we opened 3 new bugs and closed 7 (yay!) Bugs opened (3) Bug #1837407 (keystone:Low) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1837407 Bug #1837513 (keystone:Undecided) opened by ahmed elbendary https://bugs.launchpad.net/keystone/+bug/1837513 Bug #1837741 (oslo.policy:High) opened by Ben Nemec https://bugs.launchpad.net/oslo.policy/+bug/1837741 Bugs closed (1) Bug #1837513 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1837513 Bugs fixed (6) Bug #1750615 (keystone:High) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1750615 Bug #1782922 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1782922 Bug #1818725 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1818725 Bug #1828565 (keystone:Low) fixed by Jose Castro Leon https://bugs.launchpad.net/keystone/+bug/1828565 Bug #1818845 (keystone:Wishlist) fixed by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1818845 Bug #1829802 (keystonemiddleware:Undecided) fixed by norman shen https://bugs.launchpad.net/keystonemiddleware/+bug/1829802 ## Milestone Outlook https://releases.openstack.org/train/schedule.html This week marks the milestone 2 spec freeze. Feature proposal freeze is in three (3!) weeks. This means code implementing approved specs needs to be in a reviewable, non-WIP state. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter

1 0

[nova] Train Cycle Themes - An Interactive Experience
by Eric Fried 26 Jul '19

26 Jul '19

Hi all. Now that we're past spec freeze, Matt suggested I write a report on how we're doing against our cycle themes. That sounded really tedious and boring and project-managementy, so I thought I would spread the joy. Bear with me, this is going to be a little different. I've created a null review in the nova-specs repository [1]. I'd like us to use this as a mechanism to collaboratively gather thoughts via review comments [2]. I've made a start on some bits. Have fun! efried [1] https://review.opendev.org/#/c/673095/ [2] I could have used an etherpad. But this will be more... permanent.

1 0

Forcing restart of a worker node with running guest
by Mauricio Tavares 26 Jul '19

26 Jul '19

So one of my worker nodes is, to put it mildly, rather unhappy: Message from syslogd@openstack-w1 at Jul 25 15:57:49 ... kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [vnc_worker:9141] Message from syslogd@openstack-w1 at Jul 25 15:57:57 ... kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:60] Message from syslogd@openstack-w1 at Jul 25 15:58:05 ... kernel:NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [migration/8:49] I found out when it was taking 30 min to delete a guest. So, what I can do in a forceful way? 1. How to kill the guest? Can I kill it through virsh or openstack compute service will get sad? 2. What would happen if I stop the compute service? 3. What would happen if I get really annoyed and tell worker node to reboot?

2 3

[goal][python3] Train unit tests weekly update (goal-7)
by Corey Bryant 26 Jul '19

26 Jul '19

This is the goal-7 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 7 weeks remaining for completion of Train community goals [2]. == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Ongoing Work == All patches have been submitted for all applicable projects. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3-train+is:open Failing patches: https://review.openstack.org/#/q/topic:python3-train+status:open+(+label:Ve… Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Completed Work == Merged patches: https://review.openstack.org/#/q/topic:python3-train+is:merged == How can you help? == Please take a look at the failing patches and help fix any failing unit tests for your projects. Python 3.7 unit tests will be self-testing in Zuul. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals /train/python3-updates.html [2] Train release schedule: https://releases.openstack.org/train/schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/2018… Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtim… Thanks, Corey

1 0

[nova][sfe] Train Spec Freeze Exception Process
by Eric Fried 26 Jul '19

26 Jul '19

Hello Nova. Spec freeze happened, which means we won't be approving any more specs for Train. Unless we do. >From this point on, you will be required to run the following gauntlet if you want your spec to land in Train. - Put your name and a link to your spec on the agenda for the next Nova meeting [1][2]. - Send a note to openstack-discuss tagged with [nova][sfe] (like this one is) including the spec for which you are requesting an exception, and noting the date of the meeting. - Attend that meeting and state your case. Assuming a quorum of cores is present, we will attempt to render a decision, or at least indicate what further actions are necessary. Thanks, efried [1] https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting [2] If you can't attend a meeting, or "next Thursday" is just too long to wait, say so in your email and we'll endeavor to discuss it entirely on the mailing list.

1 0

[nova][oslo.log] global_request_id help
by Eric Fried 26 Jul '19

26 Jul '19

While doing something unrelated, I noticed this code [0] that shouldn't need to exist if global_request_id is working. I threw out some debug patches [1] to dump headers. Results: [2] (nova), [3] (placement). We're tracking global_request_id ``req-a51f...``. Nova is sending the global_request_id properly in the X-Openstack-Request-Id header, and Placement is receiving it correctly, because it's showing up in the expected spot in the Placement log line (after the opening bracket, per the logging_context_format_string in placement.conf [4]). Placement is sending back the request_id (``req-b3ce...``) in the x-openstack-request-id header in the response. (I thought this was a bug, but it turns out this is correct per the design [5].) So the problem is that the Nova log is printing ``None`` for the global_request_id. Based on logging_context_format_string in nova-cpu.conf [6], I expect it in the first field after the opening bracket, just like for Placement. If the global_request_id were showing up in the Nova log, we wouldn't need to correlate the Nova and Placement sides of this conversation via the Placement request_id, and we could get rid of [0] and its usages. I don't understand how LOG.xxx() is supposed to glean this value. Can someone help me find the disconnect? Thanks, efried [0] https://opendev.org/openstack/nova/src/commit/d358df32f7a2a1125962e27f24253… [1] https://review.opendev.org/#/q/topic:where-are-you-global_request_id [2] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log… [3] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log… [4] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log… [5] https://specs.openstack.org/openstack/oslo-specs/specs/pike/global-req-id.h… [6] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log…

1 0