July 2019 - openstack-discuss - lists.openstack.org

[ops][glance][nova] scheduling problem because of ImagePropertiesFilter
by Massimo Sgaravatto 29 Jul '19

29 Jul '19

We have just finished the update of our cloud to Rocky but we are seeing a strange issue with images with property: hypervisor_type='QEMU' The scheduling of instances created with such images fails because of the ImagePropertiesFilter [*] Removing that property from the image, the scheduling works. We also tried changing hypervisor_type='QEMU' --> hypervisor_type='qemu', but this didn't help. Any suggestions? Thanks, Massimo [*] 2019-07-17 13:52:58.148 13312 INFO nova.filters [req-1863bef0-0326-46d1-a836-436227e91eef 6e3b136d578f4292a5c03b16f443ab3d d27fe2becea94a3e980fb9f66e2f291a - default default] Filtering removed all hosts f\ or the request with instance ID '63810b60-76e4-4e76-a1c3-e4d3932c002e'. Filter results: ['AggregateMultiTenancyIsolation: (start: 49, end: 37)', 'AggregateInstanceExtraSpecsFilter: (start: 37, end: 34)', \ 'RetryFilter: (start: 34, end: 34)', 'AvailabilityZoneFilter: (start: 34, end: 34)', 'ComputeFilter: (start: 34, end: 32)', 'ComputeCapabilitiesFilter: (start: 32, end: 32)', 'ImagePropertiesFilter: (star\ t: 32, end: 0)']

5 21

[tc][all] seeking candidates to run in the next TC election
by Doug Hellmann 29 Jul '19

29 Jul '19

Nominations for the next Technical Committee term open on 27 August, just over 4 weeks from now. In the past, we have had people wait until the last minute to consider running, and then in some cases they had issues getting approval from their managers to serve in the role. This time, since I know there will be at least one open seat when I step down, I thought it would be a good idea to start that recruiting process a little earlier. Our community places a high value on democratically electing representative leadership — selecting members of the community to help steer the project. That system only works if we have people willing and able to serve in the elected roles like the Technical Committee. There is no fall-back option. We need community members to step up and run. So, if you are interested in serving on the TC, now is the time ask the necessary questions within your company. If you need advice about how to approach your manager, any current TC member, including me, would be happy to help you. Doug

1 0

[requirements] changes in requests library maintainership
by Doug Hellmann 29 Jul '19

29 Jul '19

It looks like the Kenneth Reitz, the primary maintainer of the requests library, is stepping down [1]. There are several offers to take over right now, so it's not really clear to me what's going to happen. I think we'll want to keep an eye on both requests and requests3 until the new maintainer is designated. And of course if anyone is interested in helping out, I encourage you to get involved in the discussion on github. [1] https://github.com/not-kennethreitz/team/issues/21 -- Doug

2 1

[nova][sfe] Nova part of Image Encryption
by Josephine Seifert 29 Jul '19

29 Jul '19

Hi, we would like to have a spec freeze exception for the spec for the Nova Part of Image Encryption [1] as this is part of a cross-project contribution with specs also in cinder (already merged) and glance. We will attend the next meeting on August 1st. Greetings Josephine (Luzi) [1] https://review.opendev.org/#/c/608696

1 0

[searchlight] Meeting today cancelled
by Trinh Nguyen 29 Jul '19

29 Jul '19

Hi team, I cannot hold the meeting today so please email or ping me on #openstack-searchlight channel. Bests, -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

1 0

[ironic][edge] L3/DHCP-Less deployments
by Julia Kreger 29 Jul '19

29 Jul '19

Greetings everyone! Over the last few weeks, I've had numerous discussions with contributors who have expressed interest in the DHCP-less deployment method specification document[0]. It seems many parties are interested! I think the path forward at this time is to get everyone talking about where and how they can help push this feature forward. If those interested could propose time windows when they are available, I'll try to find a mutual time window where we can try to get everyone on IRC or on a call and try to map out the next steps together. Sound good? -Julia [0]: https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3…

2 2

[tc] weekly update
by Mohammed Naser 29 Jul '19

29 Jul '19

Hi everyone, This is the weekly update for what happened inside the Openstack TC last week. You can check for more information in the openstack/governance repository. We had one change only which was the retirement of puppet-crane (from TripleO). Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

1 0

[Watcher] No IRC meeting this week
by li.canwei2＠zte.com.cn 29 Jul '19

29 Jul '19

Hi team, I'll be on vacation until to August 5, so I can't host meeting this week. Thanks, licanwei

1 0

[neutron] Bug Deputy July 22 - July 29
by Boden Russell 29 Jul '19

29 Jul '19

Below is a list of neutron bugs that came in last week. To the best of my knowledge, the only bug still under discussion is #2 below (and the RFE in #3). Thanks to everyone on the team for their bug diligence. It looks like Brian H. is on deck for bug duty this week. 1) https://bugs.launchpad.net/neutron/+bug/1838068 "QoSTest:test_qos_basic_and_update" failing in DVR node scenario 2) https://bugs.launchpad.net/neutron/+bug/1837916 Trunk subports sometimes not becoming ACTIVE and trunk:subport 3) https://bugs.launchpad.net/neutron/+bug/1837847 [RFE] neutron-vpnaas OpenVPN driver 4) https://bugs.launchpad.net/neutron/+bug/1837635 HA router state change from "standby" to "master" should be delayed 5) https://bugs.launchpad.net/neutron/+bug/1837553 Neutron api-ref should mention that list of e.g. IDs is supported in GET requests 6) https://bugs.launchpad.net/neutron/+bug/1837552 neutron-tempest-with-uwsgi job finish with timeout very often 7) https://bugs.launchpad.net/neutron/+bug/1837529 Cannot use push-notification with custom objects

1 0

[nova][sfe] Use PCPU and VCPU in One Instance
by David Wang 29 Jul '19

29 Jul '19

Hi, We are proposing a Nova spec for the feature of creating an instance of mixed dedicated CPUs and shared CPUs and experience communities review. The spec link is https://review.opendev.org/#/c/668656/ and spec name is "Use PCPU and VCPU in One Instance". I recognized that Nova Sepc for Train release is frozen, I'd like to request a spec freeze exception process and let the spec to be merged in train release. I may have trouble in attending next week's team meeting, on Aug 1, so, please discuss in the email thread. Thanks Huaqiang

1 0

[osc] Very slow openstack commands in Stein Devstack running on VM
by Bernd Bausch 29 Jul '19

29 Jul '19

I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication. Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed): Traceback (most recent call last): File "/usr/local/bin/openstack", line 7, in <module> from openstackclient.shell import main File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in <module> from osc_lib.api import auth File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in <module> from keystoneauth1.loading import base File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in <module> from keystoneauth1.loading import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in <module> from keystoneauth1 import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in <module> from keystoneauth1 import session File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in <module> import requests File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in <module> from urllib3.contrib import pyopenssl File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in <module> from cryptography.hazmat.backends.openssl import backend as openssl_backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in <module> backend = Backend() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__ self.activate_osrandom_engine() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine with self._get_osurandom_engine() as e: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine res = self._lib.ENGINE_init(e) KeyboardInterrupt My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI? Thus my questions: *Is the delay really caused by my running the cloud on a VM? What can I do to improve this?* Thanks, Bernd

4 4

[goals][IPv6-Only Deployments and Testing] Week R-12 Update
by Ghanshyam Mann 29 Jul '19

29 Jul '19

Hello Everyone, I am starting this status email for 'IPv6-Only Deployments and Testing ' community goal for Train. Storyboard: ========= - https://storyboard.openstack.org/#!/story/2005477 Current status: ============ 1. Base job 'devstack-tempest-ipv6' has been prepared on Tempest (patch yet to merge)[1] 2. 'tempest-ipv6-only' is defined in Tempest (patch yet to merge)[1] 3. 'tempest-ipv6-only' has been proposed to run on 5 services project date side [2]. 4. I have proposed few more projects jobs like congress etc and will keep doing for other projects also. 5. Neutron stadium projects jobs. I talked to slaweq on IRC [3]on what all neutron stadium falls under the IPv6 address communication criteria so that we can add ipv6-only job for those project only. He is going to prepare the list of those project and accordingly we will add jobs Everything related to this goal can be found under this topic: Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(stat… How to define and run new IPv6 Job on project side: ======================================= - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing - I am adding this wiki info in goal doc also[4]. Review suggestion: ============== - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with that point of view. If anything missing, comment on patch. - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var setting. But if your project needs more specific varification then it can be added in project side job as post-run playbooks as described in wiki page[5]. [1] https://review.opendev.org/#/c/671231/ [2] https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(stat… [3] http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-ne… [4] https://review.opendev.org/#/c/671898/ [5] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing -gmann

2 2

[tripleo] Roadmap to simplify TripleO
by Emilien Macchi 29 Jul '19

29 Jul '19

Even though TripleO is well known for its maturity, it also has a reputation of being complex when it comes to the number of tools that it uses. Somewhat related to the efforts that James is leading with "Scaling TripleO" [1] [2], I would like to formalize our joint efforts to make TripleO simpler in the future. Some work has been done over the last releases already and yet we have seen net benefits; however we still have challenges ahead of us. - With no UI anymore, do we really need an API? - How can we reduce the number of languages in TripleO? ... and make Python + Ansible the main ones. - How can we reduce our dependencies? I created a document which explains the problems and propose some solutions: https://docs.google.com/document/d/1vY9rsccgp7NHFXpLtCFTHQm14F15Tij7lhn5X_P… For those who can't or don't want Google Doc, I've put together the notes into etherpad [3] and I'll take care of making sure it's updated at last at the beginning until we sort things out. Feel free to be involved: - comment or suggest edits if you have feedback / ideas - sign-up if you're interested to contribute For now I expect this document to be a clear place what our plan is but I wouldn't be surprised if in the future we break it down into specs / blueprints / etc for better tracking. Thanks, [1] https://docs.google.com/document/d/12tPc4NC5fo8ytGuFZ4DSZXXyzes1x3U7oYz9nea… [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007638.html [3] https://etherpad.openstack.org/p/tripleo-simplification -- Emilien Macchi

3 2

[requirements] docutils-0.15.1 pypi missing wheel
by Matthew Thode 28 Jul '19

28 Jul '19

Hi, In trying to update openstack past the broken 0.15 release to the 0.15.1 bugfix release I ran into a problem when trying to install it in python3 (we are currently testing on ppython 3.6). It seems like the missing wheel is the issue. When do you plan on publishing it? The following demonstrates the issue. The logs are not kept forever but it's a standard error. ERROR: No matching distribution found for docutils===0.15.1 http://logs.openstack.org/73/672873/3/check/requirements-tox-py36-check-uc/… It seems like other devs/users have reported the problem as well but there hasn't been any activity to address it. See the following messages: https://sourceforge.net/p/docutils/mailman/message/36726543/ https://sourceforge.net/p/docutils/mailman/message/36726888/ Please let me know if you need anything else from me. -- Matthew Thode

2 2

[keystone] Keystone Team Update - Week of 22 July 2019
by Colleen Murphy 27 Jul '19

27 Jul '19

# Keystone Team Update - Week of 22 July 2019 ## News ### Midcycle held this week We held our first midcycle since the PTGs became a thing, possibly also the team's first virtual midcycle (I'm not sure). It went quite well and it was great to sync up with everyone. Recap posted on the mailing list[1]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007987.html ## Action Items As discussed during the midcycle retrospective, we'll start calling out action items from the last meeting (or last midcycle, or last PTG, or perhaps any other commitment worth mentioning in a weekly update). * Remove the keystone socket timeout option (IN PROGRESS: https://bugs.launchpad.net/keystone/+bug/1837407) * Seek feedback from operators on caching guide (guide draft IN PROGRESS: https://review.opendev.org/672120) ## Office Hours When there are topics to cover, the keystone team holds office hours on Tuesdays at 17:00 UTC. The topic for next week's office hour will be: bug triage! The location for next week's office hour will be: https://meet.jit.si/keystone-office-hours (fallback to #openstack-keystone if needed) Add topics you would like to see covered during office hours to the etherpad: https://etherpad.openstack.org/p/keystone-office-hours-topics ## Open Specs Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 16 changes this week, which included progress made on system scope/default roles[2] and the last of the changes needed for the python3-train goal[3][4]. [2] https://review.opendev.org/670926 [3] https://review.opendev.org/667755 [4] https://review.opendev.org/667756 ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 35 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ### Priority Reviews * Train Roadmap Stories System scope/default roles (https://trello.com/c/ERo50T7r , https://trello.com/c/RlYyb4DU) - https://review.opendev.org/645968 (Implement domain reader support for grants) - https://review.opendev.org/667730 (Implement domain admin support for grants) Application credential access rules (https://trello.com/c/XyBGhKrE) - https://review.opendev.org/668238 (Expose access rules as its own API) - https://review.opendev.org/663440 (Add user_id, external_id to access rules table) Caching Guide (https://trello.com/c/UCFt3mfF) - https://review.opendev.org/672120 (Update the caching guide) Predictable IDs (https://trello.com/c/MVuu6DbU) - https://review.opendev.org/651655 (Predictable IDs for Roles) Federation Improvements (https://trello.com/c/GEaIpbpm) - https://review.opendev.org/637305 (Add new attribute to the federation protocol API) Oslo.limit (https://trello.com/c/KGGkNijR) - https://review.opendev.org/667242 (Add usage example) - https://review.opendev.org/666444 (Flush out basic enforcer and model relationship) - https://review.opendev.org/666085 (Add ksa connection logic) YAML Catalog (https://trello.com/c/Qv14G0xp) - https://review.opendev.org/483514 (Add yaml-loaded filesystem catalog backend) * Needs Discussion - https://review.opendev.org/669959 (discourage using X.509 with external auth) - https://review.opendev.org/655166 (Allows to use application credentials through group membership) * Oldest - https://review.opendev.org/448755 (Add federated support for creating a user) ## Bugs This week we opened 3 new bugs and closed 7 (yay!) Bugs opened (3) Bug #1837407 (keystone:Low) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1837407 Bug #1837513 (keystone:Undecided) opened by ahmed elbendary https://bugs.launchpad.net/keystone/+bug/1837513 Bug #1837741 (oslo.policy:High) opened by Ben Nemec https://bugs.launchpad.net/oslo.policy/+bug/1837741 Bugs closed (1) Bug #1837513 (keystone:Undecided) https://bugs.launchpad.net/keystone/+bug/1837513 Bugs fixed (6) Bug #1750615 (keystone:High) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1750615 Bug #1782922 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1782922 Bug #1818725 (keystone:Medium) fixed by Guang Yee https://bugs.launchpad.net/keystone/+bug/1818725 Bug #1828565 (keystone:Low) fixed by Jose Castro Leon https://bugs.launchpad.net/keystone/+bug/1828565 Bug #1818845 (keystone:Wishlist) fixed by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1818845 Bug #1829802 (keystonemiddleware:Undecided) fixed by norman shen https://bugs.launchpad.net/keystonemiddleware/+bug/1829802 ## Milestone Outlook https://releases.openstack.org/train/schedule.html This week marks the milestone 2 spec freeze. Feature proposal freeze is in three (3!) weeks. This means code implementing approved specs needs to be in a reviewable, non-WIP state. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter

1 0

[nova] Train Cycle Themes - An Interactive Experience
by Eric Fried 26 Jul '19

26 Jul '19

Hi all. Now that we're past spec freeze, Matt suggested I write a report on how we're doing against our cycle themes. That sounded really tedious and boring and project-managementy, so I thought I would spread the joy. Bear with me, this is going to be a little different. I've created a null review in the nova-specs repository [1]. I'd like us to use this as a mechanism to collaboratively gather thoughts via review comments [2]. I've made a start on some bits. Have fun! efried [1] https://review.opendev.org/#/c/673095/ [2] I could have used an etherpad. But this will be more... permanent.

1 0

Forcing restart of a worker node with running guest
by Mauricio Tavares 26 Jul '19

26 Jul '19

So one of my worker nodes is, to put it mildly, rather unhappy: Message from syslogd@openstack-w1 at Jul 25 15:57:49 ... kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [vnc_worker:9141] Message from syslogd@openstack-w1 at Jul 25 15:57:57 ... kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:60] Message from syslogd@openstack-w1 at Jul 25 15:58:05 ... kernel:NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [migration/8:49] I found out when it was taking 30 min to delete a guest. So, what I can do in a forceful way? 1. How to kill the guest? Can I kill it through virsh or openstack compute service will get sad? 2. What would happen if I stop the compute service? 3. What would happen if I get really annoyed and tell worker node to reboot?

2 3

[goal][python3] Train unit tests weekly update (goal-7)
by Corey Bryant 26 Jul '19

26 Jul '19

This is the goal-7 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 7 weeks remaining for completion of Train community goals [2]. == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Ongoing Work == All patches have been submitted for all applicable projects. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3-train+is:open Failing patches: https://review.openstack.org/#/q/topic:python3-train+status:open+(+label:Ve… Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Completed Work == Merged patches: https://review.openstack.org/#/q/topic:python3-train+is:merged == How can you help? == Please take a look at the failing patches and help fix any failing unit tests for your projects. Python 3.7 unit tests will be self-testing in Zuul. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals /train/python3-updates.html [2] Train release schedule: https://releases.openstack.org/train/schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/2018… Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtim… Thanks, Corey

1 0

[nova][sfe] Train Spec Freeze Exception Process
by Eric Fried 26 Jul '19

26 Jul '19

Hello Nova. Spec freeze happened, which means we won't be approving any more specs for Train. Unless we do. >From this point on, you will be required to run the following gauntlet if you want your spec to land in Train. - Put your name and a link to your spec on the agenda for the next Nova meeting [1][2]. - Send a note to openstack-discuss tagged with [nova][sfe] (like this one is) including the spec for which you are requesting an exception, and noting the date of the meeting. - Attend that meeting and state your case. Assuming a quorum of cores is present, we will attempt to render a decision, or at least indicate what further actions are necessary. Thanks, efried [1] https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting [2] If you can't attend a meeting, or "next Thursday" is just too long to wait, say so in your email and we'll endeavor to discuss it entirely on the mailing list.

1 0

[nova][oslo.log] global_request_id help
by Eric Fried 26 Jul '19

26 Jul '19

While doing something unrelated, I noticed this code [0] that shouldn't need to exist if global_request_id is working. I threw out some debug patches [1] to dump headers. Results: [2] (nova), [3] (placement). We're tracking global_request_id ``req-a51f...``. Nova is sending the global_request_id properly in the X-Openstack-Request-Id header, and Placement is receiving it correctly, because it's showing up in the expected spot in the Placement log line (after the opening bracket, per the logging_context_format_string in placement.conf [4]). Placement is sending back the request_id (``req-b3ce...``) in the x-openstack-request-id header in the response. (I thought this was a bug, but it turns out this is correct per the design [5].) So the problem is that the Nova log is printing ``None`` for the global_request_id. Based on logging_context_format_string in nova-cpu.conf [6], I expect it in the first field after the opening bracket, just like for Placement. If the global_request_id were showing up in the Nova log, we wouldn't need to correlate the Nova and Placement sides of this conversation via the Placement request_id, and we could get rid of [0] and its usages. I don't understand how LOG.xxx() is supposed to glean this value. Can someone help me find the disconnect? Thanks, efried [0] https://opendev.org/openstack/nova/src/commit/d358df32f7a2a1125962e27f24253… [1] https://review.opendev.org/#/q/topic:where-are-you-global_request_id [2] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log… [3] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log… [4] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log… [5] https://specs.openstack.org/openstack/oslo-specs/specs/pike/global-req-id.h… [6] http://logs.openstack.org/86/672986/1/check/nova-live-migration/e426c8a/log…

1 0

[nova] instence running, but status is Error
by Budai Laszlo 26 Jul '19

26 Jul '19

Dear all, I am testing different migration scenarios in our openstack, and I ended up with an instance that is running (I can access it by ssh), but its Status is Error. and its task_state is resize_migrating. I suppose that I got here due to the fact that the `allow_resize_to_same_host` is set to True, and the scheduler has tried to use the same compute node in the migration as described here: https://bugs.launchpad.net/nova/+bug/1811235. Is there any possibility to be able to manage again that instance using openstack (except to delete it)? I have tried the shut off (from horizon) and it gives me an error: fault: code: 400 created: '2019-07-26T14:46:05Z' message: Unavailable console type novnc. Thank you, Laszlo

2 3

[placement] update 19-29
by Chris Dent 26 Jul '19

26 Jul '19

HTML: https://anticdent.org/placement-update-19-29.html Welcome to a rushed pupdate 19-29. My morning was consumed by other things. A reminder: The Placement project holds office hours every Wednesday at 1500 UTC in the `#openstack-placement` IRC channel. If you have a topic that needs some synchronous discussion, then is an ideal time. Just start talking! # Most Important The two main things on the Placement radar are implementing Consumer Types and cleanups, performance analysis, and documentation related to nested resource providers. # What's Changed * The [api-ref](https://docs.openstack.org/api-ref/placement/) has moved to `docs.openstack.org` from `developer.openstack.org`. Redirects are in place. * Both traits and resource classes are [now cached](https://review.opendev.org/672298) per request, allowing for some name to id and id to name optimizations. * A new zuul template is [being used](https://review.opendev.org/671257) in placement that means fewer irrelevant tempest tests are run on placement changes. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 21 (-1) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 2 (0) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 10 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 17 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (0). # osc-placement osc-placement is currently behind by 12 microversions. * <https://review.opendev.org/666542> Add support for multiple member_of. There's been some useful discussion about how to achieve this, and a consensus has emerged on how to get the best results. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * <https://review.opendev.org/#/q/topic:bp/support-consumer-types> A WIP, as microversion 1.37, has started. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:op…. There are two [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:… being discussed. And zero [os-resource-classes changes](https://review.opendev.org/#/q/project:openstack/os-resource-class… (yay!). # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * <https://review.openstack.org/#/q/topic:bug/1819923> Nova: nova-manage: heal port allocations * <https://review.opendev.org/659233> Cyborg: Placement report * <https://review.opendev.org/662229> helm: add placement chart * <https://review.opendev.org/656023> Nova: Use OpenStack SDK for placement * <https://review.opendev.org/634551> libvirt: report pmem namespaces resources by provider tree * <https://review.opendev.org/660852> Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * <https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports> Nova: support move ops with qos ports * <https://review.opendev.org/664689> Nova: get_ksa_adapter: nix by-service-type confgrp hack * <https://review.opendev.org/666202> Blazar: Create placement client for each request * <https://review.opendev.org/667952> nova: Support filtering of hosts by forbidden aggregates * <https://review.opendev.org/669079> blazar: Send global_request_id for tracing calls * <https://review.opendev.org/668252> Nova: Update HostState.\*\_allocation_ratio earlier * <https://review.opendev.org/670112> Nova: WIP: Add a placement audit command * <https://review.opendev.org/586960> zun: [WIP] Use placement for unified resource management * <https://review.opendev.org/671478> Kayobe: Build placement images by default * <https://review.opendev.org/671312> blazar: Fix placement operations in multi-region deployments * <https://review.opendev.org/672870> Watcher: Getting data from placement when updating datamodel * <https://review.opendev.org/671793> Nova: libvirt: Start reporting PCPU inventory to placement * <https://review.opendev.org/672649> chef: placement_api: create valid apache config before installing package * <https://review.opendev.org/670696> tempest: Add placement API methods for testing routed provider nets * <https://review.opendev.org/672678> openstack-helm: Build placement in OSH-images # End If we get the chance, it will be interesting to start working with placement with 1 million providers. Just to see. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

1 0

[openstack-ansible] weekly update
by Mohammed Naser 26 Jul '19

26 Jul '19

Hi everyone, I'm going to try doing these more often to keep everyone updated on the current state of openstack-ansible (mostly around what is discussed during our office hours / meetings). - Dmitriy has been working on removing unused code in playbooks, mostly around logs as we move to 100% systemd-journald. - We're discussing the proposal to remove os-log-dir-setup.yml, as well as rsyslog_client and rsyslog_server roles as they aren’t used anymore with the move to systemd-journald - We're also discussing removing all the complex bind mounts, which would clean it and simplify the code base. - Our continuous integration has been stable but needs upgrade jobs and increased coverage. Our coverage would go up significantly by running aio_lxc for all supported systems. - Jonathan's work on binding to management IPs is making progress (and seems to be at the Galera stage). - CentOS CI has been very slow and nobody knows why. It seems to be something we've regressed at some point. - A unicast flood issue affecting deployments using linuxbridge that happened inside os-vif needs to be patched. It's still pending os-vif to patch. - Namrata mentioned an issue during the upgrade from Rocky to Stein (when upgrading WSREP SST method from xtrabackup-V2 to mariabackup) that will have to be fixed is Stein. That's something that's in progress now. Thanks everyone for reading! Regards, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

2 1

[cinder][ceph][RBD mirroring] Error while creating a mirrored volume
by Ajitha Robert 26 Jul '19

26 Jul '19

Hi all Following picture shows my setup [image: image.png] I m facing following error when i issue a command to enable rbd mirroring on a particular image either manually or through cinder. *"rbd::mirror::InstanceWatcher: C_NotifyInstanceRequest: 0x7fb364025310 finish: resending after timeout"* in rbd-mirroring.log and getting *"failed to get omap key: client_0921b3ce-716a-4462-b3cf-86b5102bfd9"* error in osd.log. . Should I enable anything else apart from timeout parametes, if the distance between the two ceph clusters is greater than 300km? when I create a bootable volume of 100 GB with a glance image.Image get downloaded and from cinder, volume log throws with "volume is busy deleting volume that has snapshot" . Image was enabled with exclusive lock, journaling, layering, object-map, fast-diff and deep-flatten Cinder volume is in error state but the rbd image is created in primary but not in secondary. pls check the logs 1)Log for manual rbd image creation http://paste.openstack.org/show/754766/ 2)Log for 16gb volume created from cinder, status in cinder volume is available http://paste.openstack.org/show/754767/ 3)Log for 100gb volume created from cinder, status in cinder volume is error http://paste.openstack.org/show/754769/ -- *Regards,Ajitha R*

1 0

[placement] update 19-27
by Chris Dent 26 Jul '19

26 Jul '19

HTML: https://anticdent.org/placement-update-19-27.html Pupdate 19-27 is here and now. # Most Important Of the features we planned to do this cycle, all are done save one: consumer types (in progress, see below). This means we have a good opportunity to focus on documentation, performance, and improving the codebase for maintainability. You do not need permission to work on these things. If you find a problem and know how to fix it, fix it. If you are not sure about the solution, please discuss it on this email list or in the `#openstack-placement` IRC channel. This also means we're in a good position to help review changes that use placement in other projects. The Foundation needs to know how much, if any, Placement time will be needed in Shanghai. I started a [thread](http://lists.openstack.org/pipermail/openstack-discuss/2019-July/00… and an [etherpad](https://etherpad.openstack.org/p/placement-shanghai-ptg). # What's Changed * The `same_subtree` query parameter has merged as [microversion 1.36](https://docs.openstack.org/placement/latest/placement-api-microversio…. This enables a form of nested provider affinity: "these providers must all share the same ancestor". * The placement projects have been updated (pending merge) to match the [Python 3 test runtimes for Train](https://governance.openstack.org/tc/goals/train/python3-updates.html) community wide goal. Since the projects have been Python3-enabled from the start, this was mostly a matter of aligning configurations with community-wide norms. When Python 3.8 becomes available we should get on that sooner than later to catch issues early. # Specs/Features All placement specs have merged. Thanks to everyone for the frequent reviews and quick followups. Some non-placement specs are listed in the Other section below. # Stories/Bugs (Numbers in () are the change since the last pupdate.) There are 23 (0) stories in [the placement group](https://storyboard.openstack.org/#!/project_group/placement). 0 (0) are [untagged](https://storyboard.openstack.org/#!/worklist/580). 3 (1) are [bugs](https://storyboard.openstack.org/#!/worklist/574). 5 (0) are [cleanups](https://storyboard.openstack.org/#!/worklist/575). 11 (0) are [rfes](https://storyboard.openstack.org/#!/worklist/594). 4 (0) are [docs](https://storyboard.openstack.org/#!/worklist/637). If you're interested in helping out with placement, those stories are good places to look. * Placement related nova [bugs not yet in progress](https://goo.gl/TgiPXb) on launchpad: 16 (0). * Placement related nova [in progress bugs](https://goo.gl/vzGGDQ) on launchpad: 4 (0). # osc-placement osc-placement is currently behind by 11 microversions. * <https://review.opendev.org/666542> Add support for multiple member_of. This is stuck and needs input from users of the tool. # Main Themes ## Consumer Types Adding a type to consumers will allow them to be grouped for various purposes, including quota accounting. * <https://review.opendev.org/#/q/topic:bp/support-consumer-types> This is currently the migration to add the necessary table and column. ## Cleanup Cleanup is an overarching theme related to improving documentation, performance and the maintainability of the code. The changes we are making this cycle are fairly complex to use and are fairly complex to write, so it is good that we're going to have plenty of time to clean and clarify all these things. As mentioned last week, one of the important cleanup tasks that is not yet in progress is updating the [gabbit](https://opendev.org/openstack/placement/src/branch/master/gate/gabb… that creates the nested topology that's used in nested performance testing. The topology there is simple, unrealistic, and doesn't sufficiently exercise the several features that may be used during a query that desires a nested response. This needs to be someone who is more closely related to real world use of nested than me. efried? gibi? Another cleanup that needs to start is satisfying the community wide goal of [PDF doc generation](https://storyboard.openstack.org/#!/story/2006110). Anyone know if there is a cookbook for this? # Other Placement Miscellaneous changes can be found in [the usual place](https://review.opendev.org/#/q/project:openstack/placement+status:op…. There are three [os-traits changes](https://review.opendev.org/#/q/project:openstack/os-traits+status:… being discussed. And one [os-resource-classes change](https://review.opendev.org/#/q/project:openstack/os-resource-classe…. # Other Service Users New discoveries are added to the end. Merged stuff is removed. Anything that has had no activity in 4 weeks has been removed. * <https://review.openstack.org/#/q/topic:bug/1819923> Nova: nova-manage: heal port allocations * <https://review.openstack.org/650188> nova-spec: Allow compute nodes to use DISK_GB from shared storage RP * <https://review.opendev.org/659233> Cyborg: Placement report * <https://review.opendev.org/662229> helm: add placement chart * <https://review.opendev.org/656023> Nova: Use OpenStack SDK for placement * <https://review.opendev.org/612497> Nova: Spec: Provider config YAML file * <https://review.opendev.org/634551> libvirt: report pmem namespaces resources by provider tree * <https://review.opendev.org/660852> Nova: Remove PlacementAPIConnectFailure handling from AggregateAPI * <https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports> Nova: support move ops with qos ports * <https://review.opendev.org/664689> Nova: get_ksa_adapter: nix by-service-type confgrp hack * <https://review.opendev.org/664862> OSA: Add nova placement to placement migration * <https://review.opendev.org/657796> Nova: Defaults missing group_policy to 'none' * <https://review.opendev.org/666202> Blazar: Create placement client for each request * <https://review.opendev.org/669309> tempest: Define the Integrated-gate-placement gate template * <https://review.opendev.org/668263> Nova: Restore RT.old_resources if ComputeNode.save() fails * <https://review.opendev.org/667952> nova: Support filtering of hosts by forbidden aggregates * <https://review.opendev.org/669079> blazar: Send global_request_id for tracing calls * <https://review.opendev.org/668252> Nova: Update HostState.\*\_allocation_ratio earlier * <https://review.opendev.org/670105> neutron: segments: fix rp inventory update * <https://review.opendev.org/670112> Nova: WIP: Add a placement audit command * <https://review.opendev.org/668526> OSA: Add placement client for neutron * <https://review.opendev.org/586960> zun: [WIP] Use placement for unified resource management # End A colleague suggested yesterday that the universe doesn't have an over subscription problem, rather there's localized contention, and what we really have is a placement problem. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

2 1