August 2019 - openstack-discuss - lists.openstack.org

[goals][IPv6-Only Deployments and Testing] Week R-8 Update
by Ghanshyam Mann 25 Aug '19

25 Aug '19

Hello Everyone, Below is the progress on Ipv6 goal during R8 week. I have finished with the project having zuulv3 jobs. The projects having zuulv2 jobs will be my next take (I have not explored the difficulties/complexity yet). Summary: * Number of Ipv6 jobs proposed Projects: 28 * Number of pass projects: 17 ** Number of project merged out of pass project: 11 * Number of failing projects: 11 Storyboard: ========= - https://storyboard.openstack.org/#!/story/2005477 Current status: ============ 1. Cinder fix is merged[1], Thanks Gorka. We need to merge the devstack patch now. 2. Configuring the tempest test regex to run only smoke tests which can be extended to include future IPv6 tests also. Running all test is not actually required as such in IPv6 job but if any project wants to run all then also fine. Example: [1] 3. Murano, zun, solumn, cloudkitty are merged. 4. Murano default dns is also on IPv6 env. 5. For Monasca, kafka was not working for IPv6 but witek is upgrading the Kafka version in Monasca. I will rebase IPv6 job patch on top of that and check the result. 6. This week new projects ipv6 jobs patch and status: - Zun: link: https://review.opendev.org/#/c/677496/ status: job is passed with few devstack plugin fixes and patch is merged. - Searchlight: links: https://review.opendev.org/#/c/678391/ status: There are no integration tests in searchlight as of now so after talking to Trinh (PTL), I am just adding a no-test job to verify that searchlight can be deployed on IPv6 or not. Later when we will have tests then those can be added in this job. IPv6 missing support found: ===================== 1. https://review.opendev.org/#/c/673397/ 2. https://review.opendev.org/#/c/673449/ 3. https://review.opendev.org/#/c/677524/ How you can help: ============== - Each project needs to look for and review the ipv6 job patch. - Verify it works fine on ipv6 and no ipv4 used in conf etc - Any other specific scenario needs to be added as part of project IPv6 verification. - Help on debugging and fix the bug in IPv6 job is failing. Everything related to this goal can be found under this topic: Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(stat… How to define and run new IPv6 Job on project side: ======================================= - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing Review suggestion: ============== - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with that point of view. If anything missing, comment on patch. - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var setting. But if your project needs more specific verification then it can be added in project side job as post-run playbooks as described in wiki page[3]. [1] https://review.opendev.org/#/c/677524/ [1] https://zuul.opendev.org/t/openstack/build/5b7b823d6faa4f5393b4c46d36e15d80… [2] https://review.opendev.org/#/c/676857/ [3] https://review.opendev.org/#/c/676900/ [4] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing -gmann

1 0

[nova] critical bug around reload/upgrades
by Mohammed Naser 24 Aug '19

24 Aug '19

Hello: I've discussed this for quite sometime with Dan over IRC and a bit with Zane as well, but basically, Nova thinks that when it gets a reload (aka SIGHUP), nothing else has occurred. However, oslo.service actually calls stop(), reload() then start() again, which potentially kills all RPC. This has caused a pretty big issue in our gates and it also means that the whole idea behind 'reload nova-compute while upgrading to refresh info' concept is fundamentally broken. I tried to do some work on this here, however, I wasn't really able to get to the bottom of it. There seems to be a decision that needs to be taken in terms of .. do we change what reload() actually means in oslo_service (it actually is more like a restart, not a reload) or does Nova (and other projects) change their implementation in assuming what reload() does? https://review.openstack.org/#/c/641907/ This seems to have been floating around for a really long time, so I'd be happy to work with someone to find the fix (and we can totally test it inside OpenStack Ansible by reloading instead of restarting). Thanks! Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

4 4

[keystone] Keystone Team Update - Week of 19 August 2019
by Colleen Murphy 24 Aug '19

24 Aug '19

# Keystone Team Update - Week of 19 August 2019 ## News ### Development focus Feature freeze as well as a number of other deadlines are fast approaching. The development focus for the next three weeks should be: * reviews for changes implementing specs: - app cred access rules[1] - resource options for all resources[2] - immutable resources[3] - renewable group membership[4] * finish implementing and reviewing remaining system-scope[5]/default roles[6] migrations (deadline is feature freeze, Sept 9-13) * reviews for keystonemiddleware and keystoneauth[7] (final release Sept 2-6) * reviews for python-keystoneclient[8] (final release Sept 9-13) * helping the requirements team with any requirements issues[9] before requirements freeze (Sept 9-13) * completing community goals (follow the PDF generation news[10]) Additionally, we're still facing issues with instability of our unit test jobs. Improving the efficiency of our unit tests in order to avoid frequent timeouts would go a long way to helping avoid the feature freeze gate crunch. [1] https://review.opendev.org/#/q/is:open+topic:bp/whitelist-extension-for-app… [2] https://review.opendev.org/678322 [3] https://review.opendev.org/#/q/is:open+topic:immutable-resources [4] https://review.opendev.org/677469 [5] https://bugs.launchpad.net/keystone/+bugs?field.tag=system-scope [6] https://bugs.launchpad.net/keystone/+bugs?field.tag=default-roles [7] https://review.opendev.org/#/q/is:open+(project:openstack/keystoneauth+OR+p… [8] https://review.opendev.org/#/q/is:open+project:openstack/python-keystonecli… [9] https://bugs.launchpad.net/keystone/+bug/1839393 [10] http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008506.h… ### keystoneauth session retries The heat team reported an issue in keystoneauth[11] where the way in which heat currently uses keystoneauth sessions is unable to take advantage of the request retry logic that is currently exposed only in the adapter. The proposal to fix the issue[12] exposes one of the retry options (connect_retries) in the session object. We've been in discussions on the bug report, on the patch, and in IRC[13][14] about whether this is the best approach and whether it would be better to change the way heat uses keystoneauth (would require a massive rewrite) or to expose the request retry options on the auth plugin in order to localize it to keystone requests (but that's an awkward home for it too). We've also been in disagreement about whether this change constitutes a feature or a bugfix with regards to backportability. [11] https://bugs.launchpad.net/keystoneauth/+bug/1840235 [12] https://review.opendev.org/676648 [13] http://eavesdrop.openstack.org/irclogs/%23openstack-sdks/%23openstack-sdks.… [14] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-k… ## Action Items * Vishakha to look into oauthlib requirements issue[15] * Kristi to propose spec to backlog about merging federation and identity backends [16] https://bugs.launchpad.net/keystone/+bug/1839393 ## Office Hours When there are topics to cover, the keystone team holds office hours on Tuesdays at 17:00 UTC. There will be no office hours this week or next week. Add topics you would like to see covered during office hours to the etherpad: https://etherpad.openstack.org/p/keystone-office-hours-topics ## Open Specs Train specs: https://bit.ly/2uZ2tRl Ongoing specs: https://bit.ly/2OyDLTh ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 13 changes this week, including changes to implement system scope for the endpoint groups API. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 68 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ### Priority Reviews * Train Roadmap Stories - System scope/default roles (https://trello.com/c/ERo50T7r , https://trello.com/c/RlYyb4DU) + https://review.opendev.org/#/q/status:open+topic:implement-default-roles+la… + https://review.opendev.org/#/q/status:open+topic:trust-policies + https://review.opendev.org/#/q/topic:bug/1805409 - Federated attributes for users (https://trello.com/c/dEmSumDQ) + https://review.opendev.org/#/q/status:open+topic:bp/support-federated-attr - Application credential access rules (https://trello.com/c/dJsWMI4W) + https://review.opendev.org/#/q/status:open+topic:bp/whitelist-extension-for… - Immutable resources (https://trello.com/c/clIb4qMq) + https://review.opendev.org/#/q/topic:immutable-resources - Resource options for all (https://trello.com/c/8ML6kvig) + https://review.opendev.org/678322 * Needs Discussion - Allow initializing session with connection retries https://review.opendev.org/676648 * Oldest - OpenID Connect improved support (spec) https://review.opendev.org/373983 * Closes bugs - Cleanup session on delete https://review.opendev.org/674139 ## Bugs This week we opened 1 new bugs and closed 2. Bugs opened (1) Bug #1840647 (keystone:Undecided) opened by Nikita Kalyanov https://bugs.launchpad.net/keystone/+bug/1840647 ) Bugs fixed (2) Bug #1840291 (keystone:Medium) fixed by Rabi Mishra https://bugs.launchpad.net/keystone/+bug/1840291 Bug #1818734 (keystone:Low) fixed by Colleen Murphy https://bugs.launchpad.net/keystone/+bug/1818734 ## Milestone Outlook https://releases.openstack.org/train/schedule.html Final release for non-client libraries (keystoneauth, keystonemiddleware) is in two weeks. Feature freeze and final client release is in three weeks. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter

1 0

[goal][python3] Train unit tests weekly update (goal-3)
by Corey Bryant 23 Aug '19

23 Aug '19

This is the goal-3 weekly update for the "Update Python 3 test runtimes for Train" goal [1]. There are 3 weeks remaining for completion of Train community goals [2]. == How can you help? == If your project has failing tests please take a look and help fix. Python 3.7 unit tests will be self-testing in Zuul. Failing patches: https://review.openstack.org/#/q/topic:python3-train +status:open+(+label:Verified-1+OR+label:Verified-2+) If your project has patches with successful tests please help get them merged. Open patches needing reviews: https://review.openstack.org/#/q/topic:python3 -train+is:open Patch automation scripts needing review: https://review.opendev.org/#/c/666934 == Ongoing Work == Thank you to all who have contributed their time and fixes to enable patches to land. We're down to 15 projects with failing tests. == Completed Work == All patches have been submitted to all applicable projects for this goal. Merged patches: https://review.openstack.org/#/q/topic:python3-train +is:merged == What's the Goal? == To ensure (in the Train cycle) that all official OpenStack repositories with Python 3 unit tests are exclusively using the 'openstack-python3-train-jobs' Zuul template or one of its variants (e.g. 'openstack-python3-train-jobs-neutron') to run unit tests, and that tests are passing. This will ensure that all official projects are running py36 and py37 unit tests in Train. For complete details please see [1]. == Reference Material == [1] Goal description: https://governance.openstack.org/tc/goals/train/ python3-updates.html [2] Train release schedule: https://releases.openstack.org/train /schedule.html (see R-5 for "Train Community Goals Completed") Storyboard: https://storyboard.openstack.org/#!/story/2005924 Porting to Python 3.7: https://docs.python.org/3/whatsnew/3.7.html#porting-to-python-3-7 Python Update Process: https://opendev.org/openstack/governance/src/branch/master/resolutions/2018… Train runtimes: https://opendev.org/openstack/governance/src/branch/master/reference/runtim… train.rst Thanks,

1 0

[nova] The pros/cons for libvirt persistent assignment and DB persistent assignment.
by Alex Xu 23 Aug '19

23 Aug '19

We get a lot of discussion on how to do the claim for the vpmem. There are a few points we are trying to match: * Avoid race problem. (the current VGPU assignment has been found having race issue https://launchpad.net/bugs/1836204) * Avoid the device assignment management to be virt driver and platform-specific. * Keep it simple. Currently, we go through two solutions here. This email is going to summary the pros/cons of these two solutions. #1 Without Nova DB persistent for the assignment info, depends on hypervisor persistent it. The idea is adding VirtDriver.claim/unclaim_for_instance(instance_uuid, flavor_id) interface. The assignment info is populated from hypervisor when nova-compute startup. And keep in the memory of VirtDriver. The instance_uuid is used to distinguish the claim from the different instance. The flavor_id is used for the same host resize, to distinguish the claim for source and target. This virt driver method is being invoked inside ResourceTracker to avoid the race problem. There is no any nova DB persistent for the assignment info. https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:ma… pros: * Hidden all the device detail and virt driver detail inside the virt driver. * Less upgrade issue in the future since it doesn't involve any nova DB model change * Expecting as simple implementation since everything inside virt driver. cons: * Two cases are being found, the domain XML being lost for Libvirt virt driver. And we don't know other hypervisor behavior yet. * For the same host resize, the source and target instance are sharing single one domain XML. After the libvirt virt driver updated the domain XML to the target instance, the source instance's assignment information will be lost when a nova-compute restart happened. That means the resized instance can't be revert, the only choice for the user is to confirm the resize. * For live migration, the target host's domain XML will be cleanup by libvirt after a host restart. The assignment information is lost before nova-compute startup and doing a cleanup. * Can not support the same host cold migration. Since we need a way to identify the source and target instance's assignment in memory. But the same host cold migration means the same instance UUID and same flavor ID, there isn't another else can be used to distinguish the assignment. * There are workarounds added for above points, the code becomes fragile. #2 With nova DB persistent, but using virt driver specific blob to store virt driver specific info. The idea is persistent the assignment for instance into DB. The resource tracker gets available resources from virt driver. The resource tracker will calculate on the fly based on available resources and assigned resources from instance DB. The new field ·instance.resources· is designed for supporting virt driver specific metadata, then hidden the virt driver and platform detail from RT. https://etherpad.openstack.org/p/vpmems-non-virt-driver-specific-new pros: * Persistent assignment into instance object. Avoid the corner case we lost the assignment. * The ResourceTracker is responsible for doing the claim job. This is more reliable and no race problem, since ResourceTracker works very well for a long time. * The virt driver specific json-blob hidden the virt driver/platform detail from the ResourceTracker. * The free resource is calculated on the fly, keeping the implementation simple. Actually, the RT just provides a point to do the claim, needn't involve the complex of RT.update_available_resources cons: * Doesn't like PCIManager which has both instance side and host side persistent info. On the fly calculation should take care of the orphaned instance(the instance is deleted from DB, but still existing on the host), so actually, it isn't unresolvable issue. And it isn't too hard to upgrade to have host side persistent info in the future if we want. * Data model change for the original proposal. Need review to decide the data model enough generic Currently, Sean, Eric and I prefer the #2 now since the #1 has flaws for the same host resize and live migration can't be skipped by design. Looking for more feedback, and will appreciate it! Thanks Alex

4 5

[openstack-helm] k8s endpoints disappear sometimes
by 김하건 23 Aug '19

23 Aug '19

1 0

[openstack-dev][magnum] Nominating brtknr for Magnum core
by Feilong Wang 23 Aug '19

23 Aug '19

Hi team, I would like to propose adding Bharat Kunwar(brtknr) for the Magnum core team. He has been an awesome contributor since joining the Magnum team. And now he is currently the most active non-core reviewer on Magnum projects for the last 180 days[1]. Bharat has great technical expertise and contributed many high quality patches/reviews. I'm sure he would be an excellent addition to the team. If no one objects, I'll proceed and add him in a week from now. Thanks. [1] https://www.stackalytics.com/report/contribution/magnum-group/180 -- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Head of R&D Catalyst Cloud - Cloud Native New Zealand Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------

2 1

[all][elections] Combined PTL/TC Election Season
by Jeremy Stanley 23 Aug '19

23 Aug '19

Election details: https://governance.openstack.org/election/ The nomination period officially begins Aug 27, 2019 23:45 UTC. Please read the stipulations and timelines for candidates and electorate contained in this governance documentation. Due to circumstances of timing, PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you which to discuss in private please email any of the election officials[1] so that we may address your concerns. Thank you, [1] https://governance.openstack.org/election/#election-officials -- Jeremy Stanley, on behalf of the technical election officials

1 0

Nova causes MySQL timeouts
by Albert Braden 23 Aug '19

23 Aug '19

It looks like nova is keeping mysql connections open until they time out. Is there a way to stop these error messages? Aborted connection 10726 to db: 'nova' user: 'nova' host: 'asdf' (Got timeout reading communication packets)

1 0

[nova][ops] What should the compute service delete behavior be wrt resource providers with allocations?
by Matt Riedemann 22 Aug '19

22 Aug '19

Before [1] when deleting a compute service in the API we did not check to see if the compute service was hosting any instances and just blindly deleted the service and related compute_node(s) records which orphaned the resource provider(s) for those nodes. With [2] we built on that and would cleanup the (first [3]) compute node resource provider by first deleting any allocations for instances still on that host - which because of the check in [1] should be none - and then deleted the resource provider itself. [2] forgot about ironic where a single compute service can be managing multiple (hundreds or even thousands) of baremetal compute nodes so I wrote [3] to delete *all* resource providers for compute nodes tied to the service - again barring there being any instances running on the service because of the check added in [1]. What we've failed to realize until recently is that there are cases where deleting the resource provider can still fail because there are allocations we haven't cleaned up, namely: 1. Residual allocations for evacuated instances from a source host. 2. Allocations held by a migration record for an unconfirmed (or not yet complete) migration. Because the delete_resource_provider method isn't checking for those, we can get ResourceProviderInUse errors which are then ignored [4]. Since that error is ignored, we continue on to delete the compute service record [5], effectively orphaning the providers (which is what [2] was meant to fix). I have recreated the evacuate scenario in a functional test here [6]. The question is what should we do about the fix? I'm getting lost thinking about this in a vacuum so trying to get some others to help think about it. Clearly with [1] we said you shouldn't be able to delete a compute service that has instances on it because that corrupts our resource tracking system. If we extend that to any allocations held against providers for that compute service, then the fix might be as simple as not ignoring the ResourceProviderInUse error and fail if we can't delete the provider(s). The question I'm struggling with is what does an operator do for the two cases mentioned above, not-yet-complete migrations and evacuated instances? For migrations, that seems pretty simple - wait for the migration to complete and confirm it (reverting a cold migration or resize would put the instance back on the compute service host you're trying to delete). The nastier thing is the allocations tied to an evacuated instance since those don't get cleaned up until the compute service is restarted [7]. If the operator never intends on restarting that compute service and just wants to clear the data, then they have to manually delete the allocations for the resource providers associated with that host before they can delete the compute service, which kind of sucks. What are our options? 1. Don't delete the compute service if we can't cleanup all resource providers - make sure to not orphan any providers. Manual cleanup may be necessary by the operator. 2. Change delete_resource_provider cascade=True logic to remove all allocations for the provider before deleting it, i.e. for not-yet-complete migrations and evacuated instances. For the evacuated instance allocations this is likely OK since restarting the source compute service is going to do that cleanup anyway. Also, if you delete the source compute service during a migration, confirming or reverting the resize later will likely fail since we'd be casting to something that is gone (and we'd orphan those allocations). Maybe we need a functional recreate test for the unconfirmed migration scenario before deciding on this? 3. Other things I'm not thinking of? Should we add a force parameter to the API to allow the operator to forcefully delete (#2 above) if #1 fails? Force parameters are hacky and usually seem to cause more problems than they solve, but it does put the control in the operators hands. If we did remove allocations for an instance when deleting it's compute service host, the operator should be able to get them back by running the "nova-manage placement heal_allocations" CLI - assuming they restart the compute service on that host. This would have to be tested of course. Help me Obi-Wan Kenobi. You're my only hope. [1] https://review.opendev.org/#/q/I0bd63b655ad3d3d39af8d15c781ce0a45efc8e3a [2] https://review.opendev.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 [3] https://review.opendev.org/#/c/657016/ [4] https://github.com/openstack/nova/blob/cb0cfc90e1e03e82c42187ec60f46fb8fd59… [5] https://github.com/openstack/nova/blob/cb0cfc90e1e03e82c42187ec60f46fb8fd59… [6] https://review.opendev.org/#/c/663737/ [7] https://github.com/openstack/nova/blob/cb0cfc90e1e03e82c42187ec60f46fb8fd59… -- Thanks, Matt

6 16