- openstack-discuss - lists.openstack.org

[nova] The test of NUMA aware live migration
by wang.ya 24 Sep '19

24 Sep '19

Hi: The main code of NUMA aware live migration was merged. I’m testing it recently. If only set NUMA property(‘hw:numa_nodes’, ‘hw:numa_cpus’, ‘hw:numa_mem’), it works well. But if add the property “hw:cpu_policy='dedicated'”, it will not correct after serval live migrations. Which means the live migrate can be success, but the vCPU pin are not correct(two instance have serval same vCPU pin on same host). Below is my test steps. env: code: master branch (build on 16 September 2019, include the patches of NUMA aware live migration) three compute node: - s1: 24C, 48G (2 NUMA nodes) - stein-2: 12C, 24G (2 NUMA nodes) - stein-3: 8C, 16G (2 NUMA nodes) flavor1 (2c2g): hw:cpu_policy='dedicated', hw:numa_cpus.0='0', hw:numa_cpus.1='1', hw:numa_mem.0='1024', hw:numa_mem.1='1024', hw:numa_nodes='2' flavor2 (4c4g): hw:cpu_policy='dedicated', hw:numa_cpus.0='0,1,2', hw:numa_cpus.1='3', hw:numa_mem.0='1024', hw:numa_mem.1='3072', hw:numa_nodes='2' image has no property. I create four instances(2*flavor1, 2* flavor2), then begin live migration on by one(one instance live migrate done, next instance begin live migrate) and check the vCPU pin whether is correct. After serval live migrations, the vCPU pin will not correct. (You can find full migration list in attached file). The last live migrate is: +-----+--------------------------------------+-------------+-----------+----------------+--------------+----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+ | Id | UUID | Source Node | Dest Node | Source Compute | Dest Compute | Dest Host | Status | Instance UUID | Old Flavor | New Flavor | Created At | Updated At | Type | +-----+--------------------------------------+-------------+-----------+----------------+--------------+----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+ | 470 | 2a9ba183-4f91-4fbf-93cf-6f0e55cc085a | s1 | stein-3 | s1 | stein-3 | 172.16.130.153 | completed | bf0466f6-4815-4824-8586-899817207564 | 1 | 1 | 2019-09-17T10:28:46.000000 | 2019-09-17T10:29:09.000000 | live-migration | | 469 | c05ea0e8-f040-463e-8957-a59f70ed8bf6 | s1 | stein-3 | s1 | stein-3 | 172.16.130.153 | completed | a3ec7a29-80de-4541-989d-4b9f4377f0bd | 1 | 1 | 2019-09-17T10:28:21.000000 | 2019-09-17T10:28:45.000000 | live-migration | | 468 | cef4c609-157e-4b39-b6cc-f5528d49c75a | s1 | stein-2 | s1 | stein-2 | 172.16.130.152 | completed | 83dab721-3343-436d-bee7-f5ffc0d0d38d | 4 | 4 | 2019-09-17T10:27:57.000000 | 2019-09-17T10:28:21.000000 | live-migration | | 467 | 5471e441-2a50-465a-bb63-3fe1bb2e81b9 | s1 | stein-2 | s1 | stein-2 | 172.16.130.152 | completed | e3c19fbe-7b94-4a65-a803-51daa9934378 | 4 | 4 | 2019-09-17T10:27:32.000000 | 2019-09-17T10:27:57.000000 | live-migration | There are two instances land on stein-3, and the two instances have same vCPU pin: (nova-libvirt)[root@stein-3 /]# virsh list --all Id Name State ---------------------------------------------------- 32 instance-00000025 running 33 instance-00000024 running (nova-libvirt)[root@stein-3 /]# virsh vcpupin 32 VCPU: CPU Affinity ---------------------------------- 0: 2 1: 7 (nova-libvirt)[root@stein-3 /]# virsh vcpupin 33 VCPU: CPU Affinity ---------------------------------- 0: 2 1: 7 I checked the nova compute’s log on stein-3(you can find the log in attached log), then I found ‘host_topology’ isn’t updated when ‘hardware.numa_fit_instance_to_host’ be called in claims. ‘host_topology’ is the property of ‘objects.ComputeNode’ and it’s cached in ‘ResourceTracker’, it will use cached ‘cn’ to build ‘claim’ when ‘check_can_live_migrate_destination’ called. Therefore, I guess the cache was not updated or updated too late or some other reason. I also checked the database, the NUMA topology of the two instances have same vCPU pin: “[0,2], [1,7]”, and the compute node: stein-3 only has vCPU pin: “[2], [7]”. Please correct me if there is something wrong :) Best Regards

4 4

[all][tc] What happened in OpenStack Governance last week
by Jean-Philippe Evrard 24 Sep '19

24 Sep '19

Hello everyone, Here are a few things that happened recently: We introduced "comparison of Official Group Structures" [1]. It should help you navigate better our structures and process, if you want to start a new initiative. We now have a plan to reduce the amount of members of our Technical Committee going forward, down to 9 in Q3 2020 [2]. We've clarified the completion criteria for the PDF goal [3]. We've got fresh PTLs! Tetsuro Nakamura for Placement, Ian Y. Choi for I18n, and Monty Taylor for OpenStackSDK. Congratulations! The Ussuri runtimes are out [4][5]. Ussuri will be the first release to not require python2. Projects are encouraged to move forward! Reminder: Projects are allowed to test extra runtimes, like python2.7, if they want to. Note: Most, if not all, initiatives above were started by TC members. Keep in mind you don't need to be in the TC to propose patches in governance! For example, Tom Barron proposed quite a few patches recently, thank you Tom! :) Regards, Jean-Philippe Evrard (evrardjp) [1]: https://review.opendev.org/668093 [2]: https://review.opendev.org/681266 [3]: https://review.opendev.org/679654 [4]: https://governance.openstack.org/tc/reference/runtimes/ussuri.html [5]: https://review.opendev.org/679798

1 0

[keystone] Ussuri Virtual PTGs
by Colleen Murphy 24 Sep '19

24 Sep '19

Hi team, As we've discussed in IRC and at the virtual midcycle, we'll be holding two virtual planning meetings in place of meeting in-person at the PTG in Shanghai. We'll hold one before the real event in order to prepare for the Forum, and one following it in order to disseminate information and recalibrate. I've drawn up a straw man agenda and proposed meeting dates/times in our brainstorming etherpad: https://etherpad.openstack.org/p/keystone-shanghai-ptg The proposal is to have the pre-PTG on Tuesday, October 29 at 14:00 UTC and the post-PTG on Tuesday, November 12 at 15:00 UTC. (Every time I send a scheduling poll we usually settle on Tuesday mornings anyway so I'm just cutting to the chase.) Please speak up if you want to attend but the proposed dates or times do not work for you, we still have plenty of time to play with the schedule. If no one voices concerns by Tuesday, October 8 we'll consider the dates final. The agenda and topics are also open for feedback and additional topic suggestions are still welcome. We'll plan on using jitsi.org as our conference tool but can fall back to gotomeeting if necessary. Colleen

1 0

[FFE][designate] Pool Manager Removal and refactor
by Graham Hayes 24 Sep '19

24 Sep '19

Hi All, I am granting a FFE for 3 patches: https://review.opendev.org/#/c/678432/ - Refactored service layer https://review.opendev.org/#/c/665476/22 - Removed deprecated pool-manager implementation https://review.opendev.org/#/c/657289/ - Removed deprecated powerdns 3 driver These patches remove a duplicate code paths that have been deprecated for an extended period of time, and reduce a lot of the complexity in our implementation. Please contact me if there is any issues or concern. Thanks, Graham

1 0

[requirements] [ironic] FFE for python-ironicclient 3.1.0
by Dmitry Tantsur 24 Sep '19

24 Sep '19

Hi all, we would like to request an exception to release python-ironicclient 3.1.0 from the stable/train branch. The current 3.0.0 release has several issues, one of them [1] is critical and breaks no-auth mode (used e.g. in bifrost). I'm also proposing [2] to exclude python-ironicclient 3.0.0 from requirement. A release request will be posted once [3] merges. The minor version bump is because we've made the previously existing implicit oslo.config dependency explicit [4]. I don't believe the new release will break anyone who is not broken by 3.0.0 already. Thanks, Dmitry [1] https://storyboard.openstack.org/#!/story/2006600 [2] https://review.opendev.org/#/c/684376/ [3] https://review.opendev.org/#/c/684363/ [4] https://review.opendev.org/#/c/684281/

2 3

[requirements][monasca] Request for FFE for monasca-common
by Witek Bedyk 24 Sep '19

24 Sep '19

Hello Requirements Team, I'm requesting FFE for monasca-common library. The new requested version 2.16.1 includes critical fix for Confluent Kafka client. The library is used only by Monasca project. https://review.opendev.org/683986 Thanks Witek

2 3

[keystone] presence of policy.json breaks Keystone?
by Bernd Bausch 24 Sep '19

24 Sep '19

This is on a stable Stein Devstack. Problem description: ubuntu@devstack:~$ oslopolicy-sample-generator --namespace keystone >/etc/keystone/policy.json ubuntu@devstack:~$ openstack user list Internal Server Error (HTTP 500) Note that I did not modify the policy.json file above. It's mere presence is sufficient to cause the problem. When I remove it and restart Keystone, the problem goes away. The Keystone log contains a huge stacktrace with two methods in oslopolicy/_checks.py playing ping-pong with each other until they give up with RuntimeError: maximum recursion depth exceeded. This only happens with Keystone. Nova and Cinder (which also keep policy in code) are fine. This looks like a bug, but I didn't find it in launchpad. Is there a workaround? I would like to use a modified Keystone policy in a training course. Thanks for any feedback. Bernd.

3 2

[ops] ops meetups team minutes
by Chris Morgan 24 Sep '19

24 Sep '19

The OpenStack Ops Meetups team met briefly today on IRC, minutes are below. The team is working on some sessions for the Forum in Shanghai, together with two mid-cycle meetups in 2020, possibly in London and then South Korea. We'll share more details of those when we can. We continue to meet at 10am EDT on #openstack-operators weekly. Meeting ended Tue Sep 24 14:35:02 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) 10:35 AM Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… 10:35 AM O<•openstack> Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… 10:35 AM Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… Cheers Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[Cloudkitty] Stepping down from CK Core team
by Guillaume Espanel 24 Sep '19

24 Sep '19

Hi all, I am stepping down from the Cloudkitty Core team: I have not been able to contribute for a while, and I'm moving to a new position that doesn't involve Cloudkitty at all. -- Guillaume Espanel Objectif Libre www.objectif-libre.com Infrastructure et Formations Linux

1 0

[gnocchi] uWSGI error
by Ignazio Cassano 24 Sep '19

24 Sep '19

Hello All, I hane an openstack queens installation with centos and since some days gnocchi api serice reports: connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] Tue Sep 24 08:36:24 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] Tue Sep 24 08:36:25 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:26 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:27 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:28 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Please, must I change some limits ? Where ? I hahe noy found any uwsgi configuration file for gnocchi api service . Regards Ignazio

1 0

[barbican][FFE] Feature Freeze Exception - Secret Consumers
by Moises Guimaraes de Medeiros 24 Sep '19

24 Sep '19

Hi, I'd like to request a FFE for the Secret Consumers API patches: <goog_213778621> https://review.opendev.org/#/q/topic:secret-consumers The changes are limited to a new feature described here: <goog_213778622> https://specs.openstack.org/openstack/barbican-specs/specs/train/secret-con… Thanks, -- Moisés Guimarães Software Engineer Red Hat <https://www.redhat.com> <https://red.ht/sig>

2 1

[mistral][FFE] Request for FFE: skipping validation
by Renat Akhmerov 24 Sep '19

24 Sep '19

Hi, We’d like to land the patch [1] (when it’s done) in Train although it adds an API query string parameter and a new configuration option. It has a big user impact (significant performance) boost. That change is backwards compatible and doesn’t break any other release (or stable branch maintenance) policies. I’ve already discussed it with the release team in IRC and got a preliminary approval from them. And since I’m the PTL of Mistral I approve this FFE as well :) Let me know if you have any concerns/comments. Thanks [1] https://review.opendev.org/#/c/683344/ Renat Akhmerov @Nokia

1 0

cron triggers execution fails with cinder.volume_snapshots_create
by Francois Scheurer 24 Sep '19

24 Sep '19

Dear All We are using Mistral with Openstack Rocky. (with federated users) We could then use cron triggers for instance with nova.servers_create_image or cinder.volume_snapshots_create with success. But we hit an issue with cinder.backups_create . This call will stores the backup on our swift backend (ceph rgw). The workflow works when executed directly but it fails when executed via cron trigger: 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied See details below. Cheers Francois 2019-09-17 10:46:02.436 8 INFO cinder.backup.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Create backup started, backup: 901e1781-02ad-46d5-8ddf-e5410670cf9f volume: c0022411-59a4-4c7c-9474-c7ea8ccc7691. 2019-09-17 10:46:02.746 20 INFO cinder.api.openstack.wsgi [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] GET http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9… 2019-09-17 10:46:02.764 20 INFO cinder.api.openstack.wsgi [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9… returned with HTTP 200 2019-09-17 10:46:03 +0200] "GET /v3/f099965b37ac41489e9cac8c9d208711/os-services HTTP/1.1" 200 2819 18532 "-" "Go-http-client/1.1" 2019-09-17 10:46:03 +0200] "GET /v3/f099965b37ac41489e9cac8c9d208711/snapshots HTTP/1.1" 200 17 23618 "-" "Go-http-client/1.1" 2019-09-17 10:46:03.098 22 INFO cinder.api.openstack.wsgi [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/vol… 2019-09-17 10:46:03.150 22 INFO cinder.volume.api [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Get all volumes completed successfully. 2019-09-17 10:46:03.152 22 INFO cinder.api.openstack.wsgi [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/vol… returned with HTTP 200 2019-09-17 10:46:03.162 18 INFO cinder.api.openstack.wsgi [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-… 2019-09-17 10:46:03.172 18 INFO cinder.api.openstack.wsgi [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-… returned with HTTP 200 2019-09-17 10:46:03.182 19 INFO cinder.api.openstack.wsgi [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/sna… 2019-09-17 10:46:03.197 19 INFO cinder.api.openstack.wsgi [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/sna… returned with HTTP 200 2019-09-17 10:46:03.197 19 INFO cinder.volume.api [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Get all snapshots completed successfully. 2019-09-17 10:46:03.878 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Initialize volume connection completed successfully. 2019-09-17 10:46:04.468 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Terminate volume connection completed successfully. 2019-09-17 10:46:04.501 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Remove volume export completed successfully. 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server container = self._create_container(backup) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server query_string=query_string) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server raise ClientException.from_response(resp, 'Container PUT failed', body) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err)) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.conn.put_container(container) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.put_container(backup.container) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server service_token=self.service_token, **kwargs) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path)) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 226, in _create_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 327, in _prepare_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 535, in backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/drivers/swift.py", line 315, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 414, in create_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 425, in create_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 502, in _run_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1061, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1722, in _retry 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1808, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Exception during message handling: ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last): -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

4 13

[nova] New gate bug 1844929, timed out waiting for response from cell during scheduling
by Matt Riedemann 23 Sep '19

23 Sep '19

I noticed this while looking at a grenade failure on an unrelated patch: https://bugs.launchpad.net/nova/+bug/1844929 The details are in the bug but it looks like this showed up around Sept 17 and hits mostly on FortNebula nodes but also OVH nodes. It's restricted to grenade jobs and while I don't see anything obvious in the rabbitmq logs (the only errors are about uwsgi [api] heartbeat issues), it's possible that these are slower infra nodes and we're just not waiting for something properly during the grenade upgrade. We also don't seem to have the mysql logs published during the grenade jobs which we need to fix (and recently did fix for devstack jobs [1] but grenade jobs are still using devstack-gate so log collection happens there). I didn't see any changes in nova, grenade or devstack since Sept 16 that look like they would be related to this so I'm guessing right now it's just a combination of performance on certain infra nodes (slower?) and something in grenade/nova not restarting properly or not waiting long enough for the upgrade to complete. [1] https://github.com/openstack/devstack/commit/f92c346131db2c89b930b1a23f8489… -- Thanks, Matt

3 5

[cloudkitty][requirements] Requesting a requirement freeze exception for python-cloudkittyclient
by Luka Peschke 23 Sep '19

23 Sep '19

Hello everybody, On behalf of the cloudkitty team, I'm asking for a requirements freeze exception for python-cloudkittyclient. The stable/train branch has been created from the 3.0.0 release, which matched an intermediary release of cloudkitty. Since then, support for some experimental API features has been added. We'd like to have these changes included in the Train release. They have been merged before the feature freeze, but we missed the deadline for the client release. Except for cloudkitty-dashboard, no openstack project depends on python-cloudkittyclient, so this exception shouldn't be impacting. Thanks, Luka Peschke (peschk_l)

2 1

Long, Slow Zuul Queues and Why They Happen
by Clark Boylan 23 Sep '19

23 Sep '19

Hello, We've been fielding a fair bit of questions and suggestions around Zuul's long change (and job) queues over the last week or so. As a result I tried to put a quick FAQ type document [0] on how we schedule jobs, why we schedule that way, and how we can improve the long queues. Hoping that gives us all a better understanding of why were are in the current situation and ideas on how we can help to improve things. [0] https://docs.openstack.org/infra/manual/testing.html#why-are-jobs-for-chang… Thanks, Clark

3 4

[neutron][drivers][ironic] FFE request - Use openstacksdk for ironic notifiers
by Riccardo Pittau 23 Sep '19

23 Sep '19

Hello fellow openstackers, I'd like to open an FFE request to convert the ironic events notifier from the current ironicclient to openstacksdk with the change https://review.opendev.org/682040 This requires also a bump of the versions of mock and openstacksdk in lower-constraints, included in the CR. The change has been tested in a devstack environment and passes the zuul checks. Thank you! Riccardo Pittau rpittau||elfosardo

4 5

[goals][IPv6-Only Deployments and Testing] Week R-3 Update
by Ghanshyam Mann 23 Sep '19

23 Sep '19

Hello Everyone, Below is the latest updated on IPv6 goal. All the projects have the ipv6 job patch proposed now. Next step is to review then as per mentioned guidelines below or help in debugging the failure. I saw tests are failing for the few projects which need debugging and help from the project team. Summary: The projects still need to prepare the IPv6 job: * None The projects waiting for IPv6 job patch to merge: If patch is failing, help me to debug that otherwise review and merge. * Barbican * Blazar * Cyborg * Tricircle * Vitrage * Zaqar * Glance * Monasca * Neutron * Qinling * Sahara * Searchlight * Senlin * Tacker * Ec2-Api * Freezer * Heat * Ironic * Karbor * Kuryr * Magnum * Masakari * Mistral * Murano * Octavia (johnsom is working on this and will take over the base patch) The projects have merged the IPv6 jobs: * Designate * Murano * Trove * Cloudkitty * Congress * Horizon * Keystone * Nova * Placement * Solum * Telemetry * Watcher * Zun * Cinder * Manila * Swift The projects do not need the IPv6 job (CLI, lib, deployment projects etc ): I have marked the tasks for below project as invalid. * Adjutant * Documentation * I18n * Infrastructure * Kolla * Loci * Openstack Charms * Openstack-Chef * Openstack-Helm * Openstackansible * Openstackclient * Openstacksdk * Oslo * Packaging-Rpm * Powervmstackers * Puppet Openstack * Rally * Release Management * Requirements * Storlets * Tripleo * Winstackers Storyboard: ========= - https://storyboard.openstack.org/#!/story/2005477 IPv6 missing support found: ===================== 1. https://review.opendev.org/#/c/673397/ 2. https://review.opendev.org/#/c/673449/ 3. https://review.opendev.org/#/c/677524/ There are few more but need to be tracked. How you can help: ============== - Each project needs to look for and review the ipv6 job patch. - Verify it works fine on ipv6 and no ipv4 used in conf etc - Any other specific scenario needs to be added as part of project IPv6 verification. - Help on debugging and fix the bug in IPv6 job is failing. Everything related to this goal can be found under this topic: Topic: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(stat… How to define and run new IPv6 Job on project side: ======================================= - I prepared a wiki page to describe this section - https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing Review suggestion: ============== - Main goal of these jobs will be whether your service is able to listen on IPv6 and can communicate to any other services either OpenStack or DB or rabbitmq etc on IPv6 or not. So check your proposed job with that point of view. If anything missing, comment on patch. - One example was - I missed to configure novnc address to IPv6- https://review.opendev.org/#/c/672493/ - base script as part of 'devstack-tempest-ipv6' will do basic checks for endpoints on IPv6 and some devstack var setting. But if your project needs more specific verification then it can be added in project side job as post-run playbooks as described in wiki page[1]. [1] https://wiki.openstack.org/wiki/Goal-IPv6-only-deployments-and-testing

1 0

[neutron] bug deputy report (the week of 2019-09-16)
by Takashi Yamamoto 23 Sep '19

23 Sep '19

here's the list of what we got reported in the week. features (nova bp follow-up) https://bugs.launchpad.net/neutron/+bug/1844131 Boot a VM with an unaddressed port High https://bugs.launchpad.net/neutron/+bug/1844168 [L3] TooManyExternalNetworks: More than one external network exists. A proposed fix: https://review.opendev.org/#/c/682418/ Medium https://bugs.launchpad.net/neutron/+bug/1844124 Not possible to change fixed-ips if port is on routed provider network A proposed fix: https://review.opendev.org/#/c/682489/ https://bugs.launchpad.net/neutron/+bug/1844516 [neutron-tempest-plugin] SSH timeout exceptions when executing remote commands A proposed fix: https://review.opendev.org/#/c/682864/ https://bugs.launchpad.net/neutron/+bug/1844688 "radvd" daemon does not work by default in some containers A proposed fix: https://review.opendev.org/#/c/683207/ Low https://bugs.launchpad.net/neutron/+bug/1844171 Configuration parameter missing at "Configure the layer-3 agent" A merged fix: https://review.opendev.org/#/c/682645/ https://bugs.launchpad.net/neutron/+bug/1844607 log error when create neutron port with wrong subnet A proposed fix: https://review.opendev.org/#/c/683273/ Incomplete https://bugs.launchpad.net/neutron/+bug/1844123 Unable to trigger IPv6 Prefix Delegation https://bugs.launchpad.net/neutron/+bug/1844595 openstackNova(Rocky)--chinese incorrect codes for db character error https://bugs.launchpad.net/neutron/+bug/1844915 Duplicate packets with two networks connected by router Invalid https://bugs.launchpad.net/neutron/+bug/1844097 Redundant ipv6 address(SLAAC/DHCPv6 stateless) created for port It was an intended behavior and the rationale was explained to the submitter.

1 0

zun-ui 4.0.0.0rc1 (train)
by no-reply＠openstack.org 23 Sep '19

23 Sep '19

Hello everyone, A new release candidate for zun-ui for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/zun-ui/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/zun-ui/log/?h=stable/train Release notes for zun-ui can be found at: https://docs.openstack.org/releasenotes/zun-ui/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/zun-ui/+bugs and tag it *train-rc-potential* to bring it to the zun-ui release crew's attention.

1 0

kuryr-libnetwork 4.0.0.0rc1 (train)
by no-reply＠openstack.org 23 Sep '19

23 Sep '19

Hello everyone, A new release candidate for kuryr-libnetwork for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/kuryr-libnetwork/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/kuryr-libnetwork/log/?h=stable/train Release notes for kuryr-libnetwork can be found at: https://docs.openstack.org/releasenotes/kuryr-libnetwork/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/kuryr-libnetwork/+bugs and tag it *train-rc-potential* to bring it to the kuryr-libnetwork release crew's attention.

1 0

Outreachy Application Deadline - Call for mentors and projects
by Samuel de Medeiros Queiroz 21 Sep '19

21 Sep '19

Hi everyone! Outreachy <https://www.outreachy.org/> helps people from underrepresented groups get involved in free and open source software by matching interns with established mentors in the upstream communities. OpenStack is a participating organization in the Outreachy Dec 2019 to Mar 2020 round. If you're interested to be a mentor, please register as a mentor in the Outreachy website and publish your project ideas. According to this round's schedule <https://www.outreachy.org/docs/applicant/#outreachy-schedule>, the initial application is due next week: - *Sept. 24, 2019 at 4pm UTC Initial application deadline * - *Nov. 5, 2019 at 4pm UTC Final application deadline* It is important to get projects submitted *as soon as possible* so that applicants can sign up before the *Sept. 24 deadline*. Once signed up, they will have between *Oct. 1, 2019 to Nov. 5, 2019 to contribute to the projects*. If you have any questions about becoming a mentor or want to sponsor an intern, please contact me (samueldmq(a)gmail.com) or Mahati Chamarthy ( mahati.chamarthy(a)gmail.com) Thank you, Samuel de Medeiros Queiroz

3 2

[keystone] Keystone Team Update - Week of 16 September 2019
by Colleen Murphy 21 Sep '19

21 Sep '19

# Keystone Team Update - Week of 16 September 2019 ## News ### RC1 Status Even after discovering some gaps in our policy migrations we managed to quickly cover them and are on track to close all our RC1-targeted bugs[1] on time! [1] https://launchpad.net/keystone/+milestone/train-rc1 ## Office Hours When there are topics to cover, the keystone team holds office hours on Tuesdays at 17:00 UTC. No office hours next week due to lack of topics. Add topics you would like to see covered during office hours to the etherpad: https://etherpad.openstack.org/p/keystone-office-hours-topics ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 32 changes this week. ## Changes that need Attention Search query: https://bit.ly/2tymTje There are 58 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. ## Milestone Outlook https://releases.openstack.org/train/schedule.html We're on target to release RC1 next week. ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter

1 0

[ironic] Tips on testing custom hardware manager?
by Jason Anderson 21 Sep '19

21 Sep '19

Hi all, I am hoping to get some tips on how to test out a custom hardware manager. One of my colleagues is working on a project that involves implementing a custom in-band cleaning step, which we are implementing by creating our own ramdisk image that includes an extra library, which is necessary for the clean step. We already have created the image and ensured it has IPA installed and that all seems to work fine (in that, it executes on the node and we see our code running--and failing!) The issue we are having is that we encounter some issues in our fully integrated environment (such as the provisioning network having different networking rules) and replicating this environment in some local development context is very difficult. Right now our workflow is really onerous as a result: my colleague has to rebuild the ramdisk image, re-upload it to Glance, update the test Ironic node to reference that image, then perform a rebuild. One cycle of this takes a while as you can imagine. I was wondering: is it possible to somehow interrupt or give a larger window for some interactive debugging? The amount of time we have to run some quick tests/debugging is small because the deploy will time out and cancel itself or it will proceed and fail. Thusfar I haven't found any documentation or written experience on this admittedly niche task. Perhaps somebody has already gone down this road and can advise on some tips? It would be much appreciated! Cheers, -- Jason Anderson Chameleon DevOps Lead Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

2 2

[i18n][tc] The future of I18n
by Thierry Carrez 21 Sep '19

21 Sep '19

Hi! The I18n project team had no PTL candidates for Ussuri, so the TC needs to decide what to do with it. It just happens that Ian kindly volunteered to be an election official, and therefore could not technically run for I18n PTL. So if Ian is still up for taking it, we could just go and appoint him. That said, I18n evolved a lot, to the point where it might fit the SIG profile better than the project team profile. As a reminder, project teams are responsible for producing OpenStack-the-software, and since they are all integral in the production of the software that we want to release on a time-based schedule, they come with a number of mandatory tasks (like designating a PTL every 6 months). SIGs (special interest groups) are OpenStack teams that work on a mission that is not directly producing a piece of the OpenStack release. SIG members are bound by their mission, rather than by a specific OpenStack release deliverable. There is no mandatory task, as it is OK if the group goes dormant for a while. The I18n team regroups translators, with an interest of making OpenStack (in general, not just the software) more accessible to non-English speakers. They currently try to translate the OpenStack user survey, the Horizon dashboard messages, and key documentation. It could still continue as a project team (since it still produces Horizon translations), but I'd argue that at this point it is not what defines them. The fact that they are translators is what defines them, which IMHO makes them fit the SIG profile better than the project team profile. They can totally continue proposing translation files for Horizon as a I18n SIG, so there would be no technical difference. Just less mandatory tasks for the team. Thoughts ? -- Thierry Carrez (ttx)

4 4