September 2019 - openstack-discuss

python-watcher 3.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for python-watcher for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/watcher/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/python-watcher/log/?h=stable/train Release notes for python-watcher can be found at: https://docs.openstack.org/releasenotes/watcher/

1 0

watcher-dashboard 2.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for watcher-dashboard for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/watcher-dashboard/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/watcher-dashboard/log/?h=stable/train Release notes for watcher-dashboard can be found at: https://docs.openstack.org/releasenotes/watcher-dashboard/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/watcher-dashboard/+bugs and tag it *train-rc-potential* to bring it to the watcher-dashboard release crew's attention.

1 0

solum 7.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for solum for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/solum/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/solum/log/?h=stable/train Release notes for solum can be found at: https://docs.openstack.org/releasenotes/solum/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/solum/+bugs and tag it *train-rc-potential* to bring it to the solum release crew's attention.

1 0

solum-dashboard 3.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for solum-dashboard for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/solum-dashboard/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/solum-dashboard/log/?h=stable/train Release notes for solum-dashboard can be found at: https://docs.openstack.org/releasenotes/solum-dashboard/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/solum/+bugs and tag it *train-rc-potential* to bring it to the solum-dashboard release crew's attention.

1 0

trove-dashboard 13.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for trove-dashboard for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/trove-dashboard/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/trove-dashboard/log/?h=stable/train Release notes for trove-dashboard can be found at: https://docs.openstack.org/releasenotes/trove-dashboard/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/trove/+bugs and tag it *train-rc-potential* to bring it to the trove-dashboard release crew's attention.

1 0

trove 12.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for trove for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/trove/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/trove/log/?h=stable/train Release notes for trove can be found at: https://docs.openstack.org/releasenotes/trove/ If you find an issue that could be considered release-critical, please file it at: https://bugs.launchpad.net/trove/+bugs and tag it *train-rc-potential* to bring it to the trove release crew's attention.

1 0

qinling-dashboard 2.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for qinling-dashboard for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/qinling-dashboard/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/qinling-dashboard/log/?h=stable/train Release notes for qinling-dashboard can be found at: https://docs.openstack.org/releasenotes/qinling-dashboard/ If you find an issue that could be considered release-critical, please file it at: https://storyboard.openstack.org/#!/project/1050 and tag it *train-rc-potential* to bring it to the qinling-dashboard release crew's attention.

1 0

qinling 3.0.0.0rc1 (train)
by no-reply＠openstack.org 25 Sep '19

25 Sep '19

Hello everyone, A new release candidate for qinling for the end of the Train cycle is available! You can find the source code tarball at: https://tarballs.openstack.org/qinling/ Unless release-critical issues are found that warrant a release candidate respin, this candidate will be formally released as the final Train release. You are therefore strongly encouraged to test and validate this tarball! Alternatively, you can directly test the stable/train release branch at: https://opendev.org/openstack/qinling/log/?h=stable/train Release notes for qinling can be found at: https://docs.openstack.org/releasenotes/qinling/ If you find an issue that could be considered release-critical, please file it at: https://storyboard.openstack.org/#!/project/927 and tag it *train-rc-potential* to bring it to the qinling release crew's attention.

1 0

[tripleo] deploying tripleo on centos8
by Emilien Macchi 25 Sep '19

25 Sep '19

As you probably know, CentOS8 was published today and the TripleO team will use this platform in the CI very soon. Today I spent a bit of time trying to deploy an undercloud on centos8 and I had some issues, and I wanted to share them in case some other folks do testing as well: https://etherpad.openstack.org/p/tripleo-centos8 Note that this is really early stage testing, and some work will have to happen on the packaging side mainly. Feel free to contribute, -- Emilien Macchi

1 0

[Watcher] no irc meeting today
by li.canwei2＠zte.com.cn 25 Sep '19

25 Sep '19

Hi, Watcher irc meeting will be cancelled because I'm no time today. Thanks, licanwei

1 0

[nova] The test of NUMA aware live migration
by wang.ya 25 Sep '19

25 Sep '19

Hi: The main code of NUMA aware live migration was merged. I’m testing it recently. If only set NUMA property(‘hw:numa_nodes’, ‘hw:numa_cpus’, ‘hw:numa_mem’), it works well. But if add the property “hw:cpu_policy='dedicated'”, it will not correct after serval live migrations. Which means the live migrate can be success, but the vCPU pin are not correct(two instance have serval same vCPU pin on same host). Below is my test steps. env: code: master branch (build on 16 September 2019, include the patches of NUMA aware live migration) three compute node: - s1: 24C, 48G (2 NUMA nodes) - stein-2: 12C, 24G (2 NUMA nodes) - stein-3: 8C, 16G (2 NUMA nodes) flavor1 (2c2g): hw:cpu_policy='dedicated', hw:numa_cpus.0='0', hw:numa_cpus.1='1', hw:numa_mem.0='1024', hw:numa_mem.1='1024', hw:numa_nodes='2' flavor2 (4c4g): hw:cpu_policy='dedicated', hw:numa_cpus.0='0,1,2', hw:numa_cpus.1='3', hw:numa_mem.0='1024', hw:numa_mem.1='3072', hw:numa_nodes='2' image has no property. I create four instances(2*flavor1, 2* flavor2), then begin live migration on by one(one instance live migrate done, next instance begin live migrate) and check the vCPU pin whether is correct. After serval live migrations, the vCPU pin will not correct. (You can find full migration list in attached file). The last live migrate is: +-----+--------------------------------------+-------------+-----------+----------------+--------------+----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+ | Id | UUID | Source Node | Dest Node | Source Compute | Dest Compute | Dest Host | Status | Instance UUID | Old Flavor | New Flavor | Created At | Updated At | Type | +-----+--------------------------------------+-------------+-----------+----------------+--------------+----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+ | 470 | 2a9ba183-4f91-4fbf-93cf-6f0e55cc085a | s1 | stein-3 | s1 | stein-3 | 172.16.130.153 | completed | bf0466f6-4815-4824-8586-899817207564 | 1 | 1 | 2019-09-17T10:28:46.000000 | 2019-09-17T10:29:09.000000 | live-migration | | 469 | c05ea0e8-f040-463e-8957-a59f70ed8bf6 | s1 | stein-3 | s1 | stein-3 | 172.16.130.153 | completed | a3ec7a29-80de-4541-989d-4b9f4377f0bd | 1 | 1 | 2019-09-17T10:28:21.000000 | 2019-09-17T10:28:45.000000 | live-migration | | 468 | cef4c609-157e-4b39-b6cc-f5528d49c75a | s1 | stein-2 | s1 | stein-2 | 172.16.130.152 | completed | 83dab721-3343-436d-bee7-f5ffc0d0d38d | 4 | 4 | 2019-09-17T10:27:57.000000 | 2019-09-17T10:28:21.000000 | live-migration | | 467 | 5471e441-2a50-465a-bb63-3fe1bb2e81b9 | s1 | stein-2 | s1 | stein-2 | 172.16.130.152 | completed | e3c19fbe-7b94-4a65-a803-51daa9934378 | 4 | 4 | 2019-09-17T10:27:32.000000 | 2019-09-17T10:27:57.000000 | live-migration | There are two instances land on stein-3, and the two instances have same vCPU pin: (nova-libvirt)[root@stein-3 /]# virsh list --all Id Name State ---------------------------------------------------- 32 instance-00000025 running 33 instance-00000024 running (nova-libvirt)[root@stein-3 /]# virsh vcpupin 32 VCPU: CPU Affinity ---------------------------------- 0: 2 1: 7 (nova-libvirt)[root@stein-3 /]# virsh vcpupin 33 VCPU: CPU Affinity ---------------------------------- 0: 2 1: 7 I checked the nova compute’s log on stein-3(you can find the log in attached log), then I found ‘host_topology’ isn’t updated when ‘hardware.numa_fit_instance_to_host’ be called in claims. ‘host_topology’ is the property of ‘objects.ComputeNode’ and it’s cached in ‘ResourceTracker’, it will use cached ‘cn’ to build ‘claim’ when ‘check_can_live_migrate_destination’ called. Therefore, I guess the cache was not updated or updated too late or some other reason. I also checked the database, the NUMA topology of the two instances have same vCPU pin: “[0,2], [1,7]”, and the compute node: stein-3 only has vCPU pin: “[2], [7]”. Please correct me if there is something wrong :) Best Regards

4 4

[all][tc] What happened in OpenStack Governance last week
by Jean-Philippe Evrard 25 Sep '19

25 Sep '19

Hello everyone, Here are a few things that happened recently: We introduced "comparison of Official Group Structures" [1]. It should help you navigate better our structures and process, if you want to start a new initiative. We now have a plan to reduce the amount of members of our Technical Committee going forward, down to 9 in Q3 2020 [2]. We've clarified the completion criteria for the PDF goal [3]. We've got fresh PTLs! Tetsuro Nakamura for Placement, Ian Y. Choi for I18n, and Monty Taylor for OpenStackSDK. Congratulations! The Ussuri runtimes are out [4][5]. Ussuri will be the first release to not require python2. Projects are encouraged to move forward! Reminder: Projects are allowed to test extra runtimes, like python2.7, if they want to. Note: Most, if not all, initiatives above were started by TC members. Keep in mind you don't need to be in the TC to propose patches in governance! For example, Tom Barron proposed quite a few patches recently, thank you Tom! :) Regards, Jean-Philippe Evrard (evrardjp) [1]: https://review.opendev.org/668093 [2]: https://review.opendev.org/681266 [3]: https://review.opendev.org/679654 [4]: https://governance.openstack.org/tc/reference/runtimes/ussuri.html [5]: https://review.opendev.org/679798

1 0

[keystone] Ussuri Virtual PTGs
by Colleen Murphy 24 Sep '19

24 Sep '19

Hi team, As we've discussed in IRC and at the virtual midcycle, we'll be holding two virtual planning meetings in place of meeting in-person at the PTG in Shanghai. We'll hold one before the real event in order to prepare for the Forum, and one following it in order to disseminate information and recalibrate. I've drawn up a straw man agenda and proposed meeting dates/times in our brainstorming etherpad: https://etherpad.openstack.org/p/keystone-shanghai-ptg The proposal is to have the pre-PTG on Tuesday, October 29 at 14:00 UTC and the post-PTG on Tuesday, November 12 at 15:00 UTC. (Every time I send a scheduling poll we usually settle on Tuesday mornings anyway so I'm just cutting to the chase.) Please speak up if you want to attend but the proposed dates or times do not work for you, we still have plenty of time to play with the schedule. If no one voices concerns by Tuesday, October 8 we'll consider the dates final. The agenda and topics are also open for feedback and additional topic suggestions are still welcome. We'll plan on using jitsi.org as our conference tool but can fall back to gotomeeting if necessary. Colleen

1 0

[FFE][designate] Pool Manager Removal and refactor
by Graham Hayes 24 Sep '19

24 Sep '19

Hi All, I am granting a FFE for 3 patches: https://review.opendev.org/#/c/678432/ - Refactored service layer https://review.opendev.org/#/c/665476/22 - Removed deprecated pool-manager implementation https://review.opendev.org/#/c/657289/ - Removed deprecated powerdns 3 driver These patches remove a duplicate code paths that have been deprecated for an extended period of time, and reduce a lot of the complexity in our implementation. Please contact me if there is any issues or concern. Thanks, Graham

1 0

[requirements] [ironic] FFE for python-ironicclient 3.1.0
by Dmitry Tantsur 24 Sep '19

24 Sep '19

Hi all, we would like to request an exception to release python-ironicclient 3.1.0 from the stable/train branch. The current 3.0.0 release has several issues, one of them [1] is critical and breaks no-auth mode (used e.g. in bifrost). I'm also proposing [2] to exclude python-ironicclient 3.0.0 from requirement. A release request will be posted once [3] merges. The minor version bump is because we've made the previously existing implicit oslo.config dependency explicit [4]. I don't believe the new release will break anyone who is not broken by 3.0.0 already. Thanks, Dmitry [1] https://storyboard.openstack.org/#!/story/2006600 [2] https://review.opendev.org/#/c/684376/ [3] https://review.opendev.org/#/c/684363/ [4] https://review.opendev.org/#/c/684281/

2 3

[requirements][monasca] Request for FFE for monasca-common
by Witek Bedyk 24 Sep '19

24 Sep '19

Hello Requirements Team, I'm requesting FFE for monasca-common library. The new requested version 2.16.1 includes critical fix for Confluent Kafka client. The library is used only by Monasca project. https://review.opendev.org/683986 Thanks Witek

2 3

[keystone] presence of policy.json breaks Keystone?
by Bernd Bausch 24 Sep '19

24 Sep '19

This is on a stable Stein Devstack. Problem description: ubuntu@devstack:~$ oslopolicy-sample-generator --namespace keystone >/etc/keystone/policy.json ubuntu@devstack:~$ openstack user list Internal Server Error (HTTP 500) Note that I did not modify the policy.json file above. It's mere presence is sufficient to cause the problem. When I remove it and restart Keystone, the problem goes away. The Keystone log contains a huge stacktrace with two methods in oslopolicy/_checks.py playing ping-pong with each other until they give up with RuntimeError: maximum recursion depth exceeded. This only happens with Keystone. Nova and Cinder (which also keep policy in code) are fine. This looks like a bug, but I didn't find it in launchpad. Is there a workaround? I would like to use a modified Keystone policy in a training course. Thanks for any feedback. Bernd.

3 2

[ops] ops meetups team minutes
by Chris Morgan 24 Sep '19

24 Sep '19

The OpenStack Ops Meetups team met briefly today on IRC, minutes are below. The team is working on some sessions for the Forum in Shanghai, together with two mid-cycle meetups in 2020, possibly in London and then South Korea. We'll share more details of those when we can. We continue to meet at 10am EDT on #openstack-operators weekly. Meeting ended Tue Sep 24 14:35:02 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) 10:35 AM Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… 10:35 AM O<•openstack> Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… 10:35 AM Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… Cheers Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[Cloudkitty] Stepping down from CK Core team
by Guillaume Espanel 24 Sep '19

24 Sep '19

Hi all, I am stepping down from the Cloudkitty Core team: I have not been able to contribute for a while, and I'm moving to a new position that doesn't involve Cloudkitty at all. -- Guillaume Espanel Objectif Libre www.objectif-libre.com Infrastructure et Formations Linux

1 0

[gnocchi] uWSGI error
by Ignazio Cassano 24 Sep '19

24 Sep '19

Hello All, I hane an openstack queens installation with centos and since some days gnocchi api serice reports: connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] Tue Sep 24 08:36:24 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] Tue Sep 24 08:36:25 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:26 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:27 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:28 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Please, must I change some limits ? Where ? I hahe noy found any uwsgi configuration file for gnocchi api service . Regards Ignazio

1 0

[barbican][FFE] Feature Freeze Exception - Secret Consumers
by Moises Guimaraes de Medeiros 24 Sep '19

24 Sep '19

Hi, I'd like to request a FFE for the Secret Consumers API patches: <goog_213778621> https://review.opendev.org/#/q/topic:secret-consumers The changes are limited to a new feature described here: <goog_213778622> https://specs.openstack.org/openstack/barbican-specs/specs/train/secret-con… Thanks, -- Moisés Guimarães Software Engineer Red Hat <https://www.redhat.com> <https://red.ht/sig>

2 1

[mistral][FFE] Request for FFE: skipping validation
by Renat Akhmerov 24 Sep '19

24 Sep '19

Hi, We’d like to land the patch [1] (when it’s done) in Train although it adds an API query string parameter and a new configuration option. It has a big user impact (significant performance) boost. That change is backwards compatible and doesn’t break any other release (or stable branch maintenance) policies. I’ve already discussed it with the release team in IRC and got a preliminary approval from them. And since I’m the PTL of Mistral I approve this FFE as well :) Let me know if you have any concerns/comments. Thanks [1] https://review.opendev.org/#/c/683344/ Renat Akhmerov @Nokia

1 0

cron triggers execution fails with cinder.volume_snapshots_create
by Francois Scheurer 24 Sep '19

24 Sep '19

Dear All We are using Mistral with Openstack Rocky. (with federated users) We could then use cron triggers for instance with nova.servers_create_image or cinder.volume_snapshots_create with success. But we hit an issue with cinder.backups_create . This call will stores the backup on our swift backend (ceph rgw). The workflow works when executed directly but it fails when executed via cron trigger: 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied See details below. Cheers Francois 2019-09-17 10:46:02.436 8 INFO cinder.backup.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Create backup started, backup: 901e1781-02ad-46d5-8ddf-e5410670cf9f volume: c0022411-59a4-4c7c-9474-c7ea8ccc7691. 2019-09-17 10:46:02.746 20 INFO cinder.api.openstack.wsgi [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] GET http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9… 2019-09-17 10:46:02.764 20 INFO cinder.api.openstack.wsgi [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9… returned with HTTP 200 2019-09-17 10:46:03 +0200] "GET /v3/f099965b37ac41489e9cac8c9d208711/os-services HTTP/1.1" 200 2819 18532 "-" "Go-http-client/1.1" 2019-09-17 10:46:03 +0200] "GET /v3/f099965b37ac41489e9cac8c9d208711/snapshots HTTP/1.1" 200 17 23618 "-" "Go-http-client/1.1" 2019-09-17 10:46:03.098 22 INFO cinder.api.openstack.wsgi [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/vol… 2019-09-17 10:46:03.150 22 INFO cinder.volume.api [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Get all volumes completed successfully. 2019-09-17 10:46:03.152 22 INFO cinder.api.openstack.wsgi [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/vol… returned with HTTP 200 2019-09-17 10:46:03.162 18 INFO cinder.api.openstack.wsgi [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-… 2019-09-17 10:46:03.172 18 INFO cinder.api.openstack.wsgi [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-… returned with HTTP 200 2019-09-17 10:46:03.182 19 INFO cinder.api.openstack.wsgi [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/sna… 2019-09-17 10:46:03.197 19 INFO cinder.api.openstack.wsgi [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/sna… returned with HTTP 200 2019-09-17 10:46:03.197 19 INFO cinder.volume.api [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Get all snapshots completed successfully. 2019-09-17 10:46:03.878 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Initialize volume connection completed successfully. 2019-09-17 10:46:04.468 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Terminate volume connection completed successfully. 2019-09-17 10:46:04.501 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Remove volume export completed successfully. 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server container = self._create_container(backup) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server query_string=query_string) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server raise ClientException.from_response(resp, 'Container PUT failed', body) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err)) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.conn.put_container(container) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.put_container(backup.container) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server service_token=self.service_token, **kwargs) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path)) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 226, in _create_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 327, in _prepare_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 535, in backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/drivers/swift.py", line 315, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 414, in create_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 425, in create_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 502, in _run_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1061, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1722, in _retry 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1808, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Exception during message handling: ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last): -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

4 13

[nova] New gate bug 1844929, timed out waiting for response from cell during scheduling
by Matt Riedemann 23 Sep '19

23 Sep '19

I noticed this while looking at a grenade failure on an unrelated patch: https://bugs.launchpad.net/nova/+bug/1844929 The details are in the bug but it looks like this showed up around Sept 17 and hits mostly on FortNebula nodes but also OVH nodes. It's restricted to grenade jobs and while I don't see anything obvious in the rabbitmq logs (the only errors are about uwsgi [api] heartbeat issues), it's possible that these are slower infra nodes and we're just not waiting for something properly during the grenade upgrade. We also don't seem to have the mysql logs published during the grenade jobs which we need to fix (and recently did fix for devstack jobs [1] but grenade jobs are still using devstack-gate so log collection happens there). I didn't see any changes in nova, grenade or devstack since Sept 16 that look like they would be related to this so I'm guessing right now it's just a combination of performance on certain infra nodes (slower?) and something in grenade/nova not restarting properly or not waiting long enough for the upgrade to complete. [1] https://github.com/openstack/devstack/commit/f92c346131db2c89b930b1a23f8489… -- Thanks, Matt

3 5

[cloudkitty][requirements] Requesting a requirement freeze exception for python-cloudkittyclient
by Luka Peschke 23 Sep '19

23 Sep '19

Hello everybody, On behalf of the cloudkitty team, I'm asking for a requirements freeze exception for python-cloudkittyclient. The stable/train branch has been created from the 3.0.0 release, which matched an intermediary release of cloudkitty. Since then, support for some experimental API features has been added. We'd like to have these changes included in the Train release. They have been merged before the feature freeze, but we missed the deadline for the client release. Except for cloudkitty-dashboard, no openstack project depends on python-cloudkittyclient, so this exception shouldn't be impacting. Thanks, Luka Peschke (peschk_l)

2 1