September 2019 - openstack-discuss

[requirements][monasca] Request for FFE for monasca-common
by Witek Bedyk 24 Sep '19

24 Sep '19

Hello Requirements Team, I'm requesting FFE for monasca-common library. The new requested version 2.16.1 includes critical fix for Confluent Kafka client. The library is used only by Monasca project. https://review.opendev.org/683986 Thanks Witek

2 3

[keystone] presence of policy.json breaks Keystone?
by Bernd Bausch 24 Sep '19

24 Sep '19

This is on a stable Stein Devstack. Problem description: ubuntu@devstack:~$ oslopolicy-sample-generator --namespace keystone >/etc/keystone/policy.json ubuntu@devstack:~$ openstack user list Internal Server Error (HTTP 500) Note that I did not modify the policy.json file above. It's mere presence is sufficient to cause the problem. When I remove it and restart Keystone, the problem goes away. The Keystone log contains a huge stacktrace with two methods in oslopolicy/_checks.py playing ping-pong with each other until they give up with RuntimeError: maximum recursion depth exceeded. This only happens with Keystone. Nova and Cinder (which also keep policy in code) are fine. This looks like a bug, but I didn't find it in launchpad. Is there a workaround? I would like to use a modified Keystone policy in a training course. Thanks for any feedback. Bernd.

3 2

[ops] ops meetups team minutes
by Chris Morgan 24 Sep '19

24 Sep '19

The OpenStack Ops Meetups team met briefly today on IRC, minutes are below. The team is working on some sessions for the Forum in Shanghai, together with two mid-cycle meetups in 2020, possibly in London and then South Korea. We'll share more details of those when we can. We continue to meet at 10am EDT on #openstack-operators weekly. Meeting ended Tue Sep 24 14:35:02 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) 10:35 AM Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… 10:35 AM O<•openstack> Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… 10:35 AM Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2019/ops_meetup_tea… Cheers Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[Cloudkitty] Stepping down from CK Core team
by Guillaume Espanel 24 Sep '19

24 Sep '19

Hi all, I am stepping down from the Cloudkitty Core team: I have not been able to contribute for a while, and I'm moving to a new position that doesn't involve Cloudkitty at all. -- Guillaume Espanel Objectif Libre www.objectif-libre.com Infrastructure et Formations Linux

1 0

[gnocchi] uWSGI error
by Ignazio Cassano 24 Sep '19

24 Sep '19

Hello All, I hane an openstack queens installation with centos and since some days gnocchi api serice reports: connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] Tue Sep 24 08:36:24 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] connect_to_tcp()/socket(): Too many open files [core/socket.c line 462] Tue Sep 24 08:36:25 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:26 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:27 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Tue Sep 24 08:36:28 2019 - *** uWSGI listen queue of socket "127.0.0.1:40000" (fd: 10) full !!! (101/100) *** Please, must I change some limits ? Where ? I hahe noy found any uwsgi configuration file for gnocchi api service . Regards Ignazio

1 0

[barbican][FFE] Feature Freeze Exception - Secret Consumers
by Moises Guimaraes de Medeiros 24 Sep '19

24 Sep '19

Hi, I'd like to request a FFE for the Secret Consumers API patches: <goog_213778621> https://review.opendev.org/#/q/topic:secret-consumers The changes are limited to a new feature described here: <goog_213778622> https://specs.openstack.org/openstack/barbican-specs/specs/train/secret-con… Thanks, -- Moisés Guimarães Software Engineer Red Hat <https://www.redhat.com> <https://red.ht/sig>

2 1

[mistral][FFE] Request for FFE: skipping validation
by Renat Akhmerov 24 Sep '19

24 Sep '19

Hi, We’d like to land the patch [1] (when it’s done) in Train although it adds an API query string parameter and a new configuration option. It has a big user impact (significant performance) boost. That change is backwards compatible and doesn’t break any other release (or stable branch maintenance) policies. I’ve already discussed it with the release team in IRC and got a preliminary approval from them. And since I’m the PTL of Mistral I approve this FFE as well :) Let me know if you have any concerns/comments. Thanks [1] https://review.opendev.org/#/c/683344/ Renat Akhmerov @Nokia

1 0

cron triggers execution fails with cinder.volume_snapshots_create
by Francois Scheurer 24 Sep '19

24 Sep '19

Dear All We are using Mistral with Openstack Rocky. (with federated users) We could then use cron triggers for instance with nova.servers_create_image or cinder.volume_snapshots_create with success. But we hit an issue with cinder.backups_create . This call will stores the backup on our swift backend (ceph rgw). The workflow works when executed directly but it fails when executed via cron trigger: 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied See details below. Cheers Francois 2019-09-17 10:46:02.436 8 INFO cinder.backup.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Create backup started, backup: 901e1781-02ad-46d5-8ddf-e5410670cf9f volume: c0022411-59a4-4c7c-9474-c7ea8ccc7691. 2019-09-17 10:46:02.746 20 INFO cinder.api.openstack.wsgi [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] GET http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9… 2019-09-17 10:46:02.764 20 INFO cinder.api.openstack.wsgi [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9… returned with HTTP 200 2019-09-17 10:46:03 +0200] "GET /v3/f099965b37ac41489e9cac8c9d208711/os-services HTTP/1.1" 200 2819 18532 "-" "Go-http-client/1.1" 2019-09-17 10:46:03 +0200] "GET /v3/f099965b37ac41489e9cac8c9d208711/snapshots HTTP/1.1" 200 17 23618 "-" "Go-http-client/1.1" 2019-09-17 10:46:03.098 22 INFO cinder.api.openstack.wsgi [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/vol… 2019-09-17 10:46:03.150 22 INFO cinder.volume.api [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Get all volumes completed successfully. 2019-09-17 10:46:03.152 22 INFO cinder.api.openstack.wsgi [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/vol… returned with HTTP 200 2019-09-17 10:46:03.162 18 INFO cinder.api.openstack.wsgi [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-… 2019-09-17 10:46:03.172 18 INFO cinder.api.openstack.wsgi [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-… returned with HTTP 200 2019-09-17 10:46:03.182 19 INFO cinder.api.openstack.wsgi [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/sna… 2019-09-17 10:46:03.197 19 INFO cinder.api.openstack.wsgi [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/sna… returned with HTTP 200 2019-09-17 10:46:03.197 19 INFO cinder.volume.api [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Get all snapshots completed successfully. 2019-09-17 10:46:03.878 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Initialize volume connection completed successfully. 2019-09-17 10:46:04.468 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Terminate volume connection completed successfully. 2019-09-17 10:46:04.501 30 INFO cinder.volume.manager [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Remove volume export completed successfully. 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server container = self._create_container(backup) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server query_string=query_string) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server raise ClientException.from_response(resp, 'Container PUT failed', body) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err)) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.conn.put_container(container) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server self.put_container(backup.container) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server service_token=self.service_token, **kwargs) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path)) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup) 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 226, in _create_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 327, in _prepare_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 535, in backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/drivers/swift.py", line 315, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 414, in create_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 425, in create_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 502, in _run_backup 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1061, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1722, in _retry 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", line 1808, in put_container 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] Exception during message handling: ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29… 401 Unauthorized AccessDenied 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last): -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

4 13

[nova] New gate bug 1844929, timed out waiting for response from cell during scheduling
by Matt Riedemann 23 Sep '19

23 Sep '19

I noticed this while looking at a grenade failure on an unrelated patch: https://bugs.launchpad.net/nova/+bug/1844929 The details are in the bug but it looks like this showed up around Sept 17 and hits mostly on FortNebula nodes but also OVH nodes. It's restricted to grenade jobs and while I don't see anything obvious in the rabbitmq logs (the only errors are about uwsgi [api] heartbeat issues), it's possible that these are slower infra nodes and we're just not waiting for something properly during the grenade upgrade. We also don't seem to have the mysql logs published during the grenade jobs which we need to fix (and recently did fix for devstack jobs [1] but grenade jobs are still using devstack-gate so log collection happens there). I didn't see any changes in nova, grenade or devstack since Sept 16 that look like they would be related to this so I'm guessing right now it's just a combination of performance on certain infra nodes (slower?) and something in grenade/nova not restarting properly or not waiting long enough for the upgrade to complete. [1] https://github.com/openstack/devstack/commit/f92c346131db2c89b930b1a23f8489… -- Thanks, Matt

3 5

[cloudkitty][requirements] Requesting a requirement freeze exception for python-cloudkittyclient
by Luka Peschke 23 Sep '19

23 Sep '19

Hello everybody, On behalf of the cloudkitty team, I'm asking for a requirements freeze exception for python-cloudkittyclient. The stable/train branch has been created from the 3.0.0 release, which matched an intermediary release of cloudkitty. Since then, support for some experimental API features has been added. We'd like to have these changes included in the Train release. They have been merged before the feature freeze, but we missed the deadline for the client release. Except for cloudkitty-dashboard, no openstack project depends on python-cloudkittyclient, so this exception shouldn't be impacting. Thanks, Luka Peschke (peschk_l)

2 1