cron triggers execution fails with cinder.volume_snapshots_create

Francois Scheurer francois.scheurer at everyware.ch
Thu Sep 19 09:43:38 UTC 2019


Dear All


We are using Mistral with  Openstack Rocky. (with federated users)
We could then use cron triggers for instance with 
nova.servers_create_image or cinder.volume_snapshots_create with success.


But we hit an issue with cinder.backups_create .

This call will stores the backup on our swift backend (ceph rgw).
The workflow works when executed directly but it fails when executed via 
cron trigger:

2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
ClientException: Container PUT failed: 
http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups 
401 Unauthorized   AccessDenied

See details below.





Cheers

Francois



2019-09-17 10:46:02.436 8 INFO cinder.backup.manager 
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa 
3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 
18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] 
Create backup started, backup: 901e1781-02ad-46d5-8ddf-e5410670cf9f 
volume: c0022411-59a4-4c7c-9474-c7ea8ccc7691.
2019-09-17 10:46:02.746 20 INFO cinder.api.openstack.wsgi 
[req-69a86fd7-b478-4e26-9692-a8416c41459a 
3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 
18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] GET 
http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f
2019-09-17 10:46:02.764 20 INFO cinder.api.openstack.wsgi 
[req-69a86fd7-b478-4e26-9692-a8416c41459a 
3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 
18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] 
http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f 
returned with HTTP 200
2019-09-17 10:46:03 +0200] "GET 
/v3/f099965b37ac41489e9cac8c9d208711/os-services HTTP/1.1" 200 2819 
18532 "-" "Go-http-client/1.1"
2019-09-17 10:46:03 +0200] "GET 
/v3/f099965b37ac41489e9cac8c9d208711/snapshots HTTP/1.1" 200 17 23618 
"-" "Go-http-client/1.1"
2019-09-17 10:46:03.098 22 INFO cinder.api.openstack.wsgi 
[req-ec93b942-2dc9-4505-8656-680bd661fc71 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] GET 
http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes
2019-09-17 10:46:03.150 22 INFO cinder.volume.api 
[req-ec93b942-2dc9-4505-8656-680bd661fc71 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] Get all volumes completed successfully.
2019-09-17 10:46:03.152 22 INFO cinder.api.openstack.wsgi 
[req-ec93b942-2dc9-4505-8656-680bd661fc71 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] 
http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes 
returned with HTTP 200
2019-09-17 10:46:03.162 18 INFO cinder.api.openstack.wsgi 
[req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] GET 
http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services
2019-09-17 10:46:03.172 18 INFO cinder.api.openstack.wsgi 
[req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] 
http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services 
returned with HTTP 200
2019-09-17 10:46:03.182 19 INFO cinder.api.openstack.wsgi 
[req-b726191c-3710-477a-b7a0-961b74f9233f 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] GET 
http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots
2019-09-17 10:46:03.197 19 INFO cinder.api.openstack.wsgi 
[req-b726191c-3710-477a-b7a0-961b74f9233f 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] 
http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots 
returned with HTTP 200
2019-09-17 10:46:03.197 19 INFO cinder.volume.api 
[req-b726191c-3710-477a-b7a0-961b74f9233f 
b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - 
default default] Get all snapshots completed successfully.
2019-09-17 10:46:03.878 30 INFO cinder.volume.manager 
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa 
3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 
18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] 
Initialize volume connection completed successfully.
2019-09-17 10:46:04.468 30 INFO cinder.volume.manager 
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa 
3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 
18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] 
Terminate volume connection completed successfully.
2019-09-17 10:46:04.501 30 INFO cinder.volume.manager 
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa 
3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 
18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] 
Remove volume export completed successfully.
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server container = 
self._create_container(backup)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
query_string=query_string)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server raise 
ClientException.from_response(resp, 'Container PUT failed', body)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server     res = 
self.dispatcher.dispatch(message)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = 
f(*args, **kwargs)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = 
func(ctxt, **new_args)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server return 
self._do_dispatch(endpoint, method, ctxt, args)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
self._update_backup_error(backup, six.text_type(err))
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
self.conn.put_container(container)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
self.force_reraise()
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
self.put_container(backup.container)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
service_token=self.service_token, **kwargs)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
six.reraise(self.type_, self.value, self.tb)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
tpool.Proxy(device_path))
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server updates = 
self._run_backup(context, backup, volume)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
volume_size_bytes) = self._prepare_backup(backup)
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", 
line 226, in _create_container
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", 
line 327, in _prepare_backup
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", 
line 535, in backup
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/drivers/swift.py", 
line 315, in put_container
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", 
line 414, in create_backup
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", 
line 425, in create_backup
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", 
line 502, in _run_backup
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", 
line 194, in _do_dispatch
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", 
line 265, in dispatch
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", 
line 163, in _process_incoming
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", 
line 159, in wrapper
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", 
line 1061, in put_container
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", 
line 1722, in _retry
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py", 
line 1808, in put_container

2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa 
3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac - 
18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a] 
Exception during message handling: ClientException: Container PUT 
failed: 
http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups 
401 Unauthorized   AccessDenied
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server 
ClientException: Container PUT failed: 
http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups 
401 Unauthorized   AccessDenied
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server Traceback 
(most recent call last):


-- 


EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer at everyware.ch
web: http://www.everyware.ch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5230 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190919/47c53197/attachment.bin>


More information about the openstack-discuss mailing list