Open Stack

On 19/09, Francois Scheurer wrote:
> Dear All
>
>
> We are using Mistral with  Openstack Rocky. (with federated users)
> We could then use cron triggers for instance with nova.servers_create_image
> or cinder.volume_snapshots_create with success.
>
>
> But we hit an issue with cinder.backups_create .
>
> This call will stores the backup on our swift backend (ceph rgw).
> The workflow works when executed directly but it fails when executed via
> cron trigger:
>
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException:
> Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups
> 401 Unauthorized   AccessDenied
>
> See details below.
>

Hi,

This makes no sense, the Swift connection credentials don't depend on
the OpenStack user calling the service, they are internal to the Backup
service.

If after this error you can still create a backup manually, then the
backup service works fine and the swiftclient as well (since we rely on
it not failing the create call on for an container).

I would start by checking on the Swift logs to see why this request was
rejected and the manual one isn't.

Cheers,
Gorka.

>
>
>
>
> Cheers
>
> Francois
>
>
>
> 2019-09-17 10:46:02.436 8 INFO cinder.backup.manager
> [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c
> aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a
> 18b20663b571455c8da31fde994d031a] Create backup started, backup:
> 901e1781-02ad-46d5-8ddf-e5410670cf9f volume:
> c0022411-59a4-4c7c-9474-c7ea8ccc7691.
> 2019-09-17 10:46:02.746 20 INFO cinder.api.openstack.wsgi
> [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c
> aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a
> 18b20663b571455c8da31fde994d031a] GET http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f
> 2019-09-17 10:46:02.764 20 INFO cinder.api.openstack.wsgi
> [req-69a86fd7-b478-4e26-9692-a8416c41459a 3e9b1a4fe95048a3b98fb5abebd44f6c
> aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a
> 18b20663b571455c8da31fde994d031a] http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f
> returned with HTTP 200
> 2019-09-17 10:46:03 +0200] "GET
> /v3/f099965b37ac41489e9cac8c9d208711/os-services HTTP/1.1" 200 2819 18532
> "-" "Go-http-client/1.1"
> 2019-09-17 10:46:03 +0200] "GET
> /v3/f099965b37ac41489e9cac8c9d208711/snapshots HTTP/1.1" 200 17 23618 "-"
> "Go-http-client/1.1"
> 2019-09-17 10:46:03.098 22 INFO cinder.api.openstack.wsgi
> [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes
> 2019-09-17 10:46:03.150 22 INFO cinder.volume.api
> [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] Get all volumes
> completed successfully.
> 2019-09-17 10:46:03.152 22 INFO cinder.api.openstack.wsgi
> [req-ec93b942-2dc9-4505-8656-680bd661fc71 b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes
> returned with HTTP 200
> 2019-09-17 10:46:03.162 18 INFO cinder.api.openstack.wsgi
> [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services
> 2019-09-17 10:46:03.172 18 INFO cinder.api.openstack.wsgi
> [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services
> returned with HTTP 200
> 2019-09-17 10:46:03.182 19 INFO cinder.api.openstack.wsgi
> [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] GET http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots
> 2019-09-17 10:46:03.197 19 INFO cinder.api.openstack.wsgi
> [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots
> returned with HTTP 200
> 2019-09-17 10:46:03.197 19 INFO cinder.volume.api
> [req-b726191c-3710-477a-b7a0-961b74f9233f b141574ee71f49a0b53a05ae968576c5
> f099965b37ac41489e9cac8c9d208711 - default default] Get all snapshots
> completed successfully.
> 2019-09-17 10:46:03.878 30 INFO cinder.volume.manager
> [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c
> aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a
> 18b20663b571455c8da31fde994d031a] Initialize volume connection completed
> successfully.
> 2019-09-17 10:46:04.468 30 INFO cinder.volume.manager
> [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c
> aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a
> 18b20663b571455c8da31fde994d031a] Terminate volume connection completed
> successfully.
> 2019-09-17 10:46:04.501 30 INFO cinder.volume.manager
> [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c
> aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a
> 18b20663b571455c8da31fde994d031a] Remove volume export completed
> successfully.
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server container =
> self._create_container(backup)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> query_string=query_string)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server raise
> ClientException.from_response(resp, 'Container PUT failed', body)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server     res =
> self.dispatcher.dispatch(message)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result = f(*args,
> **kwargs)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result =
> func(ctxt, **new_args)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> self._update_backup_error(backup, six.text_type(err))
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> self.conn.put_container(container)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> self.put_container(backup.container)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> service_token=self.service_token, **kwargs)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> six.reraise(self.type_, self.value, self.tb)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> tpool.Proxy(device_path))
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server updates =
> self._run_backup(context, backup, volume)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server volume_size_bytes)
> = self._prepare_backup(backup)
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
> line 226, in _create_container
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
> line 327, in _prepare_backup
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
> line 535, in backup
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/drivers/swift.py",
> line 315, in put_container
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
> line 414, in create_backup
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
> line 425, in create_backup
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
> line 502, in _run_backup
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 194, in _do_dispatch
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 265, in dispatch
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
> line 163, in _process_incoming
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
> line 220, in __exit__
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py",
> line 159, in wrapper
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
> line 1061, in put_container
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
> line 1722, in _retry
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
> line 1808, in put_container
>
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
> [req-3b5104f4-4aca-489f-86e0-78c5523d6faa 3e9b1a4fe95048a3b98fb5abebd44f6c
> aeac4b07d8b144178c43c65f29fa9dac - 18b20663b571455c8da31fde994d031a
> 18b20663b571455c8da31fde994d031a] Exception during message handling:
> ClientException: Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups
> 401 Unauthorized   AccessDenied
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server ClientException:
> Container PUT failed: http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups
> 401 Unauthorized   AccessDenied
> 2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server Traceback (most
> recent call last):
>
>
> --
>
>
> EveryWare AG
> François Scheurer
> Senior Systems Engineer
> Zurlindenstrasse 52a
> CH-8003 Zürich
>
> tel: +41 44 466 60 00
> fax: +41 44 466 60 10
> mail: francois.scheurer at everyware.ch
> web: http://www.everyware.ch