cron triggers execution fails with cinder.volume_snapshots_create

Francois Scheurer francois.scheurer at everyware.ch
Thu Sep 19 12:28:40 UTC 2019 

Hi Herve


Thank you for your reply.

I am using the same input & params as when executing the workflow 
directly from horizon (successfully):

     {
         "incremental": "false",
         "force": "true",
         "name": "fsc-create-vol-backup",
         "volume_id": "c0022411-59a4-4c7c-9474-c7ea8ccc7691"
     }

     {
         "namespace": "",
         "env": {},
         "task_name": "create_vol_backup_task"
     }

Maybe I need some additional params when executing via cron?

I will try specfying the objectstore container explicitly.


Best Regards

Francois




On 9/19/19 1:18 PM, Herve Beraud wrote:
> Hello François,
>
> Given your error, are you sure your cron task load the right config 
> with the right authorized user or something related?
>
> Le jeu. 19 sept. 2019 à 11:51, Francois Scheurer 
> <francois.scheurer at everyware.ch 
> <mailto:francois.scheurer at everyware.ch>> a écrit :
>
>     Dear All
>
>
>     We are using Mistral with  Openstack Rocky. (with federated users)
>     We could then use cron triggers for instance with
>     nova.servers_create_image or cinder.volume_snapshots_create with
>     success.
>
>
>     But we hit an issue with cinder.backups_create .
>
>     This call will stores the backup on our swift backend (ceph rgw).
>     The workflow works when executed directly but it fails when
>     executed via
>     cron trigger:
>
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     ClientException: Container PUT failed:
>     http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups
>
>     401 Unauthorized   AccessDenied
>
>     See details below.
>
>
>
>
>
>     Cheers
>
>     Francois
>
>
>
>     2019-09-17 10:46:02.436 8 INFO cinder.backup.manager
>     [req-3b5104f4-4aca-489f-86e0-78c5523d6faa
>     3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac -
>     18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a]
>     Create backup started, backup: 901e1781-02ad-46d5-8ddf-e5410670cf9f
>     volume: c0022411-59a4-4c7c-9474-c7ea8ccc7691.
>     2019-09-17 10:46:02.746 20 INFO cinder.api.openstack.wsgi
>     [req-69a86fd7-b478-4e26-9692-a8416c41459a
>     3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac -
>     18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a]
>     GET
>     http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f
>     2019-09-17
>     <http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f2019-09-17>
>     10:46:02.764 20 INFO cinder.api.openstack.wsgi
>     [req-69a86fd7-b478-4e26-9692-a8416c41459a
>     3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac -
>     18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a]
>     http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f
>
>     returned with HTTP 200
>     2019-09-17 10:46:03 +0200] "GET
>     /v3/f099965b37ac41489e9cac8c9d208711/os-services HTTP/1.1" 200 2819
>     18532 "-" "Go-http-client/1.1"
>     2019-09-17 10:46:03 +0200] "GET
>     /v3/f099965b37ac41489e9cac8c9d208711/snapshots HTTP/1.1" 200 17 23618
>     "-" "Go-http-client/1.1"
>     2019-09-17 10:46:03.098 22 INFO cinder.api.openstack.wsgi
>     [req-ec93b942-2dc9-4505-8656-680bd661fc71
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default] GET
>     http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes
>     2019-09-17
>     <http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes2019-09-17>
>     10:46:03.150 22 INFO cinder.volume.api
>     [req-ec93b942-2dc9-4505-8656-680bd661fc71
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default] Get all volumes completed successfully.
>     2019-09-17 10:46:03.152 22 INFO cinder.api.openstack.wsgi
>     [req-ec93b942-2dc9-4505-8656-680bd661fc71
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default]
>     http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes
>
>     returned with HTTP 200
>     2019-09-17 10:46:03.162 18 INFO cinder.api.openstack.wsgi
>     [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default] GET
>     http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services
>     2019-09-17
>     <http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services2019-09-17>
>     10:46:03.172 18 INFO cinder.api.openstack.wsgi
>     [req-3e1ce449-305e-4e1f-9b51-aa56da6e2076
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default]
>     http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services
>
>     returned with HTTP 200
>     2019-09-17 10:46:03.182 19 INFO cinder.api.openstack.wsgi
>     [req-b726191c-3710-477a-b7a0-961b74f9233f
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default] GET
>     http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots
>     2019-09-17
>     <http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots2019-09-17>
>     10:46:03.197 19 INFO cinder.api.openstack.wsgi
>     [req-b726191c-3710-477a-b7a0-961b74f9233f
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default]
>     http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots
>
>     returned with HTTP 200
>     2019-09-17 10:46:03.197 19 INFO cinder.volume.api
>     [req-b726191c-3710-477a-b7a0-961b74f9233f
>     b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 -
>     default default] Get all snapshots completed successfully.
>     2019-09-17 10:46:03.878 30 INFO cinder.volume.manager
>     [req-3b5104f4-4aca-489f-86e0-78c5523d6faa
>     3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac -
>     18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a]
>     Initialize volume connection completed successfully.
>     2019-09-17 10:46:04.468 30 INFO cinder.volume.manager
>     [req-3b5104f4-4aca-489f-86e0-78c5523d6faa
>     3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac -
>     18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a]
>     Terminate volume connection completed successfully.
>     2019-09-17 10:46:04.501 30 INFO cinder.volume.manager
>     [req-3b5104f4-4aca-489f-86e0-78c5523d6faa
>     3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac -
>     18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a]
>     Remove volume export completed successfully.
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server container =
>     self._create_container(backup)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     query_string=query_string)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server raise
>     ClientException.from_response(resp, 'Container PUT failed', body)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server res =
>     self.dispatcher.dispatch(message)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result =
>     f(*args, **kwargs)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server result =
>     func(ctxt, **new_args)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server return
>     self._do_dispatch(endpoint, method, ctxt, args)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     self._update_backup_error(backup, six.text_type(err))
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     self.conn.put_container(container)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     self.force_reraise()
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     self.put_container(backup.container)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     service_token=self.service_token, **kwargs)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     six.reraise(self.type_, self.value, self.tb)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     tpool.Proxy(device_path))
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server updates =
>     self._run_backup(context, backup, volume)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     volume_size_bytes) = self._prepare_backup(backup)
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
>
>     line 226, in _create_container
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
>
>     line 327, in _prepare_backup
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
>
>     line 535, in backup
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/drivers/swift.py",
>
>     line 315, in put_container
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
>
>     line 414, in create_backup
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
>
>     line 425, in create_backup
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
>
>     line 502, in _run_backup
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
>
>     line 194, in _do_dispatch
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
>
>     line 265, in dispatch
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
>
>     line 163, in _process_incoming
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
>
>     line 196, in force_reraise
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
>
>     line 220, in __exit__
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py",
>
>     line 159, in wrapper
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
>
>     line 1061, in put_container
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
>
>     line 1722, in _retry
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server File
>     "/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
>
>     line 1808, in put_container
>
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     [req-3b5104f4-4aca-489f-86e0-78c5523d6faa
>     3e9b1a4fe95048a3b98fb5abebd44f6c aeac4b07d8b144178c43c65f29fa9dac -
>     18b20663b571455c8da31fde994d031a 18b20663b571455c8da31fde994d031a]
>     Exception during message handling: ClientException: Container PUT
>     failed:
>     http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups
>
>     401 Unauthorized   AccessDenied
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
>     ClientException: Container PUT failed:
>     http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups
>
>     401 Unauthorized   AccessDenied
>     2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server Traceback
>     (most recent call last):
>
>
>     -- 
>
>
>     EveryWare AG
>     François Scheurer
>     Senior Systems Engineer
>     Zurlindenstrasse 52a
>     CH-8003 Zürich
>
>     tel: +41 44 466 60 00
>     fax: +41 44 466 60 10
>     mail: francois.scheurer at everyware.ch
>     <mailto:francois.scheurer at everyware.ch>
>     web: http://www.everyware.ch
>
>
>
> -- 
> Hervé Beraud
> Senior Software Engineer
> Red Hat - Openstack Oslo
> irc: hberaud
> -----BEGIN PGP SIGNATURE-----
>
> wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+
> Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+
> RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP
> F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G
> 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g
> glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw
> m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ
> hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0
> qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y
> F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3
> B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O
> v6rDpkeNksZ9fFSyoY2o
> =ECSj
> -----END PGP SIGNATURE-----
>
-- 


EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer at everyware.ch
web: http://www.everyware.ch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190919/f488fed0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5230 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190919/f488fed0/attachment-0001.bin>

More information about the openstack-discuss mailing list