<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Herve</p>
<p><br>
</p>
<p>Thank you for your reply.<br>
</p>
<p>I am using the same input & params as when executing the
workflow directly from horizon (successfully):<br>
</p>
<p> {<br>
"incremental": "false", <br>
"force": "true", <br>
"name": "fsc-create-vol-backup", <br>
"volume_id": "c0022411-59a4-4c7c-9474-c7ea8ccc7691"<br>
}<br>
<br>
{<br>
"namespace": "", <br>
"env": {}, <br>
"task_name": "create_vol_backup_task"<br>
}<br>
</p>
<p>Maybe I need some additional params when executing via cron?</p>
<p>I will try specfying the objectstore container explicitly.</p>
<p><br>
</p>
<p>Best Regards</p>
<p>Francois<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 9/19/19 1:18 PM, Herve Beraud wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAFDq9gV5770HTwaEv-wFeCxwiuzy=C1eQpkpZHG8Co-gc73yag@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hello François,</div>
<div><br>
</div>
<div>Given your error, are you sure your cron task load the
right config with the right authorized user or something
related?<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Le jeu. 19 sept. 2019 à 11:51,
Francois Scheurer <<a
href="mailto:francois.scheurer@everyware.ch"
moz-do-not-send="true">francois.scheurer@everyware.ch</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear
All<br>
<br>
<br>
We are using Mistral with Openstack Rocky. (with federated
users)<br>
We could then use cron triggers for instance with <br>
nova.servers_create_image or cinder.volume_snapshots_create
with success.<br>
<br>
<br>
But we hit an issue with cinder.backups_create .<br>
<br>
This call will stores the backup on our swift backend (ceph
rgw).<br>
The workflow works when executed directly but it fails when
executed via <br>
cron trigger:<br>
<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
ClientException: Container PUT failed: <br>
<a
href="http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups</a>
<br>
401 Unauthorized AccessDenied<br>
<br>
See details below.<br>
<br>
<br>
<br>
<br>
<br>
Cheers<br>
<br>
Francois<br>
<br>
<br>
<br>
2019-09-17 10:46:02.436 8 INFO cinder.backup.manager <br>
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa <br>
3e9b1a4fe95048a3b98fb5abebd44f6c
aeac4b07d8b144178c43c65f29fa9dac - <br>
18b20663b571455c8da31fde994d031a
18b20663b571455c8da31fde994d031a] <br>
Create backup started, backup:
901e1781-02ad-46d5-8ddf-e5410670cf9f <br>
volume: c0022411-59a4-4c7c-9474-c7ea8ccc7691.<br>
2019-09-17 10:46:02.746 20 INFO cinder.api.openstack.wsgi <br>
[req-69a86fd7-b478-4e26-9692-a8416c41459a <br>
3e9b1a4fe95048a3b98fb5abebd44f6c
aeac4b07d8b144178c43c65f29fa9dac - <br>
18b20663b571455c8da31fde994d031a
18b20663b571455c8da31fde994d031a] GET <br>
<a
href="http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f2019-09-17"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f<br>
2019-09-17</a> 10:46:02.764 20 INFO
cinder.api.openstack.wsgi <br>
[req-69a86fd7-b478-4e26-9692-a8416c41459a <br>
3e9b1a4fe95048a3b98fb5abebd44f6c
aeac4b07d8b144178c43c65f29fa9dac - <br>
18b20663b571455c8da31fde994d031a
18b20663b571455c8da31fde994d031a] <br>
<a
href="http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.i.ewcs.ch:8776/v2/aeac4b07d8b144178c43c65f29fa9dac/backups/901e1781-02ad-46d5-8ddf-e5410670cf9f</a>
<br>
returned with HTTP 200<br>
2019-09-17 10:46:03 +0200] "GET <br>
/v3/f099965b37ac41489e9cac8c9d208711/os-services HTTP/1.1" 200
2819 <br>
18532 "-" "Go-http-client/1.1"<br>
2019-09-17 10:46:03 +0200] "GET <br>
/v3/f099965b37ac41489e9cac8c9d208711/snapshots HTTP/1.1" 200
17 23618 <br>
"-" "Go-http-client/1.1"<br>
2019-09-17 10:46:03.098 22 INFO cinder.api.openstack.wsgi <br>
[req-ec93b942-2dc9-4505-8656-680bd661fc71 <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] GET <br>
<a
href="http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes2019-09-17"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes<br>
2019-09-17</a> 10:46:03.150 22 INFO cinder.volume.api <br>
[req-ec93b942-2dc9-4505-8656-680bd661fc71 <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] Get all volumes completed successfully.<br>
2019-09-17 10:46:03.152 22 INFO cinder.api.openstack.wsgi <br>
[req-ec93b942-2dc9-4505-8656-680bd661fc71 <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] <br>
<a
href="http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/volumes</a>
<br>
returned with HTTP 200<br>
2019-09-17 10:46:03.162 18 INFO cinder.api.openstack.wsgi <br>
[req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] GET <br>
<a
href="http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services2019-09-17"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services<br>
2019-09-17</a> 10:46:03.172 18 INFO
cinder.api.openstack.wsgi <br>
[req-3e1ce449-305e-4e1f-9b51-aa56da6e2076 <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] <br>
<a
href="http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/os-services</a>
<br>
returned with HTTP 200<br>
2019-09-17 10:46:03.182 19 INFO cinder.api.openstack.wsgi <br>
[req-b726191c-3710-477a-b7a0-961b74f9233f <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] GET <br>
<a
href="http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots2019-09-17"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots<br>
2019-09-17</a> 10:46:03.197 19 INFO
cinder.api.openstack.wsgi <br>
[req-b726191c-3710-477a-b7a0-961b74f9233f <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] <br>
<a
href="http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://cinder.service.stage.ewcs.ch/v3/f099965b37ac41489e9cac8c9d208711/snapshots</a>
<br>
returned with HTTP 200<br>
2019-09-17 10:46:03.197 19 INFO cinder.volume.api <br>
[req-b726191c-3710-477a-b7a0-961b74f9233f <br>
b141574ee71f49a0b53a05ae968576c5
f099965b37ac41489e9cac8c9d208711 - <br>
default default] Get all snapshots completed successfully.<br>
2019-09-17 10:46:03.878 30 INFO cinder.volume.manager <br>
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa <br>
3e9b1a4fe95048a3b98fb5abebd44f6c
aeac4b07d8b144178c43c65f29fa9dac - <br>
18b20663b571455c8da31fde994d031a
18b20663b571455c8da31fde994d031a] <br>
Initialize volume connection completed successfully.<br>
2019-09-17 10:46:04.468 30 INFO cinder.volume.manager <br>
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa <br>
3e9b1a4fe95048a3b98fb5abebd44f6c
aeac4b07d8b144178c43c65f29fa9dac - <br>
18b20663b571455c8da31fde994d031a
18b20663b571455c8da31fde994d031a] <br>
Terminate volume connection completed successfully.<br>
2019-09-17 10:46:04.501 30 INFO cinder.volume.manager <br>
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa <br>
3e9b1a4fe95048a3b98fb5abebd44f6c
aeac4b07d8b144178c43c65f29fa9dac - <br>
18b20663b571455c8da31fde994d031a
18b20663b571455c8da31fde994d031a] <br>
Remove volume export completed successfully.<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
container = <br>
self._create_container(backup)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
query_string=query_string)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
raise <br>
ClientException.from_response(resp, 'Container PUT failed',
body)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
res = <br>
self.dispatcher.dispatch(message)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
result = <br>
f(*args, **kwargs)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
result = <br>
func(ctxt, **new_args)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
return <br>
self._do_dispatch(endpoint, method, ctxt, args)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
self._update_backup_error(backup, six.text_type(err))<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
self.conn.put_container(container)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
self.force_reraise()<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
self.put_container(backup.container)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
service_token=self.service_token, **kwargs)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
six.reraise(self.type_, self.value, self.tb)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
tpool.Proxy(device_path))<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
updates = <br>
self._run_backup(context, backup, volume)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
volume_size_bytes) = self._prepare_backup(backup)<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
<br>
line 226, in _create_container<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
<br>
line 327, in _prepare_backup<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py",
<br>
line 535, in backup<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/drivers/swift.py",
<br>
line 315, in put_container<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
<br>
line 414, in create_backup<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
<br>
line 425, in create_backup<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py",
<br>
line 502, in _run_backup<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
<br>
line 194, in _do_dispatch<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
<br>
line 265, in dispatch<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
<br>
line 163, in _process_incoming<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
<br>
line 196, in force_reraise<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
<br>
line 220, in __exit__<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py",
<br>
line 159, in wrapper<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
<br>
line 1061, in put_container<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
<br>
line 1722, in _retry<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
File <br>
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/swiftclient/client.py",
<br>
line 1808, in put_container<br>
<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
[req-3b5104f4-4aca-489f-86e0-78c5523d6faa <br>
3e9b1a4fe95048a3b98fb5abebd44f6c
aeac4b07d8b144178c43c65f29fa9dac - <br>
18b20663b571455c8da31fde994d031a
18b20663b571455c8da31fde994d031a] <br>
Exception during message handling: ClientException: Container
PUT <br>
failed: <br>
<a
href="http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups</a>
<br>
401 Unauthorized AccessDenied<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server <br>
ClientException: Container PUT failed: <br>
<a
href="http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://rgw.service.stage.i.ewcs.ch/swift/v1/AUTH_aeac4b07d8b144178c43c65f29fa9dac/volumebackups</a>
<br>
401 Unauthorized AccessDenied<br>
2019-09-17 10:46:04.525 8 ERROR oslo_messaging.rpc.server
Traceback <br>
(most recent call last):<br>
<br>
<br>
-- <br>
<br>
<br>
EveryWare AG<br>
François Scheurer<br>
Senior Systems Engineer<br>
Zurlindenstrasse 52a<br>
CH-8003 Zürich<br>
<br>
tel: +41 44 466 60 00<br>
fax: +41 44 466 60 10<br>
mail: <a href="mailto:francois.scheurer@everyware.ch"
target="_blank" moz-do-not-send="true">francois.scheurer@everyware.ch</a><br>
web: <a href="http://www.everyware.ch" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://www.everyware.ch</a><br>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Hervé Beraud</div>
<div>Senior Software Engineer<br>
</div>
<div>Red Hat - Openstack Oslo</div>
<div>irc: hberaud</div>
<div>-----BEGIN PGP SIGNATURE-----<br>
<br>
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>
Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>
RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>
F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>
5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>
glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>
m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>
hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>
qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>
F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>
B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>
v6rDpkeNksZ9fFSyoY2o<br>
=ECSj<br>
-----END PGP SIGNATURE-----<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: <a class="moz-txt-link-abbreviated" href="mailto:francois.scheurer@everyware.ch">francois.scheurer@everyware.ch</a>
web: <a class="moz-txt-link-freetext" href="http://www.everyware.ch">http://www.everyware.ch</a> </pre>
</body>
</html>