<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection">
<div dir="auto">Hi!
<div dir="auto"><br /></div>
<div dir="auto">I would kindly ask you to add [mistral] into the subject of the emails related to Mistral. I just saw this thread accidentally (since I can’t read everything) and missed it in the first place.</div>
<div dir="auto"><br /></div>
<div dir="auto">On the issue itself… So yes, the discovery you made makes perfect sense. I agree that a workflow should <span style="caret-color: rgb(39, 39, 40);">probably</span><span style="caret-color: rgb(39, 39, 40);"> </span>be responsible for tracking a status of an operation. We’ve discussed a more generic solution in the past for similar situations but it seems to be virtually impossible to find it. If you have some ideas, please share. We can discuss it.</div>
<div dir="auto"><br /></div>
</div>
</div>
<div name="messageSignatureSection"><br />
<div class="matchFont">Thanks<br />
<br />
Renat Akhmerov<br />
@Nokia</div>
</div>
<div name="messageReplySection">On 23 Sep 2019, 14:41 +0700, Gorka Eguileor <geguileo@redhat.com>, wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #1abc9c;">On 20/09, Francois Scheurer wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">Hi Gorka<br />
<br />
<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #3498db;">Then I assume you prefer the Swift backup driver over the Ceph one<br />
because you are using one of the OpenStack releases that had trouble >with<br /></blockquote>
Incremental Backups on the Ceph backup driver.<br />
<br />
<br />
You are probably right. But I cannot answer that because I was not involve<br />
in that decision.<br />
<br />
<br />
Ok in the radosgw logs I see this:<br />
<br />
<br />
2019-09-20 15:40:06.805529 7f19edb9b700 20 token_id=gAAAAABdhNauRvNev5P90ovX7_cb5_4MkY1tg5JHFpAH8JL-_0vDs06lHW5F9Iphua7fxCWTxxdL-0fRzhR8We_nN6Hx9z3FTWcTXLUMtIUPe0WMKQgW6JkUTP8RwSjAfF4W04OztEg3VAUGN_5gWRlBX-KT9uypnEszadG1yA7gpjkCokNnD8oaIeE6arvs_EjfJib51rao<br />
2019-09-20 15:40:06.805664 7f19edb9b700 20 sending request to<br />
https://keystone.service.stage.ewcs.ch/v3/auth/tokens<br />
2019-09-20 15:40:06.805803 7f19edb9b700 20 ssl verification is set to off<br />
2019-09-20 15:40:07.235356 7f19edb9b700 20 sending request to<br />
https://keystone.service.stage.ewcs.ch/v3/auth/tokens<br />
2019-09-20 15:40:07.235404 7f19edb9b700 20 ssl verification is set to off<br />
2019-09-20 15:40:07.267091 7f19edb9b700 5 Failed keystone auth from<br />
https://keystone.service.stage.ewcs.ch/v3/auth/tokens with 404<br />
BTW: our radosgw is configured to delegate user authentication to keystone.<br />
<br />
In keystone logs I see this:<br />
<br />
2019-09-20 15:40:07.218 24 INFO keystone.token.provider<br />
[req-21b2f11c-9e67-4487-af05-420acfb65ace - - - - -] Token being processed:<br />
token.user_id [f7c7296949f84a4387c5172808a0965b],<br />
token.expires_at[2019-09-21T13:40:07.000000Z],<br />
token.audit_ids[[u'hFweMPCrSO2D00rNcRNECw']], token.methods[[u'password']],<br />
token.system[None], token.domain_id[None],<br />
token.project_id[4120792f50bc4cf2b4f97c4546462f06], token.trust_id[None],<br />
token.federated_groups[None], token.identity_provider_id[None],<br />
token.protocol_id[None],<br />
token.access_token_id[None],token.application_credential_id[None].<br />
2019-09-20 15:40:07.257 21 INFO keystone.common.wsgi<br />
[req-9f858abb-68f9-42cf-b71a-f1cafca91844 f7c7296949f84a4387c5172808a0965b<br />
4120792f50bc4cf2b4f97c4546462f06 - default default] GET<br />
http://keystone.service.stage.ewcs.ch/v3/auth/tokens<br />
2019-09-20 15:40:07.265 21 WARNING keystone.common.wsgi<br />
[req-9f858abb-68f9-42cf-b71a-f1cafca91844 f7c7296949f84a4387c5172808a0965b<br />
4120792f50bc4cf2b4f97c4546462f06 - default default] Could not find trust:<br />
934ed82d2b14413899023da0bee6a953.: TrustNotFound: Could not find trust:<br />
934ed82d2b14413899023da0bee6a953.<br />
<br />
<br />
So what happens is following:<br />
<br />
1. when the user creates the cron trigger, mistral creates a trust<br />
2. when the cron trigger executes the workflow, openstack create a<br />
volume snapshot (a rbd image) then copy it to swift (rgw) then<br />
delete the snapshot<br />
3. when the execution finishes, if the cron trigger has no remaining<br />
executions scheduled, then mistral remove the cron trigger and the trust<br />
<br />
The problem is a racing issue: apprently the copying of the snapshot to<br />
swift run in the background and mistral removes the trust before the<br />
operation completes...<br />
<br />
That explains the error in keystone and also the cron trigger execution<br />
result which is "success" even if the resulting backup is actually "failed".<br />
<br />
<br />
To test this theory I set up the same cron trigger with more than one<br />
scheduled execution and the backups were suddenly created correctly ;-).<br />
<br />
<br />
So something need to be done on the code to deal with this racing issue.<br />
<br />
In the meantime, I will try to put a sleep action after the 'create backup'<br />
action.<br />
<br /></blockquote>
<br />
Hi,<br />
<br />
Congrats on figuring out the issue. :-)<br />
<br />
Instead of a sleep, which may get you through this issue but fall into a<br />
different one and won't return the right status code, you should<br />
probably have a loop checking the status of the backup and return a non<br />
zero status code if it ends up in "error" state.<br />
<br />
Cheers,<br />
Gorka.<br />
<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;"><br />
Best Regards<br />
<br />
Francois<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
On 9/20/19 4:02 PM, Gorka Eguileor wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #3498db;">On 20/09, Francois Scheurer wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #d35400;">Hi Gorka<br />
<br />
<br />
We have a swift endpoint set up on opentstack, which points to our ceph<br />
radosgw backend<br />
<br />
Radosgw provides s3 & swift.<br />
<br />
So the swift logs are here actually the radosgw logs.<br />
<br /></blockquote>
Hi,<br />
<br />
OK, thanks for the clarification.<br />
<br />
Then I assume you prefer the Swift backup driver over the Ceph one<br />
because you are using one of the OpenStack releases that had trouble<br />
with Incremental Backups on the Ceph backup driver.<br />
<br />
Cheers,<br />
Gorka.<br />
<br />
<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #d35400;">Cheers<br />
<br />
Francois<br />
<br />
<br />
<br />
On 9/20/19 2:46 PM, Gorka Eguileor wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #34495e;">On 20/09, Francois Scheurer wrote:<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #2ecc71;">Dear Gorka and Hervé<br />
<br />
<br />
Thanks for your hints.<br />
<br />
I have set the debug log level on radosgw.<br />
<br />
I will retest now and post here the results.<br />
<br />
<br />
Cheers<br />
<br />
Francois<br /></blockquote>
Hi,<br />
<br />
Sorry, I may have missed something in the conversation, weren't you<br />
using Swift?<br />
<br />
I think you need to see the Swift logs as well, since that's the API<br />
service that complained about the authorization.<br />
<br />
Cheers,<br />
Gorka.<br />
<br />
<blockquote type="cite" class="spark_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #2ecc71;"><br />
<br />
--<br />
<br />
<br />
EveryWare AG<br />
François Scheurer<br />
Senior Systems Engineer<br />
Zurlindenstrasse 52a<br />
CH-8003 Zürich<br />
<br />
tel: +41 44 466 60 00<br />
fax: +41 44 466 60 10<br />
mail: francois.scheurer@everyware.ch<br />
web: http://www.everyware.ch<br /></blockquote>
</blockquote>
--<br />
<br />
<br />
EveryWare AG<br />
François Scheurer<br />
Senior Systems Engineer<br />
Zurlindenstrasse 52a<br />
CH-8003 Zürich<br />
<br />
tel: +41 44 466 60 00<br />
fax: +41 44 466 60 10<br />
mail: francois.scheurer@everyware.ch<br />
web: http://www.everyware.ch<br /></blockquote>
<br /></blockquote>
--<br />
<br />
<br />
EveryWare AG<br />
François Scheurer<br />
Senior Systems Engineer<br />
Zurlindenstrasse 52a<br />
CH-8003 Zürich<br />
<br />
tel: +41 44 466 60 00<br />
fax: +41 44 466 60 10<br />
mail: francois.scheurer@everyware.ch<br />
web: http://www.everyware.ch<br />
<br /></blockquote>
<br />
<br />
<br /></blockquote>
</div>
</body>
</html>