<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 6 January 2016 at 12:30, Ryan Brown <span dir="ltr"><<a href="mailto:rybrown@redhat.com" target="_blank">rybrown@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 12/23/2015 04:19 AM, Dougal Matthews wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
<br>
On 22 December 2015 at 17:59, Ben Nemec <<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a><br></span><div><div class="h5">
<mailto:<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>>> wrote:<br>
<br>
Can we just do git like I've been suggesting all along? ;-)<br>
<br>
More serious discussion inline. :-)<br>
<br>
On 12/22/2015 09:36 AM, Dougal Matthews wrote:<br>
> Hi all,<br>
><br>
> This topic came up in the 2015-12-15 meeting[1], and again<br>
briefly today.<br>
> After working with the code that came out of the deployment library<br>
> spec[2] I<br>
> had some concerns with how we are storing the templates.<br>
><br>
> Simply put, when we are dealing with 100+ files from<br>
tripleo-heat-templates<br>
> how can we ensure consistency in Swift without any atomicity or<br>
> transactions.<br>
> I think this is best explained with a couple of examples.<br>
><br>
> - When we create a new deployment plan (upload all the templates<br>
to swift)<br>
> how do we handle the case where there is an error? For<br>
example, if we are<br>
> uploading 10 files - what do we do if the 5th one fails for<br>
some reason?<br>
> There is a patch to do a manual rollback[3], but I have<br>
concerns about<br>
> doing this in Python. If Swift is completely inaccessible for<br>
a short<br>
> period the rollback wont work either.<br>
><br>
> - When deploying to Heat, we need to download all the YAML files<br>
from<br>
> Swift.<br>
> This can take a couple of seconds. What happens if somebody<br>
starts to<br>
> upload a new version of the plan in the middle? We could end<br>
up trying to<br>
> deploy half old and half new files. We wouldn't have a<br>
consistent view of<br>
> the database.<br>
><br>
> We had a few suggestions in the meeting:<br>
><br>
> - Add a locking mechanism. I would be concerned about deadlocks or<br>
> having to<br>
> lock for the full duration of a deploy.<br>
<br>
There should be no need to lock the plan for the entire deploy. It's<br>
not like we're re-reading the templates at the end of the deploy today.<br>
It's a one-shot read and then the plan could be unlocked, at least as<br>
far as I know.<br>
<br>
<br>
Good point. That would be holding the lock for longer than we need.<br>
<br>
<br>
The only option where we wouldn't need locking at all is the<br>
read-copy-update model Clint mentions, which might be a valid option as<br>
well. Whatever we do, there are going to be concurrency issues though.<br>
For example, what happens if two users try to make updates to the plan<br>
at the same time? If you don't either merge the changes or disallow one<br>
of them completely then one user's changes might be lost.<br>
<br>
TBH, this is further convincing me that we should just make this git<br>
backed and let git handle the merging and conflict resolution (never<br>
mind the fact that it gets us a well-understood version control system<br>
for "free"). For updates that don't conflict with other changes, git<br>
can merge them automatically, but for merge conflicts you just return a<br>
rebase error to the user and make them resolve it. I have a feeling<br>
this is the behavior we'll converge on eventually anyway, and rather<br>
than reimplement git, let's just use the real thing.<br>
<br>
<br>
I'd be curious to hear more how you would go about doing this with git. I've<br>
never automated git to this level, so I am concerned about what issues we<br>
might hit.<br>
<br>
Do you happen to know of any good sources that I can find out more?<br>
</div></div></blockquote>
<br>
One source of inspiration could be the way Openshift handles things. You get a git URL to push your $whatever to, and they have git hooks that deploy when you push. This would be perfect for merge conflict situations. Imagine two cases.<br></blockquote><div><br></div><div>Interesting. I hadn't even considered that we could act as a git remote. That would<br></div><div>be pretty neat. It would mean that we are very tied to git, which isn't necessarily a<br></div><div>bad thing but it is something to consider.<br><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Normal case:<br>
1. User does stuff in web interface or over API<br>
2. The API adds templates and such to git repo, then deploys as needed<br>
3. User is happy!<br>
<br>
Merge conflict case:<br>
1. User does stuff in web interface or over API<br>
2. The API tries, but there's a merge conflict. It hands back a 409 and a git URL<br>
3. User clones the repo and resolves conflict<br>
4. User pushes repo back up<br>
5. API deploys stuff<br>
6. User is happy!<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
/2 cents<br>
<br>
> - Store the files in a tarball (so we only deal with one file). I think we<br>
> could still hit issues with multiple operations unless we guarantee that<br>
> only one instance of the API is ever running.<br>
><br>
> I think these could potentially be made to work, but at this point are we<br>
> using the best tool for the job?<br>
><br>
> For alternatives, I see a can think of a couple of options:<br>
><br>
> - Use a database, like we did for Tuskar and most OpenStack API's do.<br>
<br>
I kind of like this, in particular because it would allow us to handle<br>
migrations between versions the same way as other OpenStack services.<br>
<br>
<br>
Yeah, I feel like this is the "safe" option. It's a well trodden and<br>
tested path.<br>
<br>
I'm not entirely sure how it maps to our template configuration method<br>
though. Storing a bunch of template blobs in the database feels a<br>
little square peg, round hole to me, and might undo a lot of the<br>
benefits of using the database in the first place.<br>
<br>
<br>
I don't follow this point. In the way we access Swift, everything is<br>
essentially<br>
a blob of text also.<br>
<br>
Now, on the other hand, having a database to store basic data like<br>
metadata on plans and such might be a useful thing regardless of where<br>
we store the templates themselves. We could also use it for locking<br>
just fine IMHO - TripleO isn't a tool targeted to have cloud-scale<br>
number of users. It's a deployer tool with a relatively limited number<br>
of users per installation, so the scaling of a locking mechanism isn't a<br>
problem I necessarily think we need to solve. We have way bigger<br>
scaling issues than that to tackle before I think we would hit the limit<br>
of a database-locking scheme.<br>
<br>
> - Invest time in building something on Swift.<br>
> - Glance was transitioning to be a Artifact store. I don't know the<br>
> status of<br>
> this or if it would meet out needs.<br>
><br>
> Any input, ideas or suggestions would be great!<br>
><br>
> Thanks,<br>
> Dougal<br>
><br>
><br>
> [1]:<br>
><a href="http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89</a><br>
> [2]:<br>
><a href="https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html" rel="noreferrer" target="_blank">https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html</a><br>
> [3]:<a href="https://review.openstack.org/#/c/257481/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/257481/</a><br>
><br>
><br>
><br>
__________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br></div></div>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span class=""><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br></span>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span class=""><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>