<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 22 December 2015 at 17:59, Ben Nemec <span dir="ltr"><<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can we just do git like I've been suggesting all along? ;-)<br>
<br>
More serious discussion inline. :-)<br>
<div><div class="h5"><br>
On 12/22/2015 09:36 AM, Dougal Matthews wrote:<br>
> Hi all,<br>
><br>
> This topic came up in the 2015-12-15 meeting[1], and again briefly today.<br>
> After working with the code that came out of the deployment library<br>
> spec[2] I<br>
> had some concerns with how we are storing the templates.<br>
><br>
> Simply put, when we are dealing with 100+ files from tripleo-heat-templates<br>
> how can we ensure consistency in Swift without any atomicity or<br>
> transactions.<br>
> I think this is best explained with a couple of examples.<br>
><br>
> - When we create a new deployment plan (upload all the templates to swift)<br>
> how do we handle the case where there is an error? For example, if we are<br>
> uploading 10 files - what do we do if the 5th one fails for some reason?<br>
> There is a patch to do a manual rollback[3], but I have concerns about<br>
> doing this in Python. If Swift is completely inaccessible for a short<br>
> period the rollback wont work either.<br>
><br>
> - When deploying to Heat, we need to download all the YAML files from<br>
> Swift.<br>
> This can take a couple of seconds. What happens if somebody starts to<br>
> upload a new version of the plan in the middle? We could end up trying to<br>
> deploy half old and half new files. We wouldn't have a consistent view of<br>
> the database.<br>
><br>
> We had a few suggestions in the meeting:<br>
><br>
> - Add a locking mechanism. I would be concerned about deadlocks or<br>
> having to<br>
> lock for the full duration of a deploy.<br>
<br>
</div></div>There should be no need to lock the plan for the entire deploy. It's<br>
not like we're re-reading the templates at the end of the deploy today.<br>
It's a one-shot read and then the plan could be unlocked, at least as<br>
far as I know.<br></blockquote><div><br></div><div>Good point. That would be holding the lock for longer than we need.<br> <br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The only option where we wouldn't need locking at all is the<br>
read-copy-update model Clint mentions, which might be a valid option as<br>
well. Whatever we do, there are going to be concurrency issues though.<br>
For example, what happens if two users try to make updates to the plan<br>
at the same time? If you don't either merge the changes or disallow one<br>
of them completely then one user's changes might be lost.<br>
<br>
TBH, this is further convincing me that we should just make this git<br>
backed and let git handle the merging and conflict resolution (never<br>
mind the fact that it gets us a well-understood version control system<br>
for "free"). For updates that don't conflict with other changes, git<br>
can merge them automatically, but for merge conflicts you just return a<br>
rebase error to the user and make them resolve it. I have a feeling<br>
this is the behavior we'll converge on eventually anyway, and rather<br>
than reimplement git, let's just use the real thing.<br></blockquote><div><br></div><div>I'd be curious to hear more how you would go about doing this with git. I've<br></div><div>never automated git to this level, so I am concerned about what issues we<br></div><div>might hit.<br><br></div><div>Do you happen to know of any good sources that I can find out more?<br> <br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
/2 cents<br>
<span class=""><br>
> - Store the files in a tarball (so we only deal with one file). I think we<br>
> could still hit issues with multiple operations unless we guarantee that<br>
> only one instance of the API is ever running.<br>
><br>
> I think these could potentially be made to work, but at this point are we<br>
> using the best tool for the job?<br>
><br>
> For alternatives, I see a can think of a couple of options:<br>
><br>
> - Use a database, like we did for Tuskar and most OpenStack API's do.<br>
<br>
</span>I kind of like this, in particular because it would allow us to handle<br>
migrations between versions the same way as other OpenStack services.<br></blockquote><div><br></div><div>Yeah, I feel like this is the "safe" option. It's a well trodden and tested path. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm not entirely sure how it maps to our template configuration method<br>
though. Storing a bunch of template blobs in the database feels a<br>
little square peg, round hole to me, and might undo a lot of the<br>
benefits of using the database in the first place.<br></blockquote><div><br></div><div>I don't follow this point. In the way we access Swift, everything is essentially<br></div><div>a blob of text also.<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Now, on the other hand, having a database to store basic data like<br>
metadata on plans and such might be a useful thing regardless of where<br>
we store the templates themselves. We could also use it for locking<br>
just fine IMHO - TripleO isn't a tool targeted to have cloud-scale<br>
number of users. It's a deployer tool with a relatively limited number<br>
of users per installation, so the scaling of a locking mechanism isn't a<br>
problem I necessarily think we need to solve. We have way bigger<br>
scaling issues than that to tackle before I think we would hit the limit<br>
of a database-locking scheme.<br>
<span class="im HOEnZb"><br>
> - Invest time in building something on Swift.<br>
> - Glance was transitioning to be a Artifact store. I don't know the<br>
> status of<br>
> this or if it would meet out needs.<br>
><br>
> Any input, ideas or suggestions would be great!<br>
><br>
> Thanks,<br>
> Dougal<br>
><br>
><br>
> [1]:<br>
> <a href="http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89</a><br>
> [2]:<br>
> <a href="https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html" rel="noreferrer" target="_blank">https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html</a><br>
> [3]: <a href="https://review.openstack.org/#/c/257481/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/257481/</a><br>
><br>
><br>
</span><div class="HOEnZb"><div class="h5">> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>