<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 23 December 2015 at 12:01, Steven Hardy <span dir="ltr"><<a href="mailto:shardy@redhat.com" target="_blank">shardy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Dec 22, 2015 at 03:36:02PM +0000, Dougal Matthews wrote:<br>
> Hi all,<br>
><br>
> This topic came up in the 2015-12-15 meeting[1], and again briefly today.<br>
> After working with the code that came out of the deployment library<br>
> spec[2] I<br>
> had some concerns with how we are storing the templates.<br>
><br>
> Simply put, when we are dealing with 100+ files from<br>
> tripleo-heat-templates<br>
> how can we ensure consistency in Swift without any atomicity or<br>
> transactions.<br>
> I think this is best explained with a couple of examples.<br>
><br>
</span>> Â - When we create a new deployment plan (upload all the templates to<br>
> swift)<br>
> Â Â how do we handle the case where there is an error? For example, if we<br>
> are<br>
> Â Â uploading 10 files - what do we do if the 5th one fails for some<br>
> reason?<br>
> Â Â There is a patch to do a manual rollback[3], but I have concerns<br>
> about<br>
> Â Â doing this in Python. If Swift is completely inaccessible for a short<br>
> Â Â period the rollback wont work either.<br>
<br>
How does using a different data store help fix this error path?<br>
<br>
Regardless of the Swift/DB/Git choice, you have to detect something failed,<br>
and mark the plan as in a failed state.<br></blockquote><div><br></div><div>If your connection to a database such as postrges drops mid-transaction it<br></div><div>will be rolled back. So for a networking issue it would help. Granted, I am not<br></div><div>sure exactly what happens if somebody pulls the plug on the database, but<br></div><div>then you probably have bigger issues. Failed network requests seem relatively<br></div><div>common.<br><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> Â - When deploying to Heat, we need to download all the YAML files from<br>
> Swift.<br>
> Â Â This can take a couple of seconds. What happens if somebody starts to<br>
> Â Â upload a new version of the plan in the middle? We could end up<br>
> trying to<br>
> Â Â deploy half old and half new files. We wouldn't have a consistent<br>
> view of<br>
> Â Â the database.<br>
<br>
If this happens, the API design is wrong - we should have a plan reference<br>
a unique identifier, including a version (e.g through swift versioned<br>
objects, git tags/branches or whatever), then on deployment heat should<br>
download exactly one version of those artefacts (e.g via a swift tempurl or<br>
a git URL, or whatever).<br></blockquote><div><br></div><div>Sure, this sounds like a valid point against the design of both the storage spec<br></div><div>that was merged and the API spec that is up for review.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
FYI heatclient already supports downloading template objects directly from<br>
swift, and heat/heatclient already support downloading from http URLs, so<br>
all we have to do AFAICS is generate a URL pointing to a consistent<br>
version of the templates.<br></blockquote><div><br></div><div>I wasn't aware of this feature, I'll take a look but that sounds useful.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class=""><br>
> We had a few suggestions in the meeting:<br>
><br>
</span>> Â - Add a locking mechanism. I would be concerned about deadlocks or<br>
> having to<br>
> Â Â lock for the full duration of a deploy.<br>
<br>
I think we need to create a new version of a plan every time it's modified,<br>
then on deployment the concern around locking is much reduced, or even<br>
eliminated?<br></blockquote><div><br></div><div>Yup, if we essentially make everything immutable that would solve a number<br></div><div>of issues.<br></div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> Â - Store the files in a tarball (so we only deal with one file). I think<br>
> we<br>
> Â Â could still hit issues with multiple operations unless we guarantee<br>
> that<br>
> Â Â only one instance of the API is ever running.<br>
<span class="">><br>
> I think these could potentially be made to work, but at this point are we<br>
> using the best tool for the job?<br>
><br>
> For alternatives, I see a can think of a couple of options:<br>
><br>
> - Use a database, like we did for Tuskar and most OpenStack API's do.<br>
<br>
</span>Storing template files in a database with Tuskar was a mistake - we should<br>
*not* do that again IMO - it was a very opaque interface and caused a lot<br>
of confusion in use.<br></blockquote><div><br></div><div>The interface is via the API, so I don't see how the storage mechanism would<br></div><div>change that. We could change the current WIP code to use a database or<br></div><div>git or anything else without changing the interface. Perhaps we need to focus<br></div><div>more on what interface we need and the specific problems with the one Tuskar<br></div><div>provided? I thought the primary issue with Tuskar was with the template<br>mangling and not the actual process of accessing the templates etc.<br></div><div><br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Storing code (in this case yaml files) in some sort of versioned repository<br>
is *such* a solved problem - it'll be total wheel-reinventing if we<br>
implement template revision control inside a bespoke DB IMHO.<br></blockquote><div><br></div><div>It's interesting that you mention revision control - this IIRC hasn't been a topic<br></div><div>mentioned in any of the specs. It might be that we need to step back and look<br></div><div>at this topic more in general.<br></div><div> <br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I think swift is probably a workable solution (using versioning and<br>
tempurl's), but we should really consider making the template store<br>
pluggable, as having the API backed by an (operator visible) git repo is<br>
likely to be a nicer and more familiar solution.<br></blockquote><div><br></div><div>The git option is referenced now and then, but I'm not aware of the details in<br></div><div>how this would work. Is this a pattern used by other projects? I'd like to do <br></div><div>some reading as I'm not sure how I would use git for this.<br><br></div><div>The code has always been somewhat pluggable, that was my feedback on<br></div><div>the early storage patches - I had a feeling this topic might come up. However,<br></div><div>at the moment there is only a Swift implementation.<br><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The question around validation data I'm less clear on - we already store<br>
discovery data in swift, which seems to work OK - how is the validation<br>
data different, in that it warrants a bespoke DB data store?<br>
<br>
Steve<br>
<div class="HOEnZb"><div class="h5"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>