Open Stack

On 22 December 2015 at 16:56, Clint Byrum <clint at fewbar.com> wrote:

> Excerpts from Dougal Matthews's message of 2015-12-22 07:36:02 -0800:
> > Hi all,
> >
> > This topic came up in the 2015-12-15 meeting[1], and again briefly today.
> > After working with the code that came out of the deployment library
> spec[2]
> > I
> > had some concerns with how we are storing the templates.
> >
> > Simply put, when we are dealing with 100+ files from
> tripleo-heat-templates
> > how can we ensure consistency in Swift without any atomicity or
> > transactions.
> > I think this is best explained with a couple of examples.
> >
> >  - When we create a new deployment plan (upload all the templates to
> swift)
> >    how do we handle the case where there is an error? For example, if we
> are
> >    uploading 10 files - what do we do if the 5th one fails for some
> reason?
> >    There is a patch to do a manual rollback[3], but I have concerns about
> >    doing this in Python. If Swift is completely inaccessible for a short
> >    period the rollback wont work either.
> >
>
> You could create a unique swift container, upload things to that, and
> then update a pointer in a well-known location to point at that container
> for the new plan only after you've verified it is available. This is a
> primitive form of Read-copy-update.
>
> >  - When deploying to Heat, we need to download all the YAML files from
> > Swift.
> >    This can take a couple of seconds. What happens if somebody starts to
> >    upload a new version of the plan in the middle? We could end up
> trying to
> >    deploy half old and half new files. We wouldn't have a consistent
> view of
> >    the database.
> >
>
> Perhaps you should land a change in Heat to allow templates to be directly
> downloaded by the engine from swift without needing to be uploaded. In
> the past allowing URLs to be downloaded unfettered was disabled because
> we don't want Heat to be a DoS engine, but swift would be in-cloud and
> could be restricted to the stack owner.
>
> > We had a few suggestions in the meeting:
> >
> >  - Add a locking mechanism. I would be concerned about deadlocks or
> having
> > to
> >    lock for the full duration of a deploy.
>
> Deadlocks only happen when competing interests lock _two_ things in
> different order. Have one thing to lock, and you don't have to worry
> about this.
>

Good point. Sorry, I am getting my terminology confused. I can't think what
the
word is, but I was worried about stale locks getting lost and locking
everything up.
Where would we store the lock? On the machine running the API maybe?


>  - Store the files in a tarball (so we only deal with one file). I think
> we
> >    could still hit issues with multiple operations unless we guarantee
> that
> >    only one instance of the API is ever running.
> >
> > I think these could potentially be made to work, but at this point are we
> > using the best tool for the job?
> >
> > For alternatives, I see a can think of a couple of options:
> >
> > - Use a database, like we did for Tuskar and most OpenStack API's do.
>
> It's worth noting that many OpenStack API's/daemons are using the
> database + MQ to provide consistency in a distributed fashion, and many
> have identified that this doesn't scale particularly well, and looking
> at tooz to help bring in DLM's. In fact, a spec recently landed around
> this:
>
>
> http://specs.openstack.org/openstack/openstack-specs/specs/chronicles-of-a-dlm.html
>
> So if you are only using the DB for consistency, you might want to just
> use tooz+swift.
>

Interesting! I hadn't come across this and I'll look into it. Thanks.



>
> > - Invest time in building something on Swift.
> > - Glance was transitioning to be a Artifact store. I don't know the
> status
> > of
> >   this or if it would meet out needs.
>
> I'd say the artifact store is more about exposing options to users, and
> less about providing primitives for an API.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151223/2149d255/attachment.html>