Open Stack

Hi all,

This topic came up in the 2015-12-15 meeting[1], and again briefly today.
After working with the code that came out of the deployment library spec[2]
I
had some concerns with how we are storing the templates.

Simply put, when we are dealing with 100+ files from tripleo-heat-templates
how can we ensure consistency in Swift without any atomicity or
transactions.
I think this is best explained with a couple of examples.

 - When we create a new deployment plan (upload all the templates to swift)
   how do we handle the case where there is an error? For example, if we are
   uploading 10 files - what do we do if the 5th one fails for some reason?
   There is a patch to do a manual rollback[3], but I have concerns about
   doing this in Python. If Swift is completely inaccessible for a short
   period the rollback wont work either.

 - When deploying to Heat, we need to download all the YAML files from
Swift.
   This can take a couple of seconds. What happens if somebody starts to
   upload a new version of the plan in the middle? We could end up trying to
   deploy half old and half new files. We wouldn't have a consistent view of
   the database.

We had a few suggestions in the meeting:

 - Add a locking mechanism. I would be concerned about deadlocks or having
to
   lock for the full duration of a deploy.
 - Store the files in a tarball (so we only deal with one file). I think we
   could still hit issues with multiple operations unless we guarantee that
   only one instance of the API is ever running.

I think these could potentially be made to work, but at this point are we
using the best tool for the job?

For alternatives, I see a can think of a couple of options:

- Use a database, like we did for Tuskar and most OpenStack API's do.
- Invest time in building something on Swift.
- Glance was transitioning to be a Artifact store. I don't know the status
of
  this or if it would meet out needs.

Any input, ideas or suggestions would be great!

Thanks,
Dougal


[1]:
http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89
[2]:
https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html
[3]: https://review.openstack.org/#/c/257481/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151222/7cd36c2b/attachment.html>