Open Stack

On 23 December 2015 at 12:01, Steven Hardy <shardy at redhat.com> wrote:

> On Tue, Dec 22, 2015 at 03:36:02PM +0000, Dougal Matthews wrote:
> >    Hi all,
> >
> >    This topic came up in the 2015-12-15 meeting[1], and again briefly
> today.
> >    After working with the code that came out of the deployment library
> >    spec[2] I
> >    had some concerns with how we are storing the templates.
> >
> >    Simply put, when we are dealing with 100+ files from
> >    tripleo-heat-templates
> >    how can we ensure consistency in Swift without any atomicity or
> >    transactions.
> >    I think this is best explained with a couple of examples.
> >
> >    Â - When we create a new deployment plan (upload all the templates to
> >    swift)
> >    Â Â  how do we handle the case where there is an error? For example,
> if we
> >    are
> >    Â Â  uploading 10 files - what do we do if the 5th one fails for some
> >    reason?
> >    Â Â  There is a patch to do a manual rollback[3], but I have concerns
> >    about
> >    Â Â  doing this in Python. If Swift is completely inaccessible for a
> short
> >    Â Â  period the rollback wont work either.
>
> How does using a different data store help fix this error path?
>
> Regardless of the Swift/DB/Git choice, you have to detect something failed,
> and mark the plan as in a failed state.
>

If your connection to a database such as postrges drops mid-transaction it
will be rolled back. So for a networking issue it would help. Granted, I am
not
sure exactly what happens if somebody pulls the plug on the database, but
then you probably have bigger issues. Failed network requests seem
relatively
common.



> >    Â - When deploying to Heat, we need to download all the YAML files
> from
> >    Swift.
> >    Â Â  This can take a couple of seconds. What happens if somebody
> starts to
> >    Â Â  upload a new version of the plan in the middle? We could end up
> >    trying to
> >    Â Â  deploy half old and half new files. We wouldn't have a consistent
> >    view of
> >    Â Â  the database.
>
> If this happens, the API design is wrong - we should have a plan reference
> a unique identifier, including a version (e.g through swift versioned
> objects, git tags/branches or whatever), then on deployment heat should
> download exactly one version of those artefacts (e.g via a swift tempurl or
> a git URL, or whatever).
>

Sure, this sounds like a valid point against the design of both the storage
spec
that was merged and the API spec that is up for review.


>
> FYI heatclient already supports downloading template objects directly from
> swift, and heat/heatclient already support downloading from http URLs, so
> all we have to do AFAICS is generate a URL pointing to a consistent
> version of the templates.
>

I wasn't aware of this feature, I'll take a look but that sounds useful.


>
> >    We had a few suggestions in the meeting:
> >
> >    Â - Add a locking mechanism. I would be concerned about deadlocks or
> >    having to
> >    Â Â  lock for the full duration of a deploy.
>
> I think we need to create a new version of a plan every time it's modified,
> then on deployment the concern around locking is much reduced, or even
> eliminated?
>

Yup, if we essentially make everything immutable that would solve a number
of issues.



> >    Â - Store the files in a tarball (so we only deal with one file). I
> think
> >    we
> >    Â Â  could still hit issues with multiple operations unless we
> guarantee
> >    that
> >    Â Â  only one instance of the API is ever running.
> >
> >    I think these could potentially be made to work, but at this point
> are we
> >    using the best tool for the job?
> >
> >    For alternatives, I see a can think of a couple of options:
> >
> >    - Use a database, like we did for Tuskar and most OpenStack API's do.
>
> Storing template files in a database with Tuskar was a mistake - we should
> *not* do that again IMO - it was a very opaque interface and caused a lot
> of confusion in use.
>

The interface is via the API, so I don't see how the storage mechanism would
change that. We could change the current WIP code to use a database or
git or anything else without changing the interface. Perhaps we need to
focus
more on what interface we need and the specific problems with the one Tuskar
provided? I thought the primary issue with Tuskar was with the template
mangling and not the actual process of accessing the templates etc.


Storing code (in this case yaml files) in some sort of versioned repository
> is *such* a solved problem - it'll be total wheel-reinventing if we
> implement template revision control inside a bespoke DB IMHO.
>

It's interesting that you mention revision control - this IIRC hasn't been
a topic
mentioned in any of the specs. It might be that we need to step back and
look
at this topic more in general.


I think swift is probably a workable solution (using versioning and
> tempurl's), but we should really consider making the template store
> pluggable, as having the API backed by an (operator visible) git repo is
> likely to be a nicer and more familiar solution.
>

The git option is referenced now and then, but I'm not aware of the details
in
how this would work. Is this a pattern used by other projects? I'd like to
do
some reading as I'm not sure how I would use git for this.

The code has always been somewhat pluggable, that was my feedback on
the early storage patches - I had a feeling this topic might come up.
However,
at the moment there is only a Swift implementation.



> The question around validation data I'm less clear on - we already store
> discovery data in swift, which seems to work OK - how is the validation
> data different, in that it warrants a bespoke DB data store?
>
> Steve
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151223/2efb7905/attachment.html>