Open Stack

On Wed, Dec 23, 2015 at 09:28:59AM -0600, Ben Nemec wrote:
> On 12/23/2015 03:19 AM, Dougal Matthews wrote:
> > 
> > 
> > On 22 December 2015 at 17:59, Ben Nemec <openstack at nemebean.com
> > <mailto:openstack at nemebean.com>> wrote:
> > 
> >     Can we just do git like I've been suggesting all along? ;-)
> > 
> >     More serious discussion inline. :-)
> > 
> >     On 12/22/2015 09:36 AM, Dougal Matthews wrote:
> >     > Hi all,
> >     >
> >     > This topic came up in the 2015-12-15 meeting[1], and again briefly
> >     today.
> >     > After working with the code that came out of the deployment library
> >     > spec[2] I
> >     > had some concerns with how we are storing the templates.
> >     >
> >     > Simply put, when we are dealing with 100+ files from
> >     tripleo-heat-templates
> >     > how can we ensure consistency in Swift without any atomicity or
> >     > transactions.
> >     > I think this is best explained with a couple of examples.
> >     >
> >     >  - When we create a new deployment plan (upload all the templates
> >     to swift)
> >     >    how do we handle the case where there is an error? For example,
> >     if we are
> >     >    uploading 10 files - what do we do if the 5th one fails for
> >     some reason?
> >     >    There is a patch to do a manual rollback[3], but I have
> >     concerns about
> >     >    doing this in Python. If Swift is completely inaccessible for a
> >     short
> >     >    period the rollback wont work either.
> >     >
> >     >  - When deploying to Heat, we need to download all the YAML files from
> >     > Swift.
> >     >    This can take a couple of seconds. What happens if somebody
> >     starts to
> >     >    upload a new version of the plan in the middle? We could end up
> >     trying to
> >     >    deploy half old and half new files. We wouldn't have a
> >     consistent view of
> >     >    the database.
> >     >
> >     > We had a few suggestions in the meeting:
> >     >
> >     >  - Add a locking mechanism. I would be concerned about deadlocks or
> >     > having to
> >     >    lock for the full duration of a deploy.
> > 
> >     There should be no need to lock the plan for the entire deploy.  It's
> >     not like we're re-reading the templates at the end of the deploy today.
> >      It's a one-shot read and then the plan could be unlocked, at least as
> >     far as I know.
> > 
> > 
> > Good point. That would be holding the lock for longer than we need.
> >  
> > 
> >     The only option where we wouldn't need locking at all is the
> >     read-copy-update model Clint mentions, which might be a valid option as
> >     well.  Whatever we do, there are going to be concurrency issues though.
> >      For example, what happens if two users try to make updates to the plan
> >     at the same time?  If you don't either merge the changes or disallow one
> >     of them completely then one user's changes might be lost.
> > 
> >     TBH, this is further convincing me that we should just make this git
> >     backed and let git handle the merging and conflict resolution (never
> >     mind the fact that it gets us a well-understood version control system
> >     for "free").  For updates that don't conflict with other changes, git
> >     can merge them automatically, but for merge conflicts you just return a
> >     rebase error to the user and make them resolve it.  I have a feeling
> >     this is the behavior we'll converge on eventually anyway, and rather
> >     than reimplement git, let's just use the real thing.
> > 
> > 
> > I'd be curious to hear more how you would go about doing this with git. I've
> > never automated git to this level, so I am concerned about what issues we
> > might hit.
> 
> TBH I haven't thought it through to that extent yet.  I'm mostly
> suggesting it because it seems like a fit for the template storage
> requirements - we know we want version control, we want to be able to
> merge changes from multiple sources, and we want some way to handle
> merge conflicts.  Git does all of this already.
> 
> That said, I'm not sure about everything here.  For example, how would
> you expose merge conflicts to the user?  I don't know that I would want
> to force a user to learn git in order to use TripleO (although that
> would be the devops-y thing to do), but maybe just passing them back the
> files with the merge conflict markers and having them resolve those
> locally and retry the update would work.  I'm not sure how that would
> map to the current version of the API though.  Do we provide any way to
> pass templates back to the user?  I feel like that was kind of a one-way
> street.

What part of the deployment API workflow could result in merge conflicts?

My understanding was that it's something like:

1. Take copy of reference templates tree
2. Introspect tempalates, expose required parameters so user can be
prompted for them
3. Create environment files(s) derived from the user input
4. Validate the combination of (1) and (3)
5. Deploy the templates+environments

On update, (1) would be "overwrite existing version of templates"

So, while I see a clear requirement for versioning (update whatever is in
the template store on update/upgrade), I'm not sure I see the requirement
to handle merge-conflicts.

I can imagine developers or advanced operators doing template hacking might
require such a workflow, but my assumption was that the deployment API
would probably impede such a user's workflow and they'd stick with a more
direct git->heat type process (such as is possible now).

Am I missing something?

Steve