Open Stack

On 12/23/2015 04:19 AM, Dougal Matthews wrote:
>
>
> On 22 December 2015 at 17:59, Ben Nemec <openstack at nemebean.com
> <mailto:openstack at nemebean.com>> wrote:
>
>     Can we just do git like I've been suggesting all along? ;-)
>
>     More serious discussion inline. :-)
>
>     On 12/22/2015 09:36 AM, Dougal Matthews wrote:
>      > Hi all,
>      >
>      > This topic came up in the 2015-12-15 meeting[1], and again
>     briefly today.
>      > After working with the code that came out of the deployment library
>      > spec[2] I
>      > had some concerns with how we are storing the templates.
>      >
>      > Simply put, when we are dealing with 100+ files from
>     tripleo-heat-templates
>      > how can we ensure consistency in Swift without any atomicity or
>      > transactions.
>      > I think this is best explained with a couple of examples.
>      >
>      >  - When we create a new deployment plan (upload all the templates
>     to swift)
>      >    how do we handle the case where there is an error? For
>     example, if we are
>      >    uploading 10 files - what do we do if the 5th one fails for
>     some reason?
>      >    There is a patch to do a manual rollback[3], but I have
>     concerns about
>      >    doing this in Python. If Swift is completely inaccessible for
>     a short
>      >    period the rollback wont work either.
>      >
>      >  - When deploying to Heat, we need to download all the YAML files
>     from
>      > Swift.
>      >    This can take a couple of seconds. What happens if somebody
>     starts to
>      >    upload a new version of the plan in the middle? We could end
>     up trying to
>      >    deploy half old and half new files. We wouldn't have a
>     consistent view of
>      >    the database.
>      >
>      > We had a few suggestions in the meeting:
>      >
>      >  - Add a locking mechanism. I would be concerned about deadlocks or
>      > having to
>      >    lock for the full duration of a deploy.
>
>     There should be no need to lock the plan for the entire deploy.  It's
>     not like we're re-reading the templates at the end of the deploy today.
>       It's a one-shot read and then the plan could be unlocked, at least as
>     far as I know.
>
>
> Good point. That would be holding the lock for longer than we need.
>
>
>     The only option where we wouldn't need locking at all is the
>     read-copy-update model Clint mentions, which might be a valid option as
>     well.  Whatever we do, there are going to be concurrency issues though.
>       For example, what happens if two users try to make updates to the plan
>     at the same time?  If you don't either merge the changes or disallow one
>     of them completely then one user's changes might be lost.
>
>     TBH, this is further convincing me that we should just make this git
>     backed and let git handle the merging and conflict resolution (never
>     mind the fact that it gets us a well-understood version control system
>     for "free").  For updates that don't conflict with other changes, git
>     can merge them automatically, but for merge conflicts you just return a
>     rebase error to the user and make them resolve it.  I have a feeling
>     this is the behavior we'll converge on eventually anyway, and rather
>     than reimplement git, let's just use the real thing.
>
>
> I'd be curious to hear more how you would go about doing this with git. I've
> never automated git to this level, so I am concerned about what issues we
> might hit.
>
> Do you happen to know of any good sources that I can find out more?

One source of inspiration could be the way Openshift handles things. You 
get a git URL to push your $whatever to, and they have git hooks that 
deploy when you push. This would be perfect for merge conflict 
situations. Imagine two cases.

Normal case:
1. User does stuff in web interface or over API
2. The API adds templates and such to git repo, then deploys as needed
3. User is happy!

Merge conflict case:
1. User does stuff in web interface or over API
2. The API tries, but there's a merge conflict. It hands back a 409 and 
a git URL
3. User clones the repo and resolves conflict
4. User pushes repo back up
5. API deploys stuff
6. User is happy!



>     /2 cents
>
>     >  - Store the files in a tarball (so we only deal with one file). I think we
>     >    could still hit issues with multiple operations unless we guarantee that
>     >    only one instance of the API is ever running.
>     >
>     > I think these could potentially be made to work, but at this point are we
>     > using the best tool for the job?
>     >
>     > For alternatives, I see a can think of a couple of options:
>     >
>     > - Use a database, like we did for Tuskar and most OpenStack API's do.
>
>     I kind of like this, in particular because it would allow us to handle
>     migrations between versions the same way as other OpenStack services.
>
>
> Yeah, I feel like this is the "safe" option. It's a well trodden and
> tested path.
>
>     I'm not entirely sure how it maps to our template configuration method
>     though.  Storing a bunch of template blobs in the database feels a
>     little square peg, round hole to me, and might undo a lot of the
>     benefits of using the database in the first place.
>
>
> I don't follow this point. In the way we access Swift, everything is
> essentially
> a blob of text also.
>
>     Now, on the other hand, having a database to store basic data like
>     metadata on plans and such might be a useful thing regardless of where
>     we store the templates themselves.  We could also use it for locking
>     just fine IMHO - TripleO isn't a tool targeted to have cloud-scale
>     number of users.  It's a deployer tool with a relatively limited number
>     of users per installation, so the scaling of a locking mechanism isn't a
>     problem I necessarily think we need to solve.  We have way bigger
>     scaling issues than that to tackle before I think we would hit the limit
>     of a database-locking scheme.
>
>     > - Invest time in building something on Swift.
>     > - Glance was transitioning to be a Artifact store. I don't know the
>     > status of
>     >   this or if it would meet out needs.
>     >
>     > Any input, ideas or suggestions would be great!
>     >
>     > Thanks,
>     > Dougal
>     >
>     >
>     > [1]:
>     >http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89
>     > [2]:
>     >https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html
>     > [3]:https://review.openstack.org/#/c/257481/
>     >
>     >
>      >
>     __________________________________________________________________________
>      > OpenStack Development Mailing List (not for usage questions)
>      > Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>      > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>      >
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

-- 
Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.