[openstack-dev] [TripleO] Is Swift a good choice of database for the TripleO API?
Dougal Matthews
dougal at redhat.com
Wed Jan 6 14:25:38 UTC 2016
On 6 January 2016 at 12:30, Ryan Brown <rybrown at redhat.com> wrote:
> On 12/23/2015 04:19 AM, Dougal Matthews wrote:
>
>>
>>
>> On 22 December 2015 at 17:59, Ben Nemec <openstack at nemebean.com
>> <mailto:openstack at nemebean.com>> wrote:
>>
>> Can we just do git like I've been suggesting all along? ;-)
>>
>> More serious discussion inline. :-)
>>
>> On 12/22/2015 09:36 AM, Dougal Matthews wrote:
>> > Hi all,
>> >
>> > This topic came up in the 2015-12-15 meeting[1], and again
>> briefly today.
>> > After working with the code that came out of the deployment library
>> > spec[2] I
>> > had some concerns with how we are storing the templates.
>> >
>> > Simply put, when we are dealing with 100+ files from
>> tripleo-heat-templates
>> > how can we ensure consistency in Swift without any atomicity or
>> > transactions.
>> > I think this is best explained with a couple of examples.
>> >
>> > - When we create a new deployment plan (upload all the templates
>> to swift)
>> > how do we handle the case where there is an error? For
>> example, if we are
>> > uploading 10 files - what do we do if the 5th one fails for
>> some reason?
>> > There is a patch to do a manual rollback[3], but I have
>> concerns about
>> > doing this in Python. If Swift is completely inaccessible for
>> a short
>> > period the rollback wont work either.
>> >
>> > - When deploying to Heat, we need to download all the YAML files
>> from
>> > Swift.
>> > This can take a couple of seconds. What happens if somebody
>> starts to
>> > upload a new version of the plan in the middle? We could end
>> up trying to
>> > deploy half old and half new files. We wouldn't have a
>> consistent view of
>> > the database.
>> >
>> > We had a few suggestions in the meeting:
>> >
>> > - Add a locking mechanism. I would be concerned about deadlocks or
>> > having to
>> > lock for the full duration of a deploy.
>>
>> There should be no need to lock the plan for the entire deploy. It's
>> not like we're re-reading the templates at the end of the deploy
>> today.
>> It's a one-shot read and then the plan could be unlocked, at least
>> as
>> far as I know.
>>
>>
>> Good point. That would be holding the lock for longer than we need.
>>
>>
>> The only option where we wouldn't need locking at all is the
>> read-copy-update model Clint mentions, which might be a valid option
>> as
>> well. Whatever we do, there are going to be concurrency issues
>> though.
>> For example, what happens if two users try to make updates to the
>> plan
>> at the same time? If you don't either merge the changes or disallow
>> one
>> of them completely then one user's changes might be lost.
>>
>> TBH, this is further convincing me that we should just make this git
>> backed and let git handle the merging and conflict resolution (never
>> mind the fact that it gets us a well-understood version control system
>> for "free"). For updates that don't conflict with other changes, git
>> can merge them automatically, but for merge conflicts you just return
>> a
>> rebase error to the user and make them resolve it. I have a feeling
>> this is the behavior we'll converge on eventually anyway, and rather
>> than reimplement git, let's just use the real thing.
>>
>>
>> I'd be curious to hear more how you would go about doing this with git.
>> I've
>> never automated git to this level, so I am concerned about what issues we
>> might hit.
>>
>> Do you happen to know of any good sources that I can find out more?
>>
>
> One source of inspiration could be the way Openshift handles things. You
> get a git URL to push your $whatever to, and they have git hooks that
> deploy when you push. This would be perfect for merge conflict situations.
> Imagine two cases.
>
Interesting. I hadn't even considered that we could act as a git remote.
That would
be pretty neat. It would mean that we are very tied to git, which isn't
necessarily a
bad thing but it is something to consider.
Normal case:
> 1. User does stuff in web interface or over API
> 2. The API adds templates and such to git repo, then deploys as needed
> 3. User is happy!
>
> Merge conflict case:
> 1. User does stuff in web interface or over API
> 2. The API tries, but there's a merge conflict. It hands back a 409 and a
> git URL
> 3. User clones the repo and resolves conflict
> 4. User pushes repo back up
> 5. API deploys stuff
> 6. User is happy!
>
>
>
> /2 cents
>>
>> > - Store the files in a tarball (so we only deal with one file). I
>> think we
>> > could still hit issues with multiple operations unless we
>> guarantee that
>> > only one instance of the API is ever running.
>> >
>> > I think these could potentially be made to work, but at this point
>> are we
>> > using the best tool for the job?
>> >
>> > For alternatives, I see a can think of a couple of options:
>> >
>> > - Use a database, like we did for Tuskar and most OpenStack API's
>> do.
>>
>> I kind of like this, in particular because it would allow us to handle
>> migrations between versions the same way as other OpenStack services.
>>
>>
>> Yeah, I feel like this is the "safe" option. It's a well trodden and
>> tested path.
>>
>> I'm not entirely sure how it maps to our template configuration method
>> though. Storing a bunch of template blobs in the database feels a
>> little square peg, round hole to me, and might undo a lot of the
>> benefits of using the database in the first place.
>>
>>
>> I don't follow this point. In the way we access Swift, everything is
>> essentially
>> a blob of text also.
>>
>> Now, on the other hand, having a database to store basic data like
>> metadata on plans and such might be a useful thing regardless of where
>> we store the templates themselves. We could also use it for locking
>> just fine IMHO - TripleO isn't a tool targeted to have cloud-scale
>> number of users. It's a deployer tool with a relatively limited
>> number
>> of users per installation, so the scaling of a locking mechanism
>> isn't a
>> problem I necessarily think we need to solve. We have way bigger
>> scaling issues than that to tackle before I think we would hit the
>> limit
>> of a database-locking scheme.
>>
>> > - Invest time in building something on Swift.
>> > - Glance was transitioning to be a Artifact store. I don't know the
>> > status of
>> > this or if it would meet out needs.
>> >
>> > Any input, ideas or suggestions would be great!
>> >
>> > Thanks,
>> > Dougal
>> >
>> >
>> > [1]:
>> >
>> http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89
>> > [2]:
>> >
>> https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html
>> > [3]:https://review.openstack.org/#/c/257481/
>> >
>> >
>> >
>>
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> --
> Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160106/add27f40/attachment.html>
More information about the OpenStack-dev
mailing list