Open Stack

On 22/12/15 11:56, Clint Byrum wrote:
> Excerpts from Dougal Matthews's message of 2015-12-22 07:36:02 -0800:
>> Hi all,
>>
>> This topic came up in the 2015-12-15 meeting[1], and again briefly today.
>> After working with the code that came out of the deployment library spec[2]
>> I
>> had some concerns with how we are storing the templates.
>>
>> Simply put, when we are dealing with 100+ files from tripleo-heat-templates
>> how can we ensure consistency in Swift without any atomicity or
>> transactions.
>> I think this is best explained with a couple of examples.
>>
>>   - When we create a new deployment plan (upload all the templates to swift)
>>     how do we handle the case where there is an error? For example, if we are
>>     uploading 10 files - what do we do if the 5th one fails for some reason?
>>     There is a patch to do a manual rollback[3], but I have concerns about
>>     doing this in Python. If Swift is completely inaccessible for a short
>>     period the rollback wont work either.
>>
>
> You could create a unique swift container, upload things to that, and
> then update a pointer in a well-known location to point at that container
> for the new plan only after you've verified it is available. This is a
> primitive form of Read-copy-update.

+1, I think immutable files (actually, whole containers) is the correct 
way to use Swift if we're going to use it (which, for the record, I 
think we should). This is how Glance stores disk images, and also how 
the 
Glance-artifact-store-that-is-now-a-separate-metadata-service-with-a-different-name 
(help me out here) worked/works. Ideally we'd use that to store any 
metadata, but a (mutable) pointer in Swift to the latest is probably 
sufficient for this use case.

>>   - When deploying to Heat, we need to download all the YAML files from
>> Swift.
>>     This can take a couple of seconds. What happens if somebody starts to
>>     upload a new version of the plan in the middle? We could end up trying to
>>     deploy half old and half new files. We wouldn't have a consistent view of
>>     the database.
>>
>
> Perhaps you should land a change in Heat to allow templates to be directly
> downloaded by the engine from swift without needing to be uploaded.

Yes, we should definitely add this in Heat.

> In
> the past allowing URLs to be downloaded unfettered was disabled because
> we don't want Heat to be a DoS engine, but swift would be in-cloud and
> could be restricted to the stack owner.

Actually, it's still allowed (but not used by python-heatclient) to 
preserve backward compatibility, and likely won't be removed until a v2 
API spin :/

>> We had a few suggestions in the meeting:
>>
>>   - Add a locking mechanism. I would be concerned about deadlocks or having
>> to
>>     lock for the full duration of a deploy.
>
> Deadlocks only happen when competing interests lock _two_ things in
> different order. Have one thing to lock, and you don't have to worry
> about this.
>
>>   - Store the files in a tarball (so we only deal with one file). I think we
>>     could still hit issues with multiple operations unless we guarantee that
>>     only one instance of the API is ever running.

I think immutable data is a better solution; you can't pass a tarball to 
Heat.

>> I think these could potentially be made to work, but at this point are we
>> using the best tool for the job?
>>
>> For alternatives, I see a can think of a couple of options:
>>
>> - Use a database, like we did for Tuskar and most OpenStack API's do.

This makes me deeply uneasy; I think one of the lessons of Tuskar is 
that putting the templates in a database accessible only from the API 
dramatically reduces our flexibility while not buying us anything in 
particular.

cheers,
Zane.

>
> It's worth noting that many OpenStack API's/daemons are using the
> database + MQ to provide consistency in a distributed fashion, and many
> have identified that this doesn't scale particularly well, and looking
> at tooz to help bring in DLM's. In fact, a spec recently landed around
> this:
>
> http://specs.openstack.org/openstack/openstack-specs/specs/chronicles-of-a-dlm.html
>
> So if you are only using the DB for consistency, you might want to just
> use tooz+swift.
>
>> - Invest time in building something on Swift.
>> - Glance was transitioning to be a Artifact store. I don't know the status
>> of
>>    this or if it would meet out needs.
>
> I'd say the artifact store is more about exposing options to users, and
> less about providing primitives for an API.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>