[openstack-dev] [Heat] Multi region support for Heat

Zane Bitter zbitter at redhat.com
Thu Jul 25 16:38:19 UTC 2013


On 25/07/13 17:08, Bartosz Górski wrote:
> First of all sorry for the late reply. I needed some time to understand
> you vision and check a few things.
>
> On 07/24/2013 08:40 AM, Clint Byrum wrote:
>> Excerpts from Adrian Otto's message of 2013-07-23 21:22:14 -0700:
>>> Clint,
>>>
>>> On Jul 23, 2013, at 10:03 AM, Clint Byrum <clint at fewbar.com>
>>>   wrote:
>>>
>>>> Excerpts from Steve Baker's message of 2013-07-22 21:43:05 -0700:
>>>>> On 07/23/2013 10:46 AM, Angus Salkeld wrote:
>>>>>> On 22/07/13 16:52 +0200, Bartosz Górski wrote:
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I would like to start a discussion about the blueprint I raised
>>>>>>> about
>>>>>>> multi region support.
>>>>>>> I would like to get feedback from you. If something is not clear or
>>>>>>> you have questions do not hesitate to ask.
>>>>>>> Please let me know what you think.
>>>>>>>
>>>>>>> Blueprint:
>>>>>>> https://blueprints.launchpad.net/heat/+spec/multi-region-support
>>>>>>>
>>>>>>> Wikipage:
>>>>>>> https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat
>>>>>>>
>>>>>>>
>>>>>> What immediatley looks odd to me is you have a MultiCloud Heat
>>>>>> talking
>>>>>> to other Heat's in each region. This seems like unneccessary
>>>>>> complexity to me.
>>>>>> I would have expected one Heat to do this job.
> Yes. You are right. I'm seeing it now. One heat talking to other heat
> service would be an overkill and unnecessary complexity.
> Better solution is to use one heat which will be talking directly to
> services (nova, glance, ...).

+1, and this is reasonably easy for Heat to do, simply by asking for a 
different region's service catalog from keystone.
>
> Also solution with two heat services (one for single region and one for
> multi region) has a lot of common parts.
> For example single region heat needs to create a dependencies graph
> where each node is a resource and multi region a graph where each node
> is template.

I'm not convinced about the need for this though.

I looked at your example on the wiki, and it just contains a bunch of 
East resources that reference each other and a bunch of West resources 
that reference each other and never the twain shall meet. And that seems 
inevitable - you can't, e.g. connect a Cinder volume in one region to a 
Nova server in another region. So I'm wondering why we would ever want 
to mix regions in a single template, with a single dependency graph, 
when it's not really meaningful to have dependencies between resources 
in different regions. There's no actual orchestration to do at that level.

It seems to me your example would have been better as two templates (or, 
even better, the same template launched in two different regions, since 
I couldn't detect any differences between East vs. West).

Note that there are plans in the works to make passing multiple files to 
Heat a more pleasant experience.

I think creating an OS::Heat::Stack resource with a Region property 
solves 95%+ of the problem, without adding or modifying any other resources.

cheers,
Zane.

> So in fact it is better to have only one but more powerful heat service.
>>>>> It should be possible to achieve this with a single Heat
>>>>> installation -
>>>>> that would make the architecture much simpler.
>>>>>
>>>> Agreed that it would be simpler and is definitely possible.
>>>>
>>>> However, consider that having a Heat in each region means Heat is more
>>>> resilient to failure. So focusing on a way to make multiple Heat's
>>>> collaborate, rather than on a way to make one Heat talk to two regions
>>>> may be a more productive exercise.
>>> I agree with Angus, Steve Baker, and Randall on this one. We should
>>> aim for simplicity where practical. Having Heat services interacting
>>> with other Heat services seems like a whole category of complexity
>>> that's difficult to justify. If it were implemented as Steve Baker
>>> described, and the local Heat service were unavailable, the client
>>> may still have the option to use a Heat service in another region and
>>> still successfully orchestrate. That seems to me like a failure mode
>>> that's easier for users to anticipate and plan for.
> Steve I really like you concept with the context as a resource. What do
> you think how we should proceed with it to make it happened?
>
> What looks wared for me is the concept that orchestration service
> deployed in one region can orchestrate other regions.
> My understating of regions was that they are separated and do not know
> about each other. So the heat service which is
> responsible for orchestrating multi region should not be deployed in any
> of those regions but somewhere else.
>
> Right now I also do not see a point for having separate heat service in
> each region.
> One heat service with multi region support not deployed in any of the
> existing regions (logically not physically) looks fine for me.
>
>>>
>> I'm all for keeping the solution simple. However, I am not for making
>> it simpler than it needs to be to actually achieve its stated goals.
>>
>>> Can you further explain your perspective? What sort of failures would
>>> you expect a network of coordinated Heat services to be more
>>> effective with? Is there any way this would be more simple or more
>>> elegant than other options?
>> I expect partitions across regions to be common. Regions should be
>> expected to operate completely isolated from one another if need be. What
>> is the point of deploying a service to two regions, if one region's
>> failure means you cannot manage the resources in the standing region?
>>
>> Active/Passive means you now have an untested passive heat engine in
>> the passive region. You also have a lot of pointers to update when the
>> active is taken offline or when there is a network partition. Also split
>> brain is basically guaranteed in that scenario.
>>
>> Active/Active(/Active/...), where each region's Heat service collaborates
>> and owns its own respective pieces of the stack, means that on partition,
>> one is simply prevented from telling one region to scale/migrate/
>> etc. onto another one. It also affords a local Heat the ability to
>> replace resources in a failed region with local resources.
>>
>> The way I see it working is actually pretty simple. One stack would
>> lead to resources in multiple regions. The collaboration I speak of
>> would simply be that if given a stack that requires crossing regions,
>> the other Heat is contacted and the same stack is deployed. Cross-region
>> attribute/ref sharing would need an efficient way to pass data about
>> resources as well.
>>
>> Anyway, I'm not the one doing the work, so I'll step back from the
>> position, but if I were a user who wanted multi-region, I'd certainly
>> want _a plan_ from day 1 to handle partitions.
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Thanks,
> Bartosz
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




More information about the OpenStack-dev mailing list