Open Stack

On 07/25/2013 06:38 PM, Zane Bitter wrote:
> On 25/07/13 17:08, Bartosz Górski wrote:
>> First of all sorry for the late reply. I needed some time to understand
>> you vision and check a few things.
>>
>> On 07/24/2013 08:40 AM, Clint Byrum wrote:
>>> Excerpts from Adrian Otto's message of 2013-07-23 21:22:14 -0700:
>>>> Clint,
>>>>
>>>> On Jul 23, 2013, at 10:03 AM, Clint Byrum <clint at fewbar.com>
>>>>   wrote:
>>>>
>>>>> Excerpts from Steve Baker's message of 2013-07-22 21:43:05 -0700:
>>>>>> On 07/23/2013 10:46 AM, Angus Salkeld wrote:
>>>>>>> On 22/07/13 16:52 +0200, Bartosz Górski wrote:
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> I would like to start a discussion about the blueprint I raised
>>>>>>>> about
>>>>>>>> multi region support.
>>>>>>>> I would like to get feedback from you. If something is not 
>>>>>>>> clear or
>>>>>>>> you have questions do not hesitate to ask.
>>>>>>>> Please let me know what you think.
>>>>>>>>
>>>>>>>> Blueprint:
>>>>>>>> https://blueprints.launchpad.net/heat/+spec/multi-region-support
>>>>>>>>
>>>>>>>> Wikipage:
>>>>>>>> https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat 
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> What immediatley looks odd to me is you have a MultiCloud Heat
>>>>>>> talking
>>>>>>> to other Heat's in each region. This seems like unneccessary
>>>>>>> complexity to me.
>>>>>>> I would have expected one Heat to do this job.
>> Yes. You are right. I'm seeing it now. One heat talking to other heat
>> service would be an overkill and unnecessary complexity.
>> Better solution is to use one heat which will be talking directly to
>> services (nova, glance, ...).
>
> +1, and this is reasonably easy for Heat to do, simply by asking for a 
> different region's service catalog from keystone.
>>
>> Also solution with two heat services (one for single region and one for
>> multi region) has a lot of common parts.
>> For example single region heat needs to create a dependencies graph
>> where each node is a resource and multi region a graph where each node
>> is template.
>
> I'm not convinced about the need for this though.
>
> I looked at your example on the wiki, and it just contains a bunch of 
> East resources that reference each other and a bunch of West resources 
> that reference each other and never the twain shall meet. And that 
> seems inevitable - you can't, e.g. connect a Cinder volume in one 
> region to a Nova server in another region. So I'm wondering why we 
> would ever want to mix regions in a single template, with a single 
> dependency graph, when it's not really meaningful to have dependencies 
> between resources in different regions. There's no actual 
> orchestration to do at that level.
>
> It seems to me your example would have been better as two templates 
> (or, even better, the same template launched in two different regions, 
> since I couldn't detect any differences between East vs. West).
>
> Note that there are plans in the works to make passing multiple files 
> to Heat a more pleasant experience.
>
> I think creating an OS::Heat::Stack resource with a Region property 
> solves 95%+ of the problem, without adding or modifying any other 
> resources.
>
We want to start from something simple. At the beginning we are assuming 
no dependencies between resources from different region. Our first use 
case (the one on the wikipage) uses this assumptions. So this is why it 
can be easily split on two separate single region templates.

Our goal is to support dependencies between resources from different 
regions. Our second use case (I will add it with more details to the 
wikipage soon) is similar to deploying two instances (app server + db 
server) wordpress in two different regions (app server in the first 
region and db server in the second). Regions will be connected to each 
other via VPN connection . In this case configuration of app server 
depends on db server. We need to know IP address of created DB server to 
properly configure App server. It forces us to wait with creating app 
server until db server will be created.

More complicated use case with load balancers and more regions are also 
in ours minds.

Thanks,
Bartosz

> cheers,
> Zane.
>
>> So in fact it is better to have only one but more powerful heat service.
>>>>>> It should be possible to achieve this with a single Heat
>>>>>> installation -
>>>>>> that would make the architecture much simpler.
>>>>>>
>>>>> Agreed that it would be simpler and is definitely possible.
>>>>>
>>>>> However, consider that having a Heat in each region means Heat is 
>>>>> more
>>>>> resilient to failure. So focusing on a way to make multiple Heat's
>>>>> collaborate, rather than on a way to make one Heat talk to two 
>>>>> regions
>>>>> may be a more productive exercise.
>>>> I agree with Angus, Steve Baker, and Randall on this one. We should
>>>> aim for simplicity where practical. Having Heat services interacting
>>>> with other Heat services seems like a whole category of complexity
>>>> that's difficult to justify. If it were implemented as Steve Baker
>>>> described, and the local Heat service were unavailable, the client
>>>> may still have the option to use a Heat service in another region and
>>>> still successfully orchestrate. That seems to me like a failure mode
>>>> that's easier for users to anticipate and plan for.
>> Steve I really like you concept with the context as a resource. What do
>> you think how we should proceed with it to make it happened?
>>
>> What looks wared for me is the concept that orchestration service
>> deployed in one region can orchestrate other regions.
>> My understating of regions was that they are separated and do not know
>> about each other. So the heat service which is
>> responsible for orchestrating multi region should not be deployed in any
>> of those regions but somewhere else.
>>
>> Right now I also do not see a point for having separate heat service in
>> each region.
>> One heat service with multi region support not deployed in any of the
>> existing regions (logically not physically) looks fine for me.
>>
>>>>
>>> I'm all for keeping the solution simple. However, I am not for making
>>> it simpler than it needs to be to actually achieve its stated goals.
>>>
>>>> Can you further explain your perspective? What sort of failures would
>>>> you expect a network of coordinated Heat services to be more
>>>> effective with? Is there any way this would be more simple or more
>>>> elegant than other options?
>>> I expect partitions across regions to be common. Regions should be
>>> expected to operate completely isolated from one another if need be. 
>>> What
>>> is the point of deploying a service to two regions, if one region's
>>> failure means you cannot manage the resources in the standing region?
>>>
>>> Active/Passive means you now have an untested passive heat engine in
>>> the passive region. You also have a lot of pointers to update when the
>>> active is taken offline or when there is a network partition. Also 
>>> split
>>> brain is basically guaranteed in that scenario.
>>>
>>> Active/Active(/Active/...), where each region's Heat service 
>>> collaborates
>>> and owns its own respective pieces of the stack, means that on 
>>> partition,
>>> one is simply prevented from telling one region to scale/migrate/
>>> etc. onto another one. It also affords a local Heat the ability to
>>> replace resources in a failed region with local resources.
>>>
>>> The way I see it working is actually pretty simple. One stack would
>>> lead to resources in multiple regions. The collaboration I speak of
>>> would simply be that if given a stack that requires crossing regions,
>>> the other Heat is contacted and the same stack is deployed. 
>>> Cross-region
>>> attribute/ref sharing would need an efficient way to pass data about
>>> resources as well.
>>>
>>> Anyway, I'm not the one doing the work, so I'll step back from the
>>> position, but if I were a user who wanted multi-region, I'd certainly
>>> want _a plan_ from day 1 to handle partitions.
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> Thanks,
>> Bartosz
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev