[openstack-dev] [kolla] Multi-Regions Support
Ronan-Alexandre Cherrueau
ronan-alexandre.cherrueau at inria.fr
Fri Jan 6 13:56:20 UTC 2017
Hello,
Thanks for your feedback.
> The original value is this:
> auth_uri = {{ internal_protocol }}://{{ kolla_internal_fqdn }}:{{ keystone_public_port }}
>
> Kolla does SSL via haproxy, not directly at the service itself. So internal traffic is http, external is https. In this case, 'kolla_internal_fqdn' points to the haproxy vip.
Got it, but if you look at the default definition of
`keystone_internal_url', inside `group_vars/all.yml', you have:
332 keystone_internal_url: "{{ internal_protocol }}://{{
kolla_internal_fqdn }}:{{ keystone_public_port }}/v3"
As you can see, the definition is close to the original value of
`auth_uri' (except the v3). So, it seems to be a better idea to set
`auth_uri' to `keystone_internal_url' instead of the canonical form.
Especially in the context of multi-regions. In this context, you cannot
change the value of `{{ kolla_internal_fqdn }}' because, as you say, it
has to target the region's HAProxy vip. However, you have to set the
value of `auth_uri' to target the Administrative Region HAProxy vip. So,
by defining in nova/neutron/glance conf:
[keystone_authtoken]
auth_uri = {{ keystone_internal_url }}
auth_url = {{ keystone_admin_url }}
Then you can target the Administrative Region HAProxy vip by redefining
`keystone_*_url' without redefining `kolla_internal_fqdn'.
> Its not legacy code, but I am sure it can be changed to work multiregion in the way you are attempting to do it.
I was talking about legacy code because, in the stable/newton branch,
the `auth_uri' is already set to `{{ keystone_internal_url }}', but
solely for Kubernetes engine[1].
Shall we proceed by pushing to Gerrit with a dedicated page in the
documentation? Also, our patch is based on stable/newton, is it better
if we follow the kolla-ansible master?
[1] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L151
On Thu, Jan 5, 2017 at 7:01 PM, Sam Yaple <samuel at yaple.net> wrote:
> On Thu, Jan 5, 2017 at 2:12 PM, Ronan-Alexandre Cherrueau
> <ronan-alexandre.cherrueau at inria.fr> wrote:
>>
>> Hello,
>>
>>
>> TL;DR: We make a multi-regions deployment with Kolla. It requires to
>> patch the code a little bit, and you can find the diff on our
>> GitHub[1]. This patch is just a first attempt to support multi-regions
>> in Kolla and it raises questions. Some modifications are not done in
>> an idiomatic way and we do not expect this to be merged in Kolla. The
>> reminder of this mail explains our patch and states our questions.
>>
>>
>> At Inria/Discovery[2], we evaluate OpenStack at scale for the
>> Performance Working Group. So far, we focus on one single OpenStack
>> region deployment with hundreds of computes and we always go with
>> Kolla for our deployment. Over the last few days, we tried to achieve
>> a multi-regions OpenStack deployment with Kolla. We want to share with
>> you our current deployment workflow, patches we had to apply on Kolla
>> to support multi-regions, and also ask you if we do things correctly.
>>
>> First of all, our multi-regions deployment follows the one described
>> by the OpenStack documentation[3]. Concretely, the deployment
>> considers /one/ Administrative Region (AR) that contains Keystone and
>> Horizon. This is a Kolla-based deployment, so Keystone is hidden
>> behind an HAProxy, and has MariaDB and memcached as backend. At the
>> same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
>> OpenStack, except Keystone. We got something as follows at the end of
>> the deployment:
>>
>>
>> Admin Region (AR):
>> - control:
>> * Horizon
>> * HAProxy
>> * Keyston
>> * MariaDB
>> * memcached
>>
>> OpenStack Region x (OSRx):
>> - control:
>> * HAProxy
>> * nova-api/conductor/scheduler
>> * neutron-server/l3/dhcp/...
>> * glance-api/registry
>> * MariaDB
>> * RabbitMQ
>>
>> - compute1:
>> * nova-compute
>> * neutron-agent
>>
>> - compute2: ...
>>
>>
>> We do the deployment by running Kolla n+1 times. The first run deploys
>> the Administrative Region (AR) and the other runs deploy OpenStack
>> Regions (OSR). For each run, we fix the value of `openstack_region_name'
>> variable to the name of the current region.
>>
>> In the context of multi-regions, Keystone (in the AR) should be
>> available to all OSRs. This means, there are as many Keystone
>> endpoints as regions. For instance, if we consider two OSRs, the
>> result of listing endpoints at the end of the AR deployment looks like
>> this:
>>
>>
>> $ openstack endpoint list
>>
>> | Region | Serv Name | Serv Type | Interface | URL
>> |
>>
>> |--------+-----------+-----------+-----------+------------------------------|
>> | AR | keystone | identity | public |
>> http://10.24.63.248:5000/v3 |
>> | AR | keystone | identity | internal |
>> http://10.24.63.248:5000/v3 |
>> | AR | keystone | identity | admin |
>> http://10.24.63.248:35357/v3 |
>> | OSR1 | keystone | identity | public |
>> http://10.24.63.248:5000/v3 |
>> | OSR1 | keystone | identity | internal |
>> http://10.24.63.248:5000/v3 |
>> | OSR1 | keystone | identity | admin |
>> http://10.24.63.248:35357/v3 |
>> | OSR2 | keystone | identity | public |
>> http://10.24.63.248:5000/v3 |
>> | OSR2 | keystone | identity | internal |
>> http://10.24.63.248:5000/v3 |
>> | OSR2 | keystone | identity | admin |
>> http://10.24.63.248:35357/v3 |
>>
>>
>> This requires patching the `keystone/tasks/register.yml' play[4] to
>> re-execute the `Creating admin project, user, role, service, and
>> endpoint' task for all regions we consider. An example of such a patch
>> is given on our GitHub[5]. In this example, the `openstack_regions'
>> variable is a list that contains the name of all regions (see [6]). As
>> a drawback, the patch implies to know in advance all OSR. A better
>> implementation would execute the `Creating admin project, user, role,
>> service, and endpoint' task every time a new OSR is going to be
>> deployed. But this requires to move the task somewhere else in the
>> Kolla workflow and we have no idea where this should be.
>>
>> In the AR, we also have to change the Horizon configuration file to
>> handle multi-regions[7]. The modification could be done easily and
>> idiomatically by setting the `node_custome_config' variable to the
>> `multi-regions' directory[8] and benefits from Kolla merging config
>> system.
>>
>> Also, deploying OSRs requires patching the kolla-toolbox as it seems
>> not region-aware. In particular, patch the `kolla_keystone_service.py'
>> module[9] that is responsible for contacting Keystone and creating a
>> new endpoint when we register a new OpenStack service.
>>
>>
>> 73 for _endpoint in cloud.keystone_client.endpoints.list():
>> 74 if _endpoint.service_id == service.id and \
>> 75 _endpoint.interface == interface:
>> 76 endpoint = _endpoint
>> 77 if endpoint.url != url:
>> 78 changed = True
>> 79 cloud.keystone_client.endpoints.update(
>> 80 endpoint, url=url)
>> 81 break
>> 82 else:
>> 83 changed = True
>> 84 cloud.keystone_client.endpoints.create(
>> 85 service=service.id,
>> 86 url=url,
>> 87 interface=interface,
>> 88 region=endpoint_region)
>>
>>
>> At some point, this module /create/ or /update/ a service endpoint. It
>> first tests if the service is already registered, and update the URL
>> if so (l. 74 to 81), or create the new endpoint in other cases (l.
>> 82). Unfortunately, the test (l. 74 to 75) only looks at the service
>> (e.g., glance) and the interface (e.g., public). This makes the test
>> wrong because we deploy the same service many times, but into
>> different regions. We have to add an extra condition to not only test
>> the service but also its region.
>>
>>
>> 74 if _endpoint.service_id == service.id and \
>> 75 _endpoint.interface == interface and \
>> 76 _endpoint.region == endpoint_region:
>>
>>
>> Finally, when we deployed OSRs, we override the value of
>> `keystone_admin_url', `keystone_internal_url' and `keystone_public_url'
>> to target Keystone in AR. We also have to change the
>> `[keystone_authtoken]' section of nova/neutron/glance conf[10] so that
>> it uses these variables instead of the canonical form with the
>> `kolla_internal_fqdn' variable[11]. In this regards, is it due to some
>> legacy code that configuration files use the `kolla_internal_fqdn'
>> variable instead of `keystone_*_url'?
>
>
> The original value is this:
> auth_uri = {{ internal_protocol }}://{{ kolla_internal_fqdn }}:{{
> keystone_public_port }}
>
> Kolla does SSL via haproxy, not directly at the service itself. So internal
> traffic is http, external is https. In this case, 'kolla_internal_fqdn'
> points to the haproxy vip. Its not legacy code, but I am sure it can be
> changed to work multiregion in the way you are attempting to do it.
>>
>>
>> That's almost all. As you can see, handle multi-regions implies a very
>> small number of modifications if you call Kolla multiple times.
>>
>> So, thanks for the great job Kolla team! And waiting for your
>> feedback.
>>
>>
>>
>> [1]
>> https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554
>> [2] https://beyondtheclouds.github.io/
>> [3] http://docs.openstack.org/arch-design/multi-site-architecture.html
>> [4]
>> https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
>> [5]
>> https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
>> [6]
>> https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
>> [7]
>> https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
>> [8]
>> https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
>> [9]
>> https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
>> [10]
>> https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
>> [11]
>> https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155
>>
>>
>> --
>> Ronan-A. Cherrueau
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> None of these pieces seem to be in major conflict with anything else, so it
> would just be a matter of upstreaming your work (and tweaking things as we
> find them). It looks great!
>
> Thanks,
> SamYaple
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Ronan-A. Cherrueau
More information about the OpenStack-dev
mailing list