Open Stack

On Thu, Jan 5, 2017 at 2:12 PM, Ronan-Alexandre Cherrueau <
ronan-alexandre.cherrueau at inria.fr> wrote:

> Hello,
>
>
> TL;DR: We make a multi-regions deployment with Kolla. It requires to
> patch the code a little bit, and you can find the diff on our
> GitHub[1]. This patch is just a first attempt to support multi-regions
> in Kolla and it raises questions. Some modifications are not done in
> an idiomatic way and we do not expect this to be merged in Kolla. The
> reminder of this mail explains our patch and states our questions.
>
>
> At Inria/Discovery[2], we evaluate OpenStack at scale for the
> Performance Working Group. So far, we focus on one single OpenStack
> region deployment with hundreds of computes and we always go with
> Kolla for our deployment. Over the last few days, we tried to achieve
> a multi-regions OpenStack deployment with Kolla. We want to share with
> you our current deployment workflow, patches we had to apply on Kolla
> to support multi-regions, and also ask you if we do things correctly.
>
> First of all, our multi-regions deployment follows the one described
> by the OpenStack documentation[3]. Concretely, the deployment
> considers /one/ Administrative Region (AR) that contains Keystone and
> Horizon. This is a Kolla-based deployment, so Keystone is hidden
> behind an HAProxy, and has MariaDB and memcached as backend. At the
> same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
> OpenStack, except Keystone. We got something as follows at the end of
> the deployment:
>
>
> Admin Region (AR):
> - control:
>   * Horizon
>   * HAProxy
>   * Keyston
>   * MariaDB
>   * memcached
>
> OpenStack Region x (OSRx):
> - control:
>   * HAProxy
>   * nova-api/conductor/scheduler
>   * neutron-server/l3/dhcp/...
>   * glance-api/registry
>   * MariaDB
>   * RabbitMQ
>
> - compute1:
>   * nova-compute
>   * neutron-agent
>
> - compute2: ...
>
>
> We do the deployment by running Kolla n+1 times. The first run deploys
> the Administrative Region (AR) and the other runs deploy OpenStack
> Regions (OSR). For each run, we fix the value of `openstack_region_name'
> variable to the name of the current region.
>
> In the context of multi-regions, Keystone (in the AR) should be
> available to all OSRs. This means, there are as many Keystone
> endpoints as regions. For instance, if we consider two OSRs, the
> result of listing endpoints at the end of the AR deployment looks like
> this:
>
>
>  $ openstack endpoint list
>
>  | Region | Serv Name | Serv Type | Interface | URL
>   |
>  |--------+-----------+-----------+-----------+--------------
> ----------------|
>  | AR     | keystone  | identity  | public    |
> http://10.24.63.248:5000/v3  |
>  | AR     | keystone  | identity  | internal  |
> http://10.24.63.248:5000/v3  |
>  | AR     | keystone  | identity  | admin     |
> http://10.24.63.248:35357/v3 |
>  | OSR1   | keystone  | identity  | public    |
> http://10.24.63.248:5000/v3  |
>  | OSR1   | keystone  | identity  | internal  |
> http://10.24.63.248:5000/v3  |
>  | OSR1   | keystone  | identity  | admin     |
> http://10.24.63.248:35357/v3 |
>  | OSR2   | keystone  | identity  | public    |
> http://10.24.63.248:5000/v3  |
>  | OSR2   | keystone  | identity  | internal  |
> http://10.24.63.248:5000/v3  |
>  | OSR2   | keystone  | identity  | admin     |
> http://10.24.63.248:35357/v3 |
>
>
> This requires patching the `keystone/tasks/register.yml' play[4] to
> re-execute the `Creating admin project, user, role, service, and
> endpoint' task for all regions we consider. An example of such a patch
> is given on our GitHub[5]. In this example, the `openstack_regions'
> variable is a list that contains the name of all regions (see [6]). As
> a drawback, the patch implies to know in advance all OSR. A better
> implementation would execute the `Creating admin project, user, role,
> service, and endpoint' task every time a new OSR is going to be
> deployed. But this requires to move the task somewhere else in the
> Kolla workflow and we have no idea where this should be.
>
> In the AR, we also have to change the Horizon configuration file to
> handle multi-regions[7]. The modification could be done easily and
> idiomatically by setting the `node_custome_config' variable to the
> `multi-regions' directory[8] and benefits from Kolla merging config
> system.
>
> Also, deploying OSRs requires patching the kolla-toolbox as it seems
> not region-aware. In particular, patch the `kolla_keystone_service.py'
> module[9] that is responsible for contacting Keystone and creating a
> new endpoint when we register a new OpenStack service.
>
>
>  73  for _endpoint in cloud.keystone_client.endpoints.list():
>  74      if _endpoint.service_id == service.id and \
>  75         _endpoint.interface == interface:
>  76        endpoint = _endpoint
>  77        if endpoint.url != url:
>  78          changed = True
>  79          cloud.keystone_client.endpoints.update(
>  80            endpoint, url=url)
>  81        break
>  82  else:
>  83    changed = True
>  84    cloud.keystone_client.endpoints.create(
>  85      service=service.id,
>  86      url=url,
>  87      interface=interface,
>  88      region=endpoint_region)
>
>
> At some point, this module /create/ or /update/ a service endpoint. It
> first tests if the service is already registered, and update the URL
> if so (l. 74 to 81), or create the new endpoint in other cases (l.
> 82). Unfortunately, the test (l. 74 to 75) only looks at the service
> (e.g., glance) and the interface (e.g., public). This makes the test
> wrong because we deploy the same service many times, but into
> different regions. We have to add an extra condition to not only test
> the service but also its region.
>
>
>  74  if _endpoint.service_id == service.id and \
>  75     _endpoint.interface == interface and \
>  76     _endpoint.region == endpoint_region:
>
>
> Finally, when we deployed OSRs, we override the value of
> `keystone_admin_url', `keystone_internal_url' and `keystone_public_url'
> to target Keystone in AR. We also have to change the
> `[keystone_authtoken]' section of nova/neutron/glance conf[10] so that
> it uses these variables instead of the canonical form with the
> `kolla_internal_fqdn' variable[11]. In this regards, is it due to some
> legacy code that configuration files use the `kolla_internal_fqdn'
> variable instead of `keystone_*_url'?
>

The original value is this:
auth_uri = {{ internal_protocol }}://{{ kolla_internal_fqdn }}:{{
keystone_public_port }}

Kolla does SSL via haproxy, not directly at the service itself. So internal
traffic is http, external is https. In this case, 'kolla_internal_fqdn'
points to the haproxy vip. Its not legacy code, but I am sure it can be
changed to work multiregion in the way you are attempting to do it.

>
> That's almost all. As you can see, handle multi-regions implies a very
> small number of modifications if you call Kolla multiple times.
>
> So, thanks for the great job Kolla team! And waiting for your
> feedback.
>
>
>
> [1] https://github.com/BeyondTheClouds/kolla/commit/
> 7e5f7e6c7936b7cc3136dfc082935e2995a65554
> [2] https://beyondtheclouds.github.io/
> [3] http://docs.openstack.org/arch-design/multi-site-architecture.html
> [4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5
> cf974216ae/ansible/roles/keystone/tasks/register.yml
> [5] https://github.com/BeyondTheClouds/kolla/commit/
> 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
> 40be75c3a2237adfd1a05178f6f60006R1
> [6] https://github.com/BeyondTheClouds/kolla/commit/
> 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
> 607e6e5925d0031dafa79eb80d198640R215
> [7] https://github.com/BeyondTheClouds/kolla/commit/
> 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
> 6cbb63bb9c4843bcf8db4710e588475bR188
> [8] https://github.com/BeyondTheClouds/kolla/tree/
> 7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
> [9] https://github.com/BeyondTheClouds/kolla/commit/
> 7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-
> ba10dd4575f65e03d50d586febdccbadR72
> [10] https://github.com/BeyondTheClouds/kolla/blob/
> 7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
> [11] https://github.com/openstack/kolla/blob/
> 11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/
> templates/nova.conf.j2#L155
>
>
> --
> Ronan-A. Cherrueau
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


None of these pieces seem to be in major conflict with anything else, so it
would just be a matter of upstreaming your work (and tweaking things as we
find them). It looks great!

Thanks,
SamYaple
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170105/5a2792f2/attachment.html>