Open Stack

On 01/05/2017 09:12 AM, Ronan-Alexandre Cherrueau wrote:
> Hello,
>
> TL;DR: We make a multi-regions deployment with Kolla. It requires to
> patch the code a little bit, and you can find the diff on our
> GitHub[1]. This patch is just a first attempt to support multi-regions
> in Kolla and it raises questions. Some modifications are not done in
> an idiomatic way and we do not expect this to be merged in Kolla. The
> reminder of this mail explains our patch and states our questions.
>
> At Inria/Discovery[2], we evaluate OpenStack at scale for the
> Performance Working Group. So far, we focus on one single OpenStack
> region deployment with hundreds of computes and we always go with
> Kolla for our deployment. Over the last few days, we tried to achieve
> a multi-regions OpenStack deployment with Kolla. We want to share with
> you our current deployment workflow, patches we had to apply on Kolla
> to support multi-regions, and also ask you if we do things correctly.
>
> First of all, our multi-regions deployment follows the one described
> by the OpenStack documentation[3].

I don't see an "Admin Region" as part of the OpenStack documentation for 
multi-region deployment. I also see LDAP mentioned as the recommended 
authentication/IdM store.

 > Concretely, the deployment
> considers /one/ Administrative Region (AR) that contains Keystone and
> Horizon.

That's not a region. Those should be shared resources *across* regions.

 > This is a Kolla-based deployment, so Keystone is hidden
> behind an HAProxy, and has MariaDB and memcached as backend.

I thought at Inria, the Nova "MySQL DB has been replaced by the noSQL 
system REDIS"? But here, you're using MariaDB -- a non-distributed 
database -- for the Keystone component which is the very thing that is 
the most highly distributed of all state storage in OpenStack.

So, you are replacing the Nova DB (which doesn't need to be distributed 
at all, since it's a centralized control plane piece) within the regions 
with a "distributed" NoSQL store (and throwing away transactional safety 
I might add) but you're going with a non-distributed traditional RDBMS 
for the very piece that needs to be shared, distributed, and 
highly-available across OpenStack. I don't understand that.

 > At the
> same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
> OpenStack, except Keystone. We got something as follows at the end of
> the deployment:
>
> Admin Region (AR):
> - control:
>   * Horizon
>   * HAProxy
>   * Keyston
>   * MariaDB
>   * memcached

Again, that's not a region. Those are merely shared services between 
regions.

> OpenStack Region x (OSRx):
> - control:
>   * HAProxy
>   * nova-api/conductor/scheduler
>   * neutron-server/l3/dhcp/...
>   * glance-api/registry
>   * MariaDB
>   * RabbitMQ
>
> - compute1:
>   * nova-compute
>   * neutron-agent
>
> - compute2: ...
>
> We do the deployment by running Kolla n+1 times. The first run deploys
> the Administrative Region (AR) and the other runs deploy OpenStack
> Regions (OSR). For each run, we fix the value of `openstack_region_name'
> variable to the name of the current region.
>
> In the context of multi-regions, Keystone (in the AR) should be
> available to all OSRs. This means, there are as many Keystone
> endpoints as regions. For instance, if we consider two OSRs, the
> result of listing endpoints at the end of the AR deployment looks like
> this:
>
>
>  $ openstack endpoint list
>
>  | Region | Serv Name | Serv Type | Interface | URL                          |
>  |--------+-----------+-----------+-----------+------------------------------|
>  | AR     | keystone  | identity  | public    | http://10.24.63.248:5000/v3  |
>  | AR     | keystone  | identity  | internal  | http://10.24.63.248:5000/v3  |
>  | AR     | keystone  | identity  | admin     | http://10.24.63.248:35357/v3 |
>  | OSR1   | keystone  | identity  | public    | http://10.24.63.248:5000/v3  |
>  | OSR1   | keystone  | identity  | internal  | http://10.24.63.248:5000/v3  |
>  | OSR1   | keystone  | identity  | admin     | http://10.24.63.248:35357/v3 |
>  | OSR2   | keystone  | identity  | public    | http://10.24.63.248:5000/v3  |
>  | OSR2   | keystone  | identity  | internal  | http://10.24.63.248:5000/v3  |
>  | OSR2   | keystone  | identity  | admin     | http://10.24.63.248:35357/v3 |

There shouldn't be an AR region. If the Keystone authentication domain 
is indeed shared between OpenStack regions, then an administrative user 
should be able to hit any Keystone endpoint in any OpenStack region and 
add users/projects/roles, etc. to the shared Keystone data store (or if 
using LDAP, the admin should be able to add a user to 
ActiveDirectory/ApacheDS in any OpenStack region and have that user 
information immediately show up in any of the other regions).

Best,
-jay

>
> This requires patching the `keystone/tasks/register.yml' play[4] to
> re-execute the `Creating admin project, user, role, service, and
> endpoint' task for all regions we consider. An example of such a patch
> is given on our GitHub[5]. In this example, the `openstack_regions'
> variable is a list that contains the name of all regions (see [6]). As
> a drawback, the patch implies to know in advance all OSR. A better
> implementation would execute the `Creating admin project, user, role,
> service, and endpoint' task every time a new OSR is going to be
> deployed. But this requires to move the task somewhere else in the
> Kolla workflow and we have no idea where this should be.
>
> In the AR, we also have to change the Horizon configuration file to
> handle multi-regions[7]. The modification could be done easily and
> idiomatically by setting the `node_custome_config' variable to the
> `multi-regions' directory[8] and benefits from Kolla merging config
> system.
>
> Also, deploying OSRs requires patching the kolla-toolbox as it seems
> not region-aware. In particular, patch the `kolla_keystone_service.py'
> module[9] that is responsible for contacting Keystone and creating a
> new endpoint when we register a new OpenStack service.
>
>
>  73  for _endpoint in cloud.keystone_client.endpoints.list():
>  74      if _endpoint.service_id == service.id and \
>  75         _endpoint.interface == interface:
>  76        endpoint = _endpoint
>  77        if endpoint.url != url:
>  78          changed = True
>  79          cloud.keystone_client.endpoints.update(
>  80            endpoint, url=url)
>  81        break
>  82  else:
>  83    changed = True
>  84    cloud.keystone_client.endpoints.create(
>  85      service=service.id,
>  86      url=url,
>  87      interface=interface,
>  88      region=endpoint_region)
>
>
> At some point, this module /create/ or /update/ a service endpoint. It
> first tests if the service is already registered, and update the URL
> if so (l. 74 to 81), or create the new endpoint in other cases (l.
> 82). Unfortunately, the test (l. 74 to 75) only looks at the service
> (e.g., glance) and the interface (e.g., public). This makes the test
> wrong because we deploy the same service many times, but into
> different regions. We have to add an extra condition to not only test
> the service but also its region.
>
>
>  74  if _endpoint.service_id == service.id and \
>  75     _endpoint.interface == interface and \
>  76     _endpoint.region == endpoint_region:
>
>
> Finally, when we deployed OSRs, we override the value of
> `keystone_admin_url', `keystone_internal_url' and `keystone_public_url'
> to target Keystone in AR. We also have to change the
> `[keystone_authtoken]' section of nova/neutron/glance conf[10] so that
> it uses these variables instead of the canonical form with the
> `kolla_internal_fqdn' variable[11]. In this regards, is it due to some
> legacy code that configuration files use the `kolla_internal_fqdn'
> variable instead of `keystone_*_url'?
>
> That's almost all. As you can see, handle multi-regions implies a very
> small number of modifications if you call Kolla multiple times.
>
> So, thanks for the great job Kolla team! And waiting for your
> feedback.
>
>
>
> [1] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554
> [2] https://beyondtheclouds.github.io/
> [3] http://docs.openstack.org/arch-design/multi-site-architecture.html
> [4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
> [5] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
> [6] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
> [7] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
> [8] https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
> [9] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
> [10] https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
> [11] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155
>
>