<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace;font-size:small">Thanks <span style="font-size:12.8px;white-space:nowrap;font-family:arial,sans-serif">Ronan</span><span style="font-size:12.8px;white-space:nowrap;font-family:arial,sans-serif">, </span></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small">Great approach. </div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small">I am always expecting multi-region support in Kolla. I hope what you did can be merged into Kolla.</div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small"><br></div><div class="gmail_default" style="font-family:monospace,monospace;font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 5, 2017 at 10:12 PM, Ronan-Alexandre Cherrueau <span dir="ltr"><<a href="mailto:ronan-alexandre.cherrueau@inria.fr" target="_blank">ronan-alexandre.cherrueau@inria.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
<br>
TL;DR: We make a multi-regions deployment with Kolla. It requires to<br>
patch the code a little bit, and you can find the diff on our<br>
GitHub[1]. This patch is just a first attempt to support multi-regions<br>
in Kolla and it raises questions. Some modifications are not done in<br>
an idiomatic way and we do not expect this to be merged in Kolla. The<br>
reminder of this mail explains our patch and states our questions.<br>
<br>
<br>
At Inria/Discovery[2], we evaluate OpenStack at scale for the<br>
Performance Working Group. So far, we focus on one single OpenStack<br>
region deployment with hundreds of computes and we always go with<br>
Kolla for our deployment. Over the last few days, we tried to achieve<br>
a multi-regions OpenStack deployment with Kolla. We want to share with<br>
you our current deployment workflow, patches we had to apply on Kolla<br>
to support multi-regions, and also ask you if we do things correctly.<br>
<br>
First of all, our multi-regions deployment follows the one described<br>
by the OpenStack documentation[3]. Concretely, the deployment<br>
considers /one/ Administrative Region (AR) that contains Keystone and<br>
Horizon. This is a Kolla-based deployment, so Keystone is hidden<br>
behind an HAProxy, and has MariaDB and memcached as backend. At the<br>
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full<br>
OpenStack, except Keystone. We got something as follows at the end of<br>
the deployment:<br>
<br>
<br>
Admin Region (AR):<br>
- control:<br>
* Horizon<br>
* HAProxy<br>
* Keyston<br>
* MariaDB<br>
* memcached<br>
<br>
OpenStack Region x (OSRx):<br>
- control:<br>
* HAProxy<br>
* nova-api/conductor/scheduler<br>
* neutron-server/l3/dhcp/...<br>
* glance-api/registry<br>
* MariaDB<br>
* RabbitMQ<br>
<br>
- compute1:<br>
* nova-compute<br>
* neutron-agent<br>
<br>
- compute2: ...<br>
<br>
<br>
We do the deployment by running Kolla n+1 times. The first run deploys<br>
the Administrative Region (AR) and the other runs deploy OpenStack<br>
Regions (OSR). For each run, we fix the value of `openstack_region_name'<br>
variable to the name of the current region.<br>
<br>
In the context of multi-regions, Keystone (in the AR) should be<br>
available to all OSRs. This means, there are as many Keystone<br>
endpoints as regions. For instance, if we consider two OSRs, the<br>
result of listing endpoints at the end of the AR deployment looks like<br>
this:<br>
<br>
<br>
$ openstack endpoint list<br>
<br>
| Region | Serv Name | Serv Type | Interface | URL |<br>
|--------+-----------+--------<wbr>---+-----------+--------------<wbr>----------------|<br>
| AR | keystone | identity | public | <a href="http://10.24.63.248:5000/v3" rel="noreferrer" target="_blank">http://10.24.63.248:5000/v3</a> |<br>
| AR | keystone | identity | internal | <a href="http://10.24.63.248:5000/v3" rel="noreferrer" target="_blank">http://10.24.63.248:5000/v3</a> |<br>
| AR | keystone | identity | admin | <a href="http://10.24.63.248:35357/v3" rel="noreferrer" target="_blank">http://10.24.63.248:35357/v3</a> |<br>
| OSR1 | keystone | identity | public | <a href="http://10.24.63.248:5000/v3" rel="noreferrer" target="_blank">http://10.24.63.248:5000/v3</a> |<br>
| OSR1 | keystone | identity | internal | <a href="http://10.24.63.248:5000/v3" rel="noreferrer" target="_blank">http://10.24.63.248:5000/v3</a> |<br>
| OSR1 | keystone | identity | admin | <a href="http://10.24.63.248:35357/v3" rel="noreferrer" target="_blank">http://10.24.63.248:35357/v3</a> |<br>
| OSR2 | keystone | identity | public | <a href="http://10.24.63.248:5000/v3" rel="noreferrer" target="_blank">http://10.24.63.248:5000/v3</a> |<br>
| OSR2 | keystone | identity | internal | <a href="http://10.24.63.248:5000/v3" rel="noreferrer" target="_blank">http://10.24.63.248:5000/v3</a> |<br>
| OSR2 | keystone | identity | admin | <a href="http://10.24.63.248:35357/v3" rel="noreferrer" target="_blank">http://10.24.63.248:35357/v3</a> |<br>
<br>
<br>
This requires patching the `keystone/tasks/register.yml' play[4] to<br>
re-execute the `Creating admin project, user, role, service, and<br>
endpoint' task for all regions we consider. An example of such a patch<br>
is given on our GitHub[5]. In this example, the `openstack_regions'<br>
variable is a list that contains the name of all regions (see [6]). As<br>
a drawback, the patch implies to know in advance all OSR. A better<br>
implementation would execute the `Creating admin project, user, role,<br>
service, and endpoint' task every time a new OSR is going to be<br>
deployed. But this requires to move the task somewhere else in the<br>
Kolla workflow and we have no idea where this should be.<br>
<br>
In the AR, we also have to change the Horizon configuration file to<br>
handle multi-regions[7]. The modification could be done easily and<br>
idiomatically by setting the `node_custome_config' variable to the<br>
`multi-regions' directory[8] and benefits from Kolla merging config<br>
system.<br>
<br>
Also, deploying OSRs requires patching the kolla-toolbox as it seems<br>
not region-aware. In particular, patch the `kolla_keystone_service.py'<br>
module[9] that is responsible for contacting Keystone and creating a<br>
new endpoint when we register a new OpenStack service.<br>
<br>
<br>
73 for _endpoint in cloud.keystone_client.<wbr>endpoints.list():<br>
74 if _endpoint.service_id == <a href="http://service.id" rel="noreferrer" target="_blank">service.id</a> and \<br>
75 _endpoint.interface == interface:<br>
76 endpoint = _endpoint<br>
77 if endpoint.url != url:<br>
78 changed = True<br>
79 cloud.keystone_client.<wbr>endpoints.update(<br>
80 endpoint, url=url)<br>
81 break<br>
82 else:<br>
83 changed = True<br>
84 cloud.keystone_client.<wbr>endpoints.create(<br>
85 service=<a href="http://service.id" rel="noreferrer" target="_blank">service.id</a>,<br>
86 url=url,<br>
87 interface=interface,<br>
88 region=endpoint_region)<br>
<br>
<br>
At some point, this module /create/ or /update/ a service endpoint. It<br>
first tests if the service is already registered, and update the URL<br>
if so (l. 74 to 81), or create the new endpoint in other cases (l.<br>
82). Unfortunately, the test (l. 74 to 75) only looks at the service<br>
(e.g., glance) and the interface (e.g., public). This makes the test<br>
wrong because we deploy the same service many times, but into<br>
different regions. We have to add an extra condition to not only test<br>
the service but also its region.<br>
<br>
<br>
74 if _endpoint.service_id == <a href="http://service.id" rel="noreferrer" target="_blank">service.id</a> and \<br>
75 _endpoint.interface == interface and \<br>
76 _endpoint.region == endpoint_region:<br>
<br>
<br>
Finally, when we deployed OSRs, we override the value of<br>
`keystone_admin_url', `keystone_internal_url' and `keystone_public_url'<br>
to target Keystone in AR. We also have to change the<br>
`[keystone_authtoken]' section of nova/neutron/glance conf[10] so that<br>
it uses these variables instead of the canonical form with the<br>
`kolla_internal_fqdn' variable[11]. In this regards, is it due to some<br>
legacy code that configuration files use the `kolla_internal_fqdn'<br>
variable instead of `keystone_*_url'?<br>
<br>
That's almost all. As you can see, handle multi-regions implies a very<br>
small number of modifications if you call Kolla multiple times.<br>
<br>
So, thanks for the great job Kolla team! And waiting for your<br>
feedback.<br>
<br>
<br>
<br>
[1] <a href="https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554" rel="noreferrer" target="_blank">https://github.com/<wbr>BeyondTheClouds/kolla/commit/<wbr>7e5f7e6c7936b7cc3136dfc082935e<wbr>2995a65554</a><br>
[2] <a href="https://beyondtheclouds.github.io/" rel="noreferrer" target="_blank">https://beyondtheclouds.<wbr>github.io/</a><br>
[3] <a href="http://docs.openstack.org/arch-design/multi-site-architecture.html" rel="noreferrer" target="_blank">http://docs.openstack.org/<wbr>arch-design/multi-site-<wbr>architecture.html</a><br>
[4] <a href="https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>kolla/blob/<wbr>11bfd9eb7518cc46ac441505a267f5<wbr>cf974216ae/ansible/roles/<wbr>keystone/tasks/register.yml</a><br>
[5] <a href="https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1" rel="noreferrer" target="_blank">https://github.com/<wbr>BeyondTheClouds/kolla/commit/<wbr>7e5f7e6c7936b7cc3136dfc082935e<wbr>2995a65554#diff-<wbr>40be75c3a2237adfd1a05178f6f600<wbr>06R1</a><br>
[6] <a href="https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215" rel="noreferrer" target="_blank">https://github.com/<wbr>BeyondTheClouds/kolla/commit/<wbr>7e5f7e6c7936b7cc3136dfc082935e<wbr>2995a65554#diff-<wbr>607e6e5925d0031dafa79eb80d1986<wbr>40R215</a><br>
[7] <a href="https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188" rel="noreferrer" target="_blank">https://github.com/<wbr>BeyondTheClouds/kolla/commit/<wbr>7e5f7e6c7936b7cc3136dfc082935e<wbr>2995a65554#diff-<wbr>6cbb63bb9c4843bcf8db4710e58847<wbr>5bR188</a><br>
[8] <a href="https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions" rel="noreferrer" target="_blank">https://github.com/<wbr>BeyondTheClouds/kolla/tree/<wbr>7e5f7e6c7936b7cc3136dfc082935e<wbr>2995a65554/multi-regions</a><br>
[9] <a href="https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72" rel="noreferrer" target="_blank">https://github.com/<wbr>BeyondTheClouds/kolla/commit/<wbr>7e5f7e6c7936b7cc3136dfc082935e<wbr>2995a65554#diff-<wbr>ba10dd4575f65e03d50d586febdccb<wbr>adR72</a><br>
[10] <a href="https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf" rel="noreferrer" target="_blank">https://github.com/<wbr>BeyondTheClouds/kolla/blob/<wbr>7e5f7e6c7936b7cc3136dfc082935e<wbr>2995a65554/multi-regions/<wbr>global.conf</a><br>
[11] <a href="https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>kolla/blob/<wbr>11bfd9eb7518cc46ac441505a267f5<wbr>cf974216ae/ansible/roles/nova/<wbr>templates/nova.conf.j2#L155</a><br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
Ronan-A. Cherrueau<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><span style="font-size:13px;border-collapse:collapse"><font face="monospace, monospace">Regards,</font></span></div><div><span style="font-size:13px;border-collapse:collapse"><font face="monospace, monospace">Jeffrey Zhang</font></span></div><div><span style="font-family:monospace,monospace;font-size:12.8px">Blog: </span><a href="http://xcodest.me/" style="font-family:monospace,monospace;font-size:12.8px" target="_blank">http://xcodest.me</a><font face="monospace, monospace"><br></font></div></div></div></div></div></div></div></div></div>
</div>