<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, 10 Oct 2018 at 08:08, Florian Engelmann <<a href="mailto:florian.engelmann@everyware.ch">florian.engelmann@everyware.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Am 10/9/18 um 1:47 PM schrieb Mark Goddard:<br>
> <br>
> <br>
> On Tue, 9 Oct 2018 at 12:03, Florian Engelmann <br>
> <<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a> <mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a>>> <br>
> wrote:<br>
> <br>
> Am 10/9/18 um 11:04 AM schrieb Mark Goddard:<br>
> > Thanks for these suggestions Florian, there are some interesting<br>
> ideas<br>
> > in here. I'm a little concerned about the maintenance overhead of<br>
> adding<br>
> > support for all of these things, and wonder if some of them could be<br>
> > done without explicit support in kolla and kolla-ansible. The kolla<br>
> > projects have been able to move quickly by providing a flexible<br>
> > configuration mechanism that avoids the need to maintain support for<br>
> > every OpenStack feature. Other thoughts inline.<br>
> ><br>
> <br>
> I do understand your apprehensions Mark. For some of the suggested<br>
> changes/additions I agree. But adding those components without<br>
> kolla/kolla-ansible integration feels not right.<br>
> <br>
> <br>
> I'm not entirely against adding some of these things, if enough people <br>
> in the community want them. I'd just like to make sure that if they can <br>
> be done in a sane way without changes, then we do that and document how <br>
> to do it instead.<br>
> <br>
<br>
Yes I agree and that is a very important role/job.<br>
<br>
> <br>
> ><br>
> > On Mon, 8 Oct 2018 at 11:15, Florian Engelmann<br>
> > <<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a><br>
> <mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a>><br>
> <mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a><br>
> <mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a>>>><br>
> > wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> > I would like to start a discussion about some changes and<br>
> additions I<br>
> > would like to see in in kolla and kolla-ansible.<br>
> ><br>
> > 1. Keepalived is a problem in layer3 spine leaf networks as any<br>
> > floating<br>
> > IP can only exist in one leaf (and VRRP is a problem in<br>
> layer3). I<br>
> > would<br>
> > like to use consul and registrar to get rid of the "internal"<br>
> floating<br>
> > IP and use consuls DNS service discovery to connect all<br>
> services with<br>
> > each other.<br>
> ><br>
> ><br>
> > Without reading up, I'm not sure exactly how this fits together. If<br>
> > kolla-ansible made the API host configurable for each service rather<br>
> > than globally, would that be a step in the right direction?<br>
> <br>
> No that would not help. The problem is HA. Right now there is a<br>
> "central" floating IP (kolla_internal_vip_address) that is used for all<br>
> services to connect to (each other). Keepalived is failing that IP over<br>
> if the "active" host fails. In a layer3 (CLOS/Spine-Leaf) network this<br>
> IP is only available in one leaf/rack. So that rack is a "SPOF".<br>
> Using service discovery fits perfect in a CLOS network and scales much<br>
> better as a HA solution.<br>
> <br>
> Right, but what I'm saying as a thought experiment is, if we gave you <br>
> the required variables in kolla-ansible (e.g. nova_internal_fqdn) to <br>
> make this possible with an externally managed consul service, could that <br>
> work?<br>
<br>
Now I get you. I would say all configuration templates need to be <br>
changed to allow, eg.<br>
<br>
$ grep http /etc/kolla/cinder-volume/cinder.conf<br>
glance_api_servers = <a href="http://10.10.10.5:9292" rel="noreferrer" target="_blank">http://10.10.10.5:9292</a><br>
auth_url = <a href="http://internal.somedomain.tld:35357" rel="noreferrer" target="_blank">http://internal.somedomain.tld:35357</a><br>
www_authenticate_uri = <a href="http://internal.somedomain.tld:5000" rel="noreferrer" target="_blank">http://internal.somedomain.tld:5000</a><br>
auth_url = <a href="http://internal.somedomain.tld:35357" rel="noreferrer" target="_blank">http://internal.somedomain.tld:35357</a><br>
auth_endpoint = <a href="http://internal.somedomain.tld:5000" rel="noreferrer" target="_blank">http://internal.somedomain.tld:5000</a><br>
<br>
to look like:<br>
<br>
glance_api_servers = <a href="http://glance.service.somedomain.consul:9292" rel="noreferrer" target="_blank">http://glance.service.somedomain.consul:9292</a><br>
auth_url = <a href="http://keystone.service.somedomain.consul:35357" rel="noreferrer" target="_blank">http://keystone.service.somedomain.consul:35357</a><br>
www_authenticate_uri = <a href="http://keystone.service.somedomain.consul:5000" rel="noreferrer" target="_blank">http://keystone.service.somedomain.consul:5000</a><br>
auth_url = <a href="http://keystone.service.somedomain.consul:35357" rel="noreferrer" target="_blank">http://keystone.service.somedomain.consul:35357</a><br>
auth_endpoint = <a href="http://keystone.service.somedomain.consul:5000" rel="noreferrer" target="_blank">http://keystone.service.somedomain.consul:5000</a></blockquote><div><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Those values are generally formed using the internal FQDN (kolla_internal_fqdn), that is supposed to resolve to the internal VIP.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">If we had something like this in group_vars/all.yml, we could make the host configurable on a per-service basis, while still retaining the default. This is a common pattern in kolla-ansible. For example:</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">nova_external_fqdn: "{{ kolla_external_fqdn }}"</div><div class="gmail_default" style="font-family:verdana,sans-serif">nova_internal_fqdn: "{{ kolla_internal_fqdn }}"<br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> <br>
> ><br>
> ><br>
> > 2. Using "ports" for external API (endpoint) access is a<br>
> major headache<br>
> > if a firewall is involved. I would like to configure the<br>
> HAProxy (or<br>
> > fabio) for the external access to use "Host:" like, eg. "Host:<br>
> > keystone.somedomain.tld", "Host: nova.somedomain.tld", ...<br>
> with HTTPS.<br>
> > Any customer would just need HTTPS access and not have to<br>
> open all<br>
> > those<br>
> > ports in his firewall. For some enterprise customers it is<br>
> not possible<br>
> > to request FW changes like that.<br>
> ><br>
> > 3. HAProxy is not capable to handle "read/write" split with<br>
> Galera. I<br>
> > would like to introduce ProxySQL to be able to scale Galera.<br>
> ><br>
> ><br>
> > It's now possible to use an external database server with<br>
> kolla-ansible,<br>
> > instead of deploying a mariadb/galera cluster. This could be<br>
> implemented<br>
> > how you like, see<br>
> ><br>
> <a href="https://docs.openstack.org/kolla-ansible/latest/reference/external-mariadb-guide.html" rel="noreferrer" target="_blank">https://docs.openstack.org/kolla-ansible/latest/reference/external-mariadb-guide.html</a>.<br>
> <br>
> Yes I agree. And this is what we will do in our first production<br>
> deployment. But I would love to see ProxySQL in Kolla as well.<br>
> As a side note: Kolla-ansible does use:<br>
> <br>
> option mysql-check user haproxy post-41<br>
> <br>
> to check Galera, but that check does not fail if the node is out of<br>
> sync<br>
> with the other nodes!<br>
> <br>
> <a href="http://galeracluster.com/documentation-webpages/monitoringthecluster.html" rel="noreferrer" target="_blank">http://galeracluster.com/documentation-webpages/monitoringthecluster.html</a><br>
> <br>
> That's good to know. Could you raise a bug in kolla-ansible on <br>
> launchpad, and offer advice on how to improve this check if you have any?<br>
> <br>
<br>
done: <a href="https://bugs.launchpad.net/kolla-ansible/+bug/1796930" rel="noreferrer" target="_blank">https://bugs.launchpad.net/kolla-ansible/+bug/1796930</a><br>
<br></blockquote><div><span class="gmail_default" style="font-family:verdana,sans-serif">Thanks</span> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> <br>
> ><br>
> > 4. HAProxy is fine but fabio integrates well with consul,<br>
> statsd and<br>
> > could be connected to a vault cluster to manage secure<br>
> certificate<br>
> > access.<br>
> ><br>
> > As above.<br>
> ><br>
> > 5. I would like to add vault as Barbican backend.<br>
> ><br>
> > Does this need explicit support in kolla and kolla-ansible, or<br>
> could it<br>
> > be done through configuration of barbican.conf? Are there additional<br>
> > packages required in the barbican container? If so, see<br>
> ><br>
> <a href="https://docs.openstack.org/kolla/latest/admin/image-building.html#package-customisation" rel="noreferrer" target="_blank">https://docs.openstack.org/kolla/latest/admin/image-building.html#package-customisation</a>.<br>
> <br>
> True but the vault (and consul) containers could be deployed and<br>
> managed<br>
> by kolla-ansible.<br>
> <br>
> I'd like to see if anyone else is interested in this. Kolla ansible <br>
> already deploys a large number of services, which is great. As with many <br>
> other projects I'm seeing the resources of core contributors fall off a <br>
> little, and I think we need to consider how to ensure the project is <br>
> maintainable long term. In my view a good way of doing that is to enable <br>
> integration with existing services, rather than deploying them. We need <br>
> to decide where the line is as a community. We have an IRC meeting at <br>
> 3pm UTC if you'd like to bring it up then.<br>
<br>
<br>
Sorry I didn't manage to attend the IRC meeting. But again, I aggree, it <br>
would be great to extend the (already great) felxibility of <br>
kolla-ansible to be "add" "external" services.<br>
<br></blockquote><div><span class="gmail_default" style="font-family:verdana,sans-serif">Sorry, I missed a key word here</span> <span class="gmail_default" style="font-family:verdana,sans-serif">- tomorrow. It is now today, i.e. Wednesdays at 3pm UTC.</span></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> <br>
> ><br>
> > 6. I would like to add an option to enable tokenless<br>
> authentication for<br>
> > all services with each other to get rid of all the openstack<br>
> service<br>
> > passwords (security issue).<br>
> ><br>
> > Again, could this be done without explicit support?<br>
> <br>
> We did not investigate here. Changes to the apache configuration are<br>
> needed. I guess we will have to change the kolla container itself to do<br>
> so? Is it possible to "inject" files in a container using kolla-ansible?<br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> <br>
> <br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> <br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div>