<div dir="ltr">Hi -<div><br></div><div>Great suggestions, Dan.</div><div><br></div><div>To recap, we followed that up with a few other ideas on irc and we eventually came to a point to test some of this, with slight modification.</div><div><br></div><div>UI also ships with a configuration file that can override the endpoint information received from Keystone. The file is located at /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js.</div><div><br></div><div>Part of making this work means enabling SSL on the Undercloud when the UI component is selected for installation in undercloud.conf. I think this is going to be a pretty reasonable request, but I'm interested in hearing feedback from this, and what other implications it may have, that I can't think of. The changes I made were all one-off, unmanaged changes, just to test this idea out. I'll be doing some more tests but will probably be looking for acceptance shortly. </div><div><br></div><div>Once SSL was enabled on the Undercloud, I made two edits to haproxy.cfg that were pretty straightforward: added a 'listen' server directive for UI to both terminate SSL and forward the request to Apache, and added a 'bind' statement for each service that UI expects to talk to (keystone, heat, ironic, mistral, swift, zaqar-websocket).</div><div><br></div><div>Once those configuration changes were made, I had a very pleasant experience using the UI. It worked exactly as expected. I think this might be a viable option.</div><div><br></div><div>Thoughts?</div><div><br></div><div>Thanks!</div><div>-dant</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 30, 2016 at 12:21 PM, Dan Sneddon <span dir="ltr"><<a href="mailto:dsneddon@redhat.com" target="_blank">dsneddon@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>Thinking about this a little more, creating a new unified endpoint on the same port as the UI doesn't solve the problem at hand. The UI will use the service catalog to find endpoints, so we would need to change the endpoints in the service catalog, which changes the flow for the underlying services as well.</div><div><br></div><div>Simply moving the control plane services to the external network won't work well in environments where the control plane is isolated and non-routed. The Undercloud can forward packets, but then becomes a bottleneck and a SPOF. </div><div><br></div><div>A few approaches come to mind, but none of these are quick fixes:</div><div><br></div><div>* Change the UI to get its list of endpoints from somewhere other than the service catalog and customize this with URLs that point to the Public VIP. Duplicate the services required for the UI on both control plane and external network. This might make it possible to make all connections over port 443, which is more firewall-friendly (which would be desirable or not depending on what kind of firewalling and traffic management is wanted).</div><div><br></div><div>* Relax the rp_filter settings on the Controllers so they accept packets destined for the external network on their control plane interfaces; add a static route to the Public VIP via the control plane VIP on all non-controller nodes. Modify the service catalog to point to the public VIP for the services the UI needs. This would need to be combined with a security review to determine if additional iptables rules are required. </div><div><br></div><div><span style="background-color:rgba(255,255,255,0)">* Split the service catalog, so we have an internal and an external view depending on where the query came from. I'm not sure how feasible this is.</span></div><div><span style="background-color:rgba(255,255,255,0)"><br></span></div><div><span style="background-color:rgba(255,255,255,0)">Of these, I think the rp_filter settings are the only ones that could be done solely with TripleO code changes. That might be worth investigating.</span></div><span class=""><div><br><div><blockquote type="cite"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="background-color:rgba(255,255,255,0)">Dan Sneddon | Principal OpenStack Engineer | </span><a href="mailto:dsneddon@redhat.com" style="font-size:13pt;background-color:rgba(255,255,255,0)" target="_blank">dsneddon@redhat.com</a></blockquote></div></div></blockquote></div></div></span><div><div class="h5"><div><br>On Sep 30, 2016, at 11:36 AM, Dan Sneddon <<a href="mailto:dsneddon@redhat.com" target="_blank">dsneddon@redhat.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div>I don't think we can rely on the Undercloud as an API proxy unless we address the lack of HA on the Undercloud. </div><div><br></div><div>Wouldn't this be better implemented as as a single, name-based HAProxy instance running SSL on port 443 on the overcloud Public VIP? Then we could have the same endpoint for Horizon and every other API. </div><div><br></div><div>I actually implemented this scheme in Havana before I joined Red Hat. At the time, we had to have a complex HAProxy config and patch the end points to support name-based URLs. I think some work has been done in OpenStack now to support this model, but I'm not sure where it stands. <br><br><blockquote type="cite"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="background-color:rgba(255,255,255,0)">Dan Sneddon | Principal OpenStack Engineer | </span><a href="mailto:dsneddon@redhat.com" style="font-size:13pt;background-color:rgba(255,255,255,0)" target="_blank">dsneddon@redhat.com</a></blockquote></div></div></blockquote></div><div><br>On Sep 30, 2016, at 9:44 AM, Dan Trainor <<a href="mailto:dtrainor@redhat.com" target="_blank">dtrainor@redhat.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Hi -<div><br></div><div>I re-read your email a few times and like a few of the things that I see, but I'd love some more clarification. As I read it, a few of these things conflict. I believe you're suggesting that we don't make these services listen on a public interface due to security concerns (and of course, enabling SSL would break this because haproxy would listen on these interfaces/ports), but this approach would be acceptable if HAProxy was listening on these ports, terminating SSL, and sending them to each respective service backend. Am I correct i understanding this?</div><div><br></div><div>Are you suggesting that these endpoint ports would still be externally accessible on the primary (public) interface of the Undercloud, but just managed by HAProxy? I think that's an acceptable approach. Even if these endpoints are, like you suggested, listening on a separate network or IP as the Undercloud's primary interface, at least then it would be easier for organizations to enforce network access policies to these ports, and subsequently, these services that UI needs to talk to directly.</div><div><br></div><div>I'm also perfectly fine with suggesting that if UI is installed, then this forces the Undercloud to be SSL enabled. This would be a good way to move the idea of a secured, by default SSL-enabled Undercloud forward a little more, which is something we'd definitely like to see more.</div><div><br></div><div>Thoughts?</div><div><br></div><div>Thanks</div><div>-dant</div><div><br></div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 29, 2016 at 9:01 AM, Dan Trainor <span dir="ltr"><<a href="mailto:dtrainor@redhat.com" target="_blank">dtrainor@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi, Juan -<div><br></div><div><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Actually, the third option is also not an option in the current undercloud setup, since making the services listen in 0.0.0.0 will break HAProxy. So when you're deploying with TLS things will break since we use HAProxy to terminate TLS connections. </div></div></div></blockquote><div><br></div></span><div>Ah, that's correct, isn't it. </div><span><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>On the other hand, we also don't want services to listen on 0.0.0.0 since that would become a security concern. We should instead be blocking everything we don't need to have exposed (as we've done with the undercloud's database and rabbitmq).<br></div></div></div></blockquote><div><br></div></span><div>I don't disagree that we want to focus on least privilege, but we do have documentation that talks about how to deal with this. I addressed this in my previous message, even if only to illustrate my understanding that there would be concerns around this. See more comments about this down below...</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div></div><br></div>Now, as we're trying to move to have more convergence between the undercloud and the overcloud (trying to deploy the undercloud via heat also, as Dan Prince has mentioned), I think some aspecs of this will bring a solution to this problem. for instance, just like we already do in the overcloud, we could have the undercloud's HAProxy always terminate the endpoints, which I'm attempting with these two patches: <a href="https://review.openstack.org/#/c/360366" target="_blank">https://review.openstack.org/#<wbr>/c/360366</a> <a href="https://review.openstack.org/#/c/360368" target="_blank">https://review.openstack.org/#<wbr>/c/360368</a> . Furthermore, we could have the public endpoints in HAProxy listen on a separate network that's accessible externally, also as we do in the overcloud. That way we don't need to change the actual interfaces the services are listening on, and rely on HAProxy, getting closer to how we do things in the overcloud. It seems to me that it would help solve the problem.<br></div></blockquote><div><br></div></span><div>I like that idea. Though, this effectively shifts the process of having these services themselves listen on different IPs/ports and offloads that to HAProxy. Whatever security concerns we have with opening up network communications would still exist, but I think that would be more broadly accepted considering these connections are now managed by HAProxy which *only* listens for SSL connections. </div><div><br></div><div>Another challenge with isolating this traffic on a separate network is that we introduce a new dependency that's going to take administrative time to set up and configure outside of OpenStack - do we really want that?</div><div><br></div><div>Thanks!<br>-dant</div><div><div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Wed, Sep 28, 2016 at 7:24 PM, Dan Trainor <span dir="ltr"><<a href="mailto:dtrainor@redhat.com" target="_blank">dtrainor@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi -<div><br></div><div><div style="font-size:12.8px">I want to bring up a subject that needs a little more attention. There are a few ideas floating around but it's important that we get this done right. </div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">UI is unique in the sense that it operates almost entirely in a browser, talking directly to service API endpoints which it either figures out from they Keystone service catalog as using the publicURL endpoint for each service, or by specifying these API endpoints in a configuration file. Though overriding the API endpoints in the UI's local configuration file is an option that's available, I understand that we want to move towards relying exclusively on Keystone for accurate and correct endpoint configuration.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Right now, all of the service API endpoints that UI needs to talk with are only listening on the ctlplane network.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">We've had several iterations of testing and development of the UI over time and as a result of that, three different solutions that work - depending on the exact circumstances - have been created which all can achieve the same goal of allowing the UI to talk to these endpoints:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">- Local SSH port tunneling of the required ports that UI talks to, from the system running the UI to the Undercloud, and configuring the UI to talk to localhost:NNNN. This method "works", but it's not a solution we can recommend</div><div style="font-size:12.8px">- Making the interface on which these services already listen on - the ctlplane network - routable. Again, this method "works", but we treat this interface in a very special manner on purpose, not the least of which because of it's ability to facilitate pxebooting</div><div style="font-size:12.8px">- Change the public endpoints in the Keystone catalog to be that of the existing external, routable interface of the Undercloud for each service required by the UI. This also requires configuring each service that UI needs to talk with, to listen on the existing, external, routable interface on the Undercloud. Some services support a list of interfaces and IPs to listen on; others require exactly one argument, in which case the address of 0.0.0.0 would need to be used</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">According to the API Endpoint Configuration Recommendation guide[1], the third option seems most viable and one that we can recommend. The document briefly describes the security implications of having these services open on a public interface but recommends the use of a stringent network policy - something we're used to recommending and helping with. The first two options, not so much. </div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Based on discussions I've had with other people, it's my impression that the third option is likely the one that we should proceed with. </div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">This concern is largely transparent to how we're currently testing and developing the UI because most of that work is done on local, virtualized environments. When this happens, libvirt does the heavy lifting of creating a network that's easily routable from the host system. If not that, then the evolution of instructions for setting up these local environments over time have recommended using SSH port forwarding. However, neither of these options should be recommended.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Thoughts?</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Thanks</div><div style="font-size:12.8px">-dant</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">--</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">P.S. and full disclosure: I'm biased towards the third option.</div><div><br></div></div>
<br></div></div>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><font style="font-family:arial narrow,sans-serif;color:rgb(102,102,102)" size="2">Juan Antonio Osorio R.<br>e-mail: <a href="mailto:jaosorior@gmail.com" target="_blank">jaosorior@gmail.com</a><br></font><font style="font-family:arial narrow,sans-serif;color:rgb(102,102,102)" size="2"><br></font></div></div></div>
</font></span></div></div></div></div></div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br></blockquote></div></div></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</div></blockquote><blockquote type="cite"><div><span>______________________________<wbr>______________________________<wbr>______________</span><br><span>OpenStack Development Mailing List (not for usage questions)</span><br><span>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org</a>?subject:<wbr>unsubscribe</span><br><span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a></span><br></div></blockquote></div></blockquote></div></div></div><br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>