Open Stack

Do you know how awesome it would be if you put this idea into a Blueprint at
http://blueprints.launchpad.net? That would be super-awesome.
File it under tripleo-ui project here if you have a few minutes:

https://blueprints.launchpad.net/specs/+new

----- Original Message -----
> Hi -
> 
> Great suggestions, Dan.
> 
> To recap, we followed that up with a few other ideas on irc and we eventually
> came to a point to test some of this, with slight modification.
> 
> UI also ships with a configuration file that can override the endpoint
> information received from Keystone. The file is located at
> /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js.
> 
> Part of making this work means enabling SSL on the Undercloud when the UI
> component is selected for installation in undercloud.conf. I think this is
> going to be a pretty reasonable request, but I'm interested in hearing
> feedback from this, and what other implications it may have, that I can't
> think of. The changes I made were all one-off, unmanaged changes, just to
> test this idea out. I'll be doing some more tests but will probably be
> looking for acceptance shortly.
> 
> Once SSL was enabled on the Undercloud, I made two edits to haproxy.cfg that
> were pretty straightforward: added a 'listen' server directive for UI to
> both terminate SSL and forward the request to Apache, and added a 'bind'
> statement for each service that UI expects to talk to (keystone, heat,
> ironic, mistral, swift, zaqar-websocket).
> 
> Once those configuration changes were made, I had a very pleasant experience
> using the UI. It worked exactly as expected. I think this might be a viable
> option.
> 
> Thoughts?
> 
> Thanks!
> -dant
> 
> 
> 
> 
> 
> 
> On Fri, Sep 30, 2016 at 12:21 PM, Dan Sneddon < dsneddon at redhat.com > wrote:
> 
> 
> 
> Thinking about this a little more, creating a new unified endpoint on the
> same port as the UI doesn't solve the problem at hand. The UI will use the
> service catalog to find endpoints, so we would need to change the endpoints
> in the service catalog, which changes the flow for the underlying services
> as well.
> 
> Simply moving the control plane services to the external network won't work
> well in environments where the control plane is isolated and non-routed. The
> Undercloud can forward packets, but then becomes a bottleneck and a SPOF.
> 
> A few approaches come to mind, but none of these are quick fixes:
> 
> * Change the UI to get its list of endpoints from somewhere other than the
> service catalog and customize this with URLs that point to the Public VIP.
> Duplicate the services required for the UI on both control plane and
> external network. This might make it possible to make all connections over
> port 443, which is more firewall-friendly (which would be desirable or not
> depending on what kind of firewalling and traffic management is wanted).
> 
> * Relax the rp_filter settings on the Controllers so they accept packets
> destined for the external network on their control plane interfaces; add a
> static route to the Public VIP via the control plane VIP on all
> non-controller nodes. Modify the service catalog to point to the public VIP
> for the services the UI needs. This would need to be combined with a
> security review to determine if additional iptables rules are required.
> 
> * Split the service catalog, so we have an internal and an external view
> depending on where the query came from. I'm not sure how feasible this is.
> 
> Of these, I think the rp_filter settings are the only ones that could be done
> solely with TripleO code changes. That might be worth investigating.
> 
> 
> 
> 
> 
> 
> Dan Sneddon | Principal OpenStack Engineer | dsneddon at redhat.com
> 
> On Sep 30, 2016, at 11:36 AM, Dan Sneddon < dsneddon at redhat.com > wrote:
> 
> 
> 
> 
> I don't think we can rely on the Undercloud as an API proxy unless we address
> the lack of HA on the Undercloud.
> 
> Wouldn't this be better implemented as as a single, name-based HAProxy
> instance running SSL on port 443 on the overcloud Public VIP? Then we could
> have the same endpoint for Horizon and every other API.
> 
> I actually implemented this scheme in Havana before I joined Red Hat. At the
> time, we had to have a complex HAProxy config and patch the end points to
> support name-based URLs. I think some work has been done in OpenStack now to
> support this model, but I'm not sure where it stands.
> 
> 
> 
> 
> 
> 
> Dan Sneddon | Principal OpenStack Engineer | dsneddon at redhat.com
> 
> On Sep 30, 2016, at 9:44 AM, Dan Trainor < dtrainor at redhat.com > wrote:
> 
> 
> 
> 
> Hi -
> 
> I re-read your email a few times and like a few of the things that I see, but
> I'd love some more clarification. As I read it, a few of these things
> conflict. I believe you're suggesting that we don't make these services
> listen on a public interface due to security concerns (and of course,
> enabling SSL would break this because haproxy would listen on these
> interfaces/ports), but this approach would be acceptable if HAProxy was
> listening on these ports, terminating SSL, and sending them to each
> respective service backend. Am I correct i understanding this?
> 
> Are you suggesting that these endpoint ports would still be externally
> accessible on the primary (public) interface of the Undercloud, but just
> managed by HAProxy? I think that's an acceptable approach. Even if these
> endpoints are, like you suggested, listening on a separate network or IP as
> the Undercloud's primary interface, at least then it would be easier for
> organizations to enforce network access policies to these ports, and
> subsequently, these services that UI needs to talk to directly.
> 
> I'm also perfectly fine with suggesting that if UI is installed, then this
> forces the Undercloud to be SSL enabled. This would be a good way to move
> the idea of a secured, by default SSL-enabled Undercloud forward a little
> more, which is something we'd definitely like to see more.
> 
> Thoughts?
> 
> Thanks
> -dant
> 
> 
> 
> On Thu, Sep 29, 2016 at 9:01 AM, Dan Trainor < dtrainor at redhat.com > wrote:
> 
> 
> 
> Hi, Juan -
> 
> 
> 
> 
> Actually, the third option is also not an option in the current undercloud
> setup, since making the services listen in 0.0.0.0 will break HAProxy. So
> when you're deploying with TLS things will break since we use HAProxy to
> terminate TLS connections.
> 
> Ah, that's correct, isn't it.
> 
> 
> 
> 
> On the other hand, we also don't want services to listen on 0.0.0.0 since
> that would become a security concern. We should instead be blocking
> everything we don't need to have exposed (as we've done with the
> undercloud's database and rabbitmq).
> 
> I don't disagree that we want to focus on least privilege, but we do have
> documentation that talks about how to deal with this. I addressed this in my
> previous message, even if only to illustrate my understanding that there
> would be concerns around this. See more comments about this down below...
> 
> 
> 
> 
> Now, as we're trying to move to have more convergence between the undercloud
> and the overcloud (trying to deploy the undercloud via heat also, as Dan
> Prince has mentioned), I think some aspecs of this will bring a solution to
> this problem. for instance, just like we already do in the overcloud, we
> could have the undercloud's HAProxy always terminate the endpoints, which
> I'm attempting with these two patches:
> https://review.openstack.org/#/c/360366
> https://review.openstack.org/#/c/360368 . Furthermore, we could have the
> public endpoints in HAProxy listen on a separate network that's accessible
> externally, also as we do in the overcloud. That way we don't need to change
> the actual interfaces the services are listening on, and rely on HAProxy,
> getting closer to how we do things in the overcloud. It seems to me that it
> would help solve the problem.
> 
> I like that idea. Though, this effectively shifts the process of having these
> services themselves listen on different IPs/ports and offloads that to
> HAProxy. Whatever security concerns we have with opening up network
> communications would still exist, but I think that would be more broadly
> accepted considering these connections are now managed by HAProxy which
> *only* listens for SSL connections.
> 
> Another challenge with isolating this traffic on a separate network is that
> we introduce a new dependency that's going to take administrative time to
> set up and configure outside of OpenStack - do we really want that?
> 
> Thanks!
> -dant
> 
> 
> 
> 
> 
> 
> On Wed, Sep 28, 2016 at 7:24 PM, Dan Trainor < dtrainor at redhat.com > wrote:
> 
> 
> 
> Hi -
> 
> I want to bring up a subject that needs a little more attention. There are a
> few ideas floating around but it's important that we get this done right.
> 
> UI is unique in the sense that it operates almost entirely in a browser,
> talking directly to service API endpoints which it either figures out from
> they Keystone service catalog as using the publicURL endpoint for each
> service, or by specifying these API endpoints in a configuration file.
> Though overriding the API endpoints in the UI's local configuration file is
> an option that's available, I understand that we want to move towards
> relying exclusively on Keystone for accurate and correct endpoint
> configuration.
> 
> Right now, all of the service API endpoints that UI needs to talk with are
> only listening on the ctlplane network.
> 
> We've had several iterations of testing and development of the UI over time
> and as a result of that, three different solutions that work - depending on
> the exact circumstances - have been created which all can achieve the same
> goal of allowing the UI to talk to these endpoints:
> 
> - Local SSH port tunneling of the required ports that UI talks to, from the
> system running the UI to the Undercloud, and configuring the UI to talk to
> localhost:NNNN. This method "works", but it's not a solution we can
> recommend
> - Making the interface on which these services already listen on - the
> ctlplane network - routable. Again, this method "works", but we treat this
> interface in a very special manner on purpose, not the least of which
> because of it's ability to facilitate pxebooting
> - Change the public endpoints in the Keystone catalog to be that of the
> existing external, routable interface of the Undercloud for each service
> required by the UI. This also requires configuring each service that UI
> needs to talk with, to listen on the existing, external, routable interface
> on the Undercloud. Some services support a list of interfaces and IPs to
> listen on; others require exactly one argument, in which case the address of
> 0.0.0.0 would need to be used
> 
> According to the API Endpoint Configuration Recommendation guide[1], the
> third option seems most viable and one that we can recommend. The document
> briefly describes the security implications of having these services open on
> a public interface but recommends the use of a stringent network policy -
> something we're used to recommending and helping with. The first two
> options, not so much.
> 
> Based on discussions I've had with other people, it's my impression that the
> third option is likely the one that we should proceed with.
> 
> This concern is largely transparent to how we're currently testing and
> developing the UI because most of that work is done on local, virtualized
> environments. When this happens, libvirt does the heavy lifting of creating
> a network that's easily routable from the host system. If not that, then the
> evolution of instructions for setting up these local environments over time
> have recommended using SSH port forwarding. However, neither of these
> options should be recommended.
> 
> Thoughts?
> 
> Thanks
> -dant
> 
> --
> 
> P.S. and full disclosure: I'm biased towards the third option.
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> --
> Juan Antonio Osorio R.
> e-mail: jaosorior at gmail.com
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org ?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>