Open Stack

Hi -

Great suggestions, Dan.

To recap, we followed that up with a few other ideas on irc and we
eventually came to a point to test some of this, with slight modification.

UI also ships with a configuration file that can override the endpoint
information received from Keystone.  The file is located
at /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js.

Part of making this work means enabling SSL on the Undercloud when the UI
component is selected for installation in undercloud.conf.  I think this is
going to be a pretty reasonable request, but I'm interested in hearing
feedback from this, and what other implications it may have, that I can't
think of.  The changes I made were all one-off, unmanaged changes, just to
test this idea out.  I'll be doing some more tests but will probably be
looking for acceptance shortly.

Once SSL was enabled on the Undercloud, I made two edits to haproxy.cfg
that were pretty straightforward:  added a 'listen' server directive for UI
to both terminate SSL and forward the request to Apache, and added a 'bind'
statement for each service that UI expects to talk to (keystone, heat,
ironic, mistral, swift, zaqar-websocket).

Once those configuration changes were made, I had a very pleasant
experience using the UI.  It worked exactly as expected.  I think this
might be a viable option.

Thoughts?

Thanks!
-dant






On Fri, Sep 30, 2016 at 12:21 PM, Dan Sneddon <dsneddon at redhat.com> wrote:

> Thinking about this a little more, creating a new unified endpoint on the
> same port as the UI doesn't solve the problem at hand. The UI will use the
> service catalog to find endpoints, so we would need to change the endpoints
> in the service catalog, which changes the flow for the underlying services
> as well.
>
> Simply moving the control plane services to the external network won't
> work well in environments where the control plane is isolated and
> non-routed. The Undercloud can forward packets, but then becomes a
> bottleneck and a SPOF.
>
> A few approaches come to mind, but none of these are quick fixes:
>
> * Change the UI to get its list of endpoints from somewhere other than the
> service catalog and customize this with URLs that point to the Public VIP.
> Duplicate the services required for the UI on both control plane and
> external network. This might make it possible to make all connections over
> port 443, which is more firewall-friendly (which would be desirable or not
> depending on what kind of firewalling and traffic management is wanted).
>
> * Relax the rp_filter settings on the Controllers so they accept packets
> destined for the external network on their control plane interfaces; add a
> static route to the Public VIP via the control plane VIP on all
> non-controller nodes. Modify the service catalog to point to the public VIP
> for the services the UI needs. This would need to be combined with a
> security review to determine if additional iptables rules are required.
>
> * Split the service catalog, so we have an internal and an external view
> depending on where the query came from. I'm not sure how feasible this is.
>
> Of these, I think the rp_filter settings are the only ones that could be
> done solely with TripleO code changes. That might be worth investigating.
>
> Dan Sneddon  |  Principal OpenStack Engineer  |  dsneddon at redhat.com
>
>
> On Sep 30, 2016, at 11:36 AM, Dan Sneddon <dsneddon at redhat.com> wrote:
>
> I don't think we can rely on the Undercloud as an API proxy unless we
> address the lack of HA on the Undercloud.
>
> Wouldn't this be better implemented as as a single, name-based HAProxy
> instance running SSL on port 443 on the overcloud Public VIP? Then we could
> have the same endpoint for Horizon and every other API.
>
> I actually implemented this scheme in Havana before I joined Red Hat. At
> the time, we had to have a complex HAProxy config and patch the end points
> to support name-based URLs. I think some work has been done in OpenStack
> now to support this model, but I'm not sure where it stands.
>
> Dan Sneddon  |  Principal OpenStack Engineer  |  dsneddon at redhat.com
>
>
> On Sep 30, 2016, at 9:44 AM, Dan Trainor <dtrainor at redhat.com> wrote:
>
> Hi -
>
> I re-read your email a few times and like a few of the things that I see,
> but I'd love some more clarification.  As I read it, a few of these things
> conflict.  I believe you're suggesting that we don't make these services
> listen on a public interface due to security concerns (and of course,
> enabling SSL would break this because haproxy would listen on these
> interfaces/ports), but this approach would be acceptable if HAProxy was
> listening on these ports, terminating SSL, and sending them to each
> respective service backend.  Am I correct i understanding this?
>
> Are you suggesting that these endpoint ports would still be externally
> accessible on the primary (public) interface of the Undercloud, but just
> managed by HAProxy?  I think that's an acceptable approach.  Even if these
> endpoints are, like you suggested, listening on a separate network or IP as
> the Undercloud's primary interface, at least then it would be easier for
> organizations to enforce network access policies to these ports, and
> subsequently, these services that UI needs to talk to directly.
>
> I'm also perfectly fine with suggesting that if UI is installed, then this
> forces the Undercloud to be SSL enabled.  This would be a good way to move
> the idea of a secured, by default SSL-enabled Undercloud forward a little
> more, which is something we'd definitely like to see more.
>
> Thoughts?
>
> Thanks
> -dant
>
>
>
> On Thu, Sep 29, 2016 at 9:01 AM, Dan Trainor <dtrainor at redhat.com> wrote:
>
>> Hi, Juan -
>>
>> Actually, the third option is also not an option in the current
>>> undercloud setup, since making the services listen in 0.0.0.0 will break
>>> HAProxy. So when you're deploying with TLS things will break since we use
>>> HAProxy to terminate TLS connections.
>>>
>>
>> Ah, that's correct, isn't it.
>>
>>
>>
>>> On the other hand, we also don't want services to listen on 0.0.0.0
>>> since that would become a security concern. We should instead be blocking
>>> everything we don't need to have exposed (as we've done with the
>>> undercloud's database and rabbitmq).
>>>
>>
>> I don't disagree that we want to focus on least privilege, but we do have
>> documentation that talks about how to deal with this.  I addressed this in
>> my previous message, even if only to illustrate my understanding that there
>> would be concerns around this.  See more comments about this down below...
>>
>>
>>>
>>> Now, as we're trying to move to have more convergence between the
>>> undercloud and the overcloud (trying to deploy the undercloud via heat
>>> also, as Dan Prince has mentioned), I think some aspecs of this will bring
>>> a solution to this problem. for instance, just like we already do in the
>>> overcloud, we could have the undercloud's HAProxy always terminate the
>>> endpoints, which I'm attempting with these two patches:
>>> https://review.openstack.org/#/c/360366  https://review.openstack.org/#
>>> /c/360368 . Furthermore, we could have the public endpoints in HAProxy
>>> listen on a separate network that's accessible externally, also as we do in
>>> the overcloud. That way we don't need to change the actual interfaces the
>>> services are listening on, and rely on HAProxy, getting closer to how we do
>>> things in the overcloud. It seems to me that it would help solve the
>>> problem.
>>>
>>
>> I like that idea.  Though, this effectively shifts the process of having
>> these services themselves listen on different IPs/ports and offloads that
>> to HAProxy.  Whatever security concerns we have with opening up network
>> communications would still exist, but I think that would be more broadly
>> accepted considering these connections are now managed by HAProxy which
>> *only* listens for SSL connections.
>>
>> Another challenge with isolating this traffic on a separate network is
>> that we introduce a new dependency that's going to take administrative time
>> to set up and configure outside of OpenStack - do we really want that?
>>
>> Thanks!
>> -dant
>>
>>
>>
>>
>>>
>>> On Wed, Sep 28, 2016 at 7:24 PM, Dan Trainor <dtrainor at redhat.com>
>>> wrote:
>>>
>>>> Hi -
>>>>
>>>> I want to bring up a subject that needs a little more attention.  There
>>>> are a few ideas floating around but it's important that we get this done
>>>> right.
>>>>
>>>> UI is unique in the sense that it operates almost entirely in a
>>>> browser, talking directly to service API endpoints which it either figures
>>>> out from they Keystone service catalog as using the publicURL endpoint for
>>>> each service, or by specifying these API endpoints in a configuration
>>>> file.  Though overriding the API endpoints in the UI's local configuration
>>>> file is an option that's available, I understand that we want to move
>>>> towards relying exclusively on Keystone for accurate and correct endpoint
>>>> configuration.
>>>>
>>>> Right now, all of the service API endpoints that UI needs to talk with
>>>> are only listening on the ctlplane network.
>>>>
>>>> We've had several iterations of testing and development of the UI over
>>>> time and as a result of that, three different solutions that work -
>>>> depending on the exact circumstances - have been created which all can
>>>> achieve the same goal of allowing the UI to talk to these endpoints:
>>>>
>>>> - Local SSH port tunneling of the required ports that UI talks to, from
>>>> the system running the UI to the Undercloud, and configuring the UI to talk
>>>> to localhost:NNNN. This method "works", but it's not a solution we can
>>>> recommend
>>>> - Making the interface on which these services already listen on - the
>>>> ctlplane network - routable.  Again, this method "works", but we treat this
>>>> interface in a very special manner on purpose, not the least of which
>>>> because of it's ability to facilitate pxebooting
>>>> - Change the public endpoints in the Keystone catalog to be that of the
>>>> existing external, routable interface of the Undercloud for each service
>>>> required by the UI.  This also requires configuring each service that UI
>>>> needs to talk with, to listen on the existing, external, routable interface
>>>> on the Undercloud.  Some services support a list of interfaces and IPs to
>>>> listen on; others require exactly one argument, in which case the address
>>>> of 0.0.0.0 would need to be used
>>>>
>>>> According to the API Endpoint Configuration Recommendation guide[1],
>>>> the third option seems most viable and one that we can recommend.  The
>>>> document briefly describes the security implications of having these
>>>> services open on a public interface but recommends the use of a stringent
>>>> network policy - something we're used to recommending and helping with.
>>>> The first two options, not so much.
>>>>
>>>> Based on discussions I've had with other people, it's my impression
>>>> that the third option is likely the one that we should proceed with.
>>>>
>>>> This concern is largely transparent to how we're currently testing and
>>>> developing the UI because most of that work is done on local, virtualized
>>>> environments.  When this happens, libvirt does the heavy lifting of
>>>> creating a network that's easily routable from the host system.  If not
>>>> that, then the evolution of instructions for setting up these local
>>>> environments over time have recommended using SSH port forwarding.
>>>> However, neither of these options should be recommended.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks
>>>> -dant
>>>>
>>>> --
>>>>
>>>> P.S. and full disclosure:  I'm biased towards the third option.
>>>>
>>>>
>>>> ____________________________________________________________
>>>> ______________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.op
>>>> enstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Juan Antonio Osorio R.
>>> e-mail: jaosorior at gmail.com
>>>
>>>
>>> ____________________________________________________________
>>> ______________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161007/ab893373/attachment.html>