[heat] keystone endpoint configuration
In openstack-ansible we are trying to help a number of our end users with their heat deployments, some of them in conjunction with magnum.
There is some uncertainty with how the following heat.conf sections should be configured:
[clients_keystone] auth_uri = ...
[keystone_authtoken] www_authenticate_uri = ...
It does not appear to be possible to define a set of internal or external keystone endpoints in heat.conf which allow the following:
* The orchestration panels being functional in horizon * Deployers isolating internal openstack from external networks * Deployers using self signed/company cert on the external endpoint * Magnum deployments completing * Heat delivering an external endpoint at [1] * Heat delivering an external endpoint at [2]
There are a number of related bugs:
https://bugs.launchpad.net/openstack-ansible/+bug/1814909 https://bugs.launchpad.net/openstack-ansible/+bug/1811086 https://storyboard.openstack.org/#!/story/2004808 https://storyboard.openstack.org/#!/story/2004524
Any help we could get from the heat team to try to understand the root cause of these issues would be really helpful.
Jon.
[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/server_b...
[2] https://github.com/openstack/heat/blob/master/heat/engine/resources/signal_r...
On Wed, Feb 20, 2019 at 1:43 PM Jonathan Rosser jonathan.rosser@rd.bbc.co.uk wrote:
In openstack-ansible we are trying to help a number of our end users with their heat deployments, some of them in conjunction with magnum.
There is some uncertainty with how the following heat.conf sections should be configured:
[clients_keystone] auth_uri = ...
[keystone_authtoken] www_authenticate_uri = ...
It does not appear to be possible to define a set of internal or external keystone endpoints in heat.conf which allow the following:
- The orchestration panels being functional in horizon
- Deployers isolating internal openstack from external networks
- Deployers using self signed/company cert on the external endpoint
- Magnum deployments completing
- Heat delivering an external endpoint at [1]
- Heat delivering an external endpoint at [2]
There are a number of related bugs:
https://bugs.launchpad.net/openstack-ansible/+bug/1814909 https://bugs.launchpad.net/openstack-ansible/+bug/1811086 https://storyboard.openstack.org/#!/story/2004808 https://storyboard.openstack.org/#!/story/2004524
Any help we could get from the heat team to try to understand the root cause of these issues would be really helpful.
I think this is a really critical issue that Jonathan has spent a lot of time on to get to work.
If we can't support this model, maybe we should consider dropping the whole idea of admin/internal/public if we can't commit to testing it properly.
Jon.
[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/server_b...
[2] https://github.com/openstack/heat/blob/master/heat/engine/resources/signal_r...
On 26/02/19 11:11 AM, Mohammed Naser wrote:
If we can't support this model, maybe we should consider dropping the whole idea of admin/internal/public
I wish to subscribe to your newsletter ;)
Did we ever support that model though? We have those different endpoints in the catalog, but whether operators could require the user to use different URIs for the catalog itself depending on where they're calling from is not controlled by the catalog code. I doubt we've said anything about it either way (though we should).
- ZB
On 20/02/19 1:40 PM, Jonathan Rosser wrote:
In openstack-ansible we are trying to help a number of our end users with their heat deployments, some of them in conjunction with magnum.
There is some uncertainty with how the following heat.conf sections should be configured:
[clients_keystone] auth_uri = ...
[keystone_authtoken] www_authenticate_uri = ...
It does not appear to be possible to define a set of internal or external keystone endpoints in heat.conf which allow the following:
* The orchestration panels being functional in horizon * Deployers isolating internal openstack from external networks * Deployers using self signed/company cert on the external endpoint * Magnum deployments completing * Heat delivering an external endpoint at [1] * Heat delivering an external endpoint at [2]
There are a number of related bugs:
https://bugs.launchpad.net/openstack-ansible/+bug/1814909 https://bugs.launchpad.net/openstack-ansible/+bug/1811086 https://storyboard.openstack.org/#!/story/2004808 https://storyboard.openstack.org/#!/story/2004524
Based on this and your comment on IRC[1] - and correct me if I'm misunderstanding here - the crux of the issue is that the Keystone auth_url must be accessed via different addresses depending on which network the request is coming from?
I don't think this was ever contemplated as a use case in developing Heat. For my part, I certainly always assumed that while the Keystone catalog could contain different Public/Internal/Admin endpoints for each service, there was only a single place to access the catalog (i.e. each cloud had a single unique auth_url).
It's entirely possible this wasn't a valid assumption about the how clouds would/should be deployed in practice. If that's the case then we likely need some richer configuration options. The design of the Keystone catalog predates both the existence of Heat and the idea that cloud workloads might have reason to access the OpenStack APIs, and nobody is really an expert on both although we've gotten better at communicating.
[1] http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2019-02-26.log.html#t...
Any help we could get from the heat team to try to understand the root cause of these issues would be really helpful.
Jon.
[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/server_b...
[2] https://github.com/openstack/heat/blob/master/heat/engine/resources/signal_r...
On 27/02/2019 20:49, Zane Bitter wrote:
On 20/02/19 1:40 PM, Jonathan Rosser wrote:
In openstack-ansible we are trying to help a number of our end users with their heat deployments, some of them in conjunction with magnum.
There is some uncertainty with how the following heat.conf sections should be configured:
[clients_keystone] auth_uri = ...
[keystone_authtoken] www_authenticate_uri = ...
It does not appear to be possible to define a set of internal or external keystone endpoints in heat.conf which allow the following:
* The orchestration panels being functional in horizon * Deployers isolating internal openstack from external networks * Deployers using self signed/company cert on the external endpoint * Magnum deployments completing * Heat delivering an external endpoint at [1] * Heat delivering an external endpoint at [2]
There are a number of related bugs:
https://bugs.launchpad.net/openstack-ansible/+bug/1814909 https://bugs.launchpad.net/openstack-ansible/+bug/1811086 https://storyboard.openstack.org/#!/story/2004808 https://storyboard.openstack.org/#!/story/2004524
Based on this and your comment on IRC[1] - and correct me if I'm misunderstanding here - the crux of the issue is that the Keystone auth_url must be accessed via different addresses depending on which network the request is coming from?
The most concrete example I can give is that of a Magnum k8s deployment, where heat is used to create several VM and deploy software. Callback URLs are embedded into those VM and SoftwareDeployments, and the Callback URL must be accessible from the VM, this would always need to be something that could reasonably be called a "Public" endpoint.
Conversely, heat itself needs to be able to talk to many other openstack components, defined in the [clients_*] config sections. It is reasonable to describe these interactions as being "Internal" - I may misunderstand some of this though.
So here lies the issue - appropriate entries in heat.conf to make internal interactions between heat and horizon (one example) work in real-world deployments results in the keystone internal URL being placed in callbacks, and then SoftwareDemployments never complete as the internal keystone URL is not usually accessible to a VM. I suspect that there is not much coverage for this kind of network separation in gate tests.
I don't think this was ever contemplated as a use case in developing Heat. For my part, I certainly always assumed that while the Keystone catalog could contain different Public/Internal/Admin endpoints for each service, there was only a single place to access the catalog (i.e. each cloud had a single unique auth_url).
I think that as far as heat itself interacting with other openstack components is concerned there does not need to be more than one auth_url. However it is very important to make a distinction between the context in which the heat code runs and the context of a VM created by heat - any callback URL created must be valid for the context of the VM, not the heat code.
It's entirely possible this wasn't a valid assumption about the how clouds would/should be deployed in practice. If that's the case then we likely need some richer configuration options. The design of the Keystone catalog predates both the existence of Heat and the idea that cloud workloads might have reason to access the OpenStack APIs, and nobody is really an expert on both although we've gotten better at communicating.
[1] http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2019-02-26.log.html#t...
Colleen makes some observations about the use of keystone config in heat - and interestingly suggests a seperate config entry for cases where a keystone URL should be handed on to another service. Mohammed and I have already discussed additional config options being a potential solution whilst trying to debug this.
There are already examples of similar config options in heat.conf, such as "heat_waitcondition_server_url" - would additonal config items such as server_base_auth_url and signal_responder_auth_url be appropriate so that we can be totally explicit about the endpoints handed on to created VM?
On Thu, Feb 28, 2019 at 4:02 AM Jonathan Rosser < jonathan.rosser@rd.bbc.co.uk> wrote:
On 27/02/2019 20:49, Zane Bitter wrote:
On 20/02/19 1:40 PM, Jonathan Rosser wrote:
In openstack-ansible we are trying to help a number of our end users with their heat deployments, some of them in conjunction with magnum.
There is some uncertainty with how the following heat.conf sections should be configured:
[clients_keystone] auth_uri = ...
[keystone_authtoken] www_authenticate_uri = ...
It does not appear to be possible to define a set of internal or external keystone endpoints in heat.conf which allow the following:
- The orchestration panels being functional in horizon
- Deployers isolating internal openstack from external networks
- Deployers using self signed/company cert on the external endpoint
- Magnum deployments completing
- Heat delivering an external endpoint at [1]
- Heat delivering an external endpoint at [2]
There are a number of related bugs:
https://bugs.launchpad.net/openstack-ansible/+bug/1814909 https://bugs.launchpad.net/openstack-ansible/+bug/1811086 https://storyboard.openstack.org/#!/story/2004808 https://storyboard.openstack.org/#!/story/2004524
Based on this and your comment on IRC[1] - and correct me if I'm misunderstanding here - the crux of the issue is that the Keystone auth_url must be accessed via different addresses depending on which network the request is coming from?
The most concrete example I can give is that of a Magnum k8s deployment, where heat is used to create several VM and deploy software. Callback URLs are embedded into those VM and SoftwareDeployments, and the Callback URL must be accessible from the VM, this would always need to be something that could reasonably be called a "Public" endpoint.
Conversely, heat itself needs to be able to talk to many other openstack components, defined in the [clients_*] config sections. It is reasonable to describe these interactions as being "Internal" - I may misunderstand some of this though.
So here lies the issue - appropriate entries in heat.conf to make internal interactions between heat and horizon (one example) work in real-world deployments results in the keystone internal URL being placed in callbacks, and then SoftwareDemployments never complete as the internal keystone URL is not usually accessible to a VM. I suspect that there is not much coverage for this kind of network separation in gate tests.
I think the crux of issue is that we use the same keystone endpoint (auth_uri/ www_authenticate_uri from clients_keystone/keystone_authtoken sections in heat.conf) when creating the auth_plugins in the context[1], heat internal keystone objects[3] and the keystone auth_url[2] we pass on to the instances for signaling.
We've auth_url middleware that sets X-Auth-Url header in the request by reading the above config options. There is probably some history[4] (i.e on why we fallback to use [keystone_authtoken]auth_uri/www_authenticate_uri) which is beyond my involvement with heat.
AFAICT, we're aware of this problem for sometime and no one ever tried to fix it. I think we can provide a separate config option for what auth_url we send to the instances as they may not be in the same network, as the ones like other services use.
[1] https://github.com/openstack/heat/blob/master/heat/common/context.py#L255
[2] https://github.com/openstack/heat/blob/master/heat/engine/resources/signal_r...
[3] https://github.com/openstack/heat/blob/master/heat/engine/clients/os/keyston...
[4] https://github.com/openstack/heat/commit/1e71566169d9ffb34ed4e1bdf3f264cbdbb...
I don't think this was ever contemplated as a use case in developing Heat. For my part, I certainly always assumed that while the Keystone catalog could contain different Public/Internal/Admin endpoints for each service, there was only a single place to access the catalog (i.e. each cloud had a single unique auth_url).
I think that as far as heat itself interacting with other openstack components is concerned there does not need to be more than one auth_url. However it is very important to make a distinction between the context in which the heat code runs and the context of a VM created by heat - any callback URL created must be valid for the context of the VM, not the heat code.
It's entirely possible this wasn't a valid assumption about the how clouds would/should be deployed in practice. If that's the case then we likely need some richer configuration options. The design of the Keystone catalog predates both the existence of Heat and the idea that cloud workloads might have reason to access the OpenStack APIs, and nobody is really an expert on both although we've gotten better at communicating.
[1]
http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2019-02-26.log.html#t...
Colleen makes some observations about the use of keystone config in heat
- and interestingly suggests a seperate config entry for cases where a
keystone URL should be handed on to another service. Mohammed and I have already discussed additional config options being a potential solution whilst trying to debug this.
There are already examples of similar config options in heat.conf, such as "heat_waitcondition_server_url" - would additonal config items such as server_base_auth_url and signal_responder_auth_url be appropriate so that we can be totally explicit about the endpoints handed on to created VM?
Hi,
On Wed, Feb 20, 2019, at 7:40 PM, Jonathan Rosser wrote:
In openstack-ansible we are trying to help a number of our end users with their heat deployments, some of them in conjunction with magnum.
There is some uncertainty with how the following heat.conf sections should be configured:
[clients_keystone] auth_uri = ...
[keystone_authtoken] www_authenticate_uri = ...
I know very little about heat, but I think there's some confusion about what [keystone_authtoken]/www_authenticate_uri is for, and after grepping a bit I think heat is misusing it. www_authenticate_uri (formerly known as auth_uri) is meant to be used by keystonemiddleware to set the WWW-Authenticate header in its response when a client request fails to present a valid keystone token. It's a wsgi middleware, so heat shouldn't be using it or even be aware of it. You would normally set it to keystone's public endpoint since it's what the server would present to an end user to help them retry their request. If the client already knows the right auth URL and grabs a token beforehand, it will never see what's in www_authenticate_uri.
Heat appears to be using it for its own purposes, which I think is not advisable. If heat needs to provide other subsystems or services with a URL for keystone, it should do that in a separate config.
I don't really have enough heat knowledge to make a recommendation for the issues below. While keystoneauth provides a way to filter the service catalog by admin/internal/public endpoint, it obviously doesn't help much when you don't know where keystone is yet.
Colleen
It does not appear to be possible to define a set of internal or external keystone endpoints in heat.conf which allow the following:
- The orchestration panels being functional in horizon
- Deployers isolating internal openstack from external networks
- Deployers using self signed/company cert on the external endpoint
- Magnum deployments completing
- Heat delivering an external endpoint at [1]
- Heat delivering an external endpoint at [2]
There are a number of related bugs:
https://bugs.launchpad.net/openstack-ansible/+bug/1814909 https://bugs.launchpad.net/openstack-ansible/+bug/1811086 https://storyboard.openstack.org/#!/story/2004808 https://storyboard.openstack.org/#!/story/2004524
Any help we could get from the heat team to try to understand the root cause of these issues would be really helpful.
Jon.
[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/server_b...
[2] https://github.com/openstack/heat/blob/master/heat/engine/resources/signal_r...
On 27/02/19 4:13 PM, Colleen Murphy wrote:
Hi,
On Wed, Feb 20, 2019, at 7:40 PM, Jonathan Rosser wrote:
In openstack-ansible we are trying to help a number of our end users with their heat deployments, some of them in conjunction with magnum.
There is some uncertainty with how the following heat.conf sections should be configured:
[clients_keystone] auth_uri = ...
[keystone_authtoken] www_authenticate_uri = ...
I know very little about heat, but I think there's some confusion about what [keystone_authtoken]/www_authenticate_uri is for, and after grepping a bit I think heat is misusing it. www_authenticate_uri (formerly known as auth_uri) is meant to be used by keystonemiddleware to set the WWW-Authenticate header in its response when a client request fails to present a valid keystone token. It's a wsgi middleware, so heat shouldn't be using it or even be aware of it. You would normally set it to keystone's public endpoint since it's what the server would present to an end user to help them retry their request. If the client already knows the right auth URL and grabs a token beforehand, it will never see what's in www_authenticate_uri.
Thanks Colleen!
Heat appears to be using it for its own purposes, which I think is not advisable. If heat needs to provide other subsystems or services with a URL for keystone, it should do that in a separate config.
Ooh, I know this one because Johannes schooled me on it a couple of months back :)
We actually do have a separate config, it's [clients_keystone]/auth_uri. If, and only if, the user does not provide a value for that, we fall back to using the value in [keystone_authtoken]/www_authenticate_uri.
https://git.openstack.org/cgit/openstack/heat/tree/heat/common/endpoint_util...
I don't really have enough heat knowledge to make a recommendation for the issues below. While keystoneauth provides a way to filter the service catalog by admin/internal/public endpoint, it obviously doesn't help much when you don't know where keystone is yet.
Yeah, that seems to be the heart of the problem. [clients_keystone]/auth_uri is being used in multiple different ways for different things that may have different access to the various networks (some of which Heat has no control over - we give the user the URL and it's up to them where it is called from).
cheers, Zane.
Colleen
It does not appear to be possible to define a set of internal or external keystone endpoints in heat.conf which allow the following:
- The orchestration panels being functional in horizon
- Deployers isolating internal openstack from external networks
- Deployers using self signed/company cert on the external endpoint
- Magnum deployments completing
- Heat delivering an external endpoint at [1]
- Heat delivering an external endpoint at [2]
There are a number of related bugs:
https://bugs.launchpad.net/openstack-ansible/+bug/1814909 https://bugs.launchpad.net/openstack-ansible/+bug/1811086 https://storyboard.openstack.org/#!/story/2004808 https://storyboard.openstack.org/#!/story/2004524
Any help we could get from the heat team to try to understand the root cause of these issues would be really helpful.
Jon.
[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/server_b...
[2] https://github.com/openstack/heat/blob/master/heat/engine/resources/signal_r...
participants (5)
-
Colleen Murphy
-
Jonathan Rosser
-
Mohammed Naser
-
Rabi Mishra
-
Zane Bitter