Sending to the list as I intended the first time... with edit to take into account new info from Rabi. On 27/02/19 5:29 PM, Jonathan Rosser wrote:

Conversely, heat itself needs to be able to talk to many other openstack components, defined in the [clients_*] config sections. It is reasonable to describe these interactions as being "Internal" - I may misunderstand some of this though.

Yeah, that is reasonable, and in fact we give you the option in the config file to choose which set of endpoints to use from the catalog. But we assume that there is only one auth_url used to fetch the catalog.

So here lies the issue - appropriate entries in heat.conf to make internal interactions between heat and horizon (one example) work in real-world deployments results in the keystone internal URL being placed in callbacks, and then SoftwareDemployments never complete as the internal keystone URL is not usually accessible to a VM.

Do you know which config options affect Horizon? It's surprising to me that anything in *Heat's* config would make a difference to Horizon. The only thing I can think of is that www_authenticate_uri might have to be on the internal network for Horizon to work, but Colleen already said that it should always be on the public network. And in any event it would be overridden for the purposes of signal URLs by setting [clients_keystone]/auth_uri to the public endpoint.

I suspect that there is not much coverage for this kind of network separation in gate tests.

Yeah, I suspect only the deployment projects run tests with complex-enough networking setups to be able to verify this, and it's not their job to check that Heat supports this use case.

There are already examples of similar config options in heat.conf, such as "heat_waitcondition_server_url" - would additonal config items such as server_base_auth_url and signal_responder_auth_url be appropriate so that we can be totally explicit about the endpoints handed on to created VM?

Yes, that's along the lines of what I was thinking too (although I think we'd only need one option, for URLs destined to be called from userspace). We already have an endpoint_type option (that defaults to PublicURL), so maybe we just need to be able to specify internal_auth_uri and public_auth_uri and we can select based on the endpoint type when we're using the clients internally, but always use the public one when gathering data to pass to a VM? [Edit: Rabi pointed out that X-Auth-Url is inserted by middleware, and gets its information from the same config options, so the following paragraph is a red herring.] Looking close at the code though, I wonder if part of the problem here is that we use the X-Auth-Url header from the request context whenever it is available (and only fall back to the config options when it is not), which should generally always be the public URL... but possibly is the internal URL if the request comes from Horizon... or Magnum?? In fact, I wonder if that could be the _whole_ problem in terms of the stuff you laid out at the beginning of the thread? Notwithstanding that there are still cases with completely isolated networks that cannot be covered by config options alone. e.g. as discussed in https://storyboard.openstack.org/#!/story/2004524 you can get the 'signal_url' attribute from OS::Heat::ScalingPolicy. That can either be passed to Aodh (should presumably use the internal auth URI) or e.g. given to a VM that might be running the user's own monitoring system (must use the public auth URI). The only way I can see to solve that would be to have separate 'internal_signal_url' and 'public_signal_url' attributes (are ordinary users even supposed to be able to see internal URLs in the catalog?). That's not an ideal experience for users since on the vast majority of clouds where the services are able to access external endpoints then the public URL will continue to work in either case (so extra work to figure out, and people will choose wrong leaving lots of non-interoperable templates floating around out there). But I guess it may be better than what we have now. cheers, Zane.