[Nova&Neutron] pagination returns an http, not https href link
Hello, we just migrated using new openstack CLI tools, and when using pagination for both Nova and Neutron, we noticed that they are returning http href link although they are expecting https, causing failure: ``` openstack port list Failure: Unable to establish connection to http://openstack....example.com:9696/v2.0/ports: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying in 0.5s.1 retries left Unable to establish connection to http://openstack.....example.com:9696/v2.0/ports: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) ``` Is this a bug in Nova/Neutron? I tried other services like Heat, Keystone, etc, and they either returning https href links or just take relative path without schema/host. Thanks, Chang
I suspect that this is a typical problem with api services run behind a load balancer, which terminates SSL. Because the port number 9696 indicates that the problem is observed with connection to neutron, a good first check is to ensure you set [oslo_middleware] enable_proxy_headers_parsing = True in your neutron.conf . (If you face the same problem with nova API then check the same option in nova.conf ) On 8/28/25 4:01 AM, Chang Xue wrote:
Hello, we just migrated using new openstack CLI tools, and when using pagination for both Nova and Neutron, we noticed that they are returning http href link although they are expecting https, causing failure: ``` openstack port list Failure: Unable to establish connection to http://openstack....example.com:9696/v2.0/ports: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying in 0.5s.1 retries left Unable to establish connection to http://openstack.....example.com:9696/v2.0/ports: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) ``` Is this a bug in Nova/Neutron? I tried other services like Heat, Keystone, etc, and they either returning https href links or just take relative path without schema/host.
Thanks, Chang
Yes I have it in my neutron.conf but it's not working. From openstack port list --debug, I could see `href":"http://openstack.bve.example.com:9696/v2.0/ports?fields=id&fields=name&fields=mac_address&fields=fixed_ips&fields=status&marker=...` in RESP BODY. Makes me think neutron is giving http in pagination href while the first call from the port list command is sending https.
Assuming you agree that you have a load balancer in front of neutron/nova api which terminates SSL, another thing you may have to check is that your load balancer is configured to pass down the Forwarded header in RFC7239 format or it adds the combination of the X-Forwarded-Proto header when a request is proxied/forwarded to api services so that the backend api services can detect the protocol in loadbalancer frontend. Also in case you customize api-paste.ini of these services then make sure that the http_proxy_to_wsgi middleware is in api pipelines (though I guess many users do not actually customize pipeline these days). On 8/29/25 12:49 AM, Chang Xue wrote:
Yes I have it in my neutron.conf but it's not working.
From openstack port list --debug, I could see `href":"http://openstack.bve.example.com:9696/v2.0/ports?fields=id&fields=name&fields=mac_address&fields=fixed_ips&fields=status&marker=...` in RESP BODY. Makes me think neutron is giving http in pagination href while the first call from the port list command is sending https.
On 28/08/2025 17:14, Takashi Kajinami wrote:
Assuming you agree that you have a load balancer in front of neutron/nova api which terminates SSL, another thing you may have to check is that your load balancer is configured to pass down the Forwarded header in RFC7239 format or it adds the combination of the X-Forwarded-Proto header when a request is proxied/forwarded to api services so that the backend api services can detect the protocol in loadbalancer frontend.
i think this would also be needed if your using apache with mod_ssl to handel the tls termintation instead of doign that in python. even with mod_wsgi instead of an external wsgi applctions server like uwsgi. it is possibel we have a bug here but this is good to check.
Also in case you customize api-paste.ini of these services then make sure that the http_proxy_to_wsgi middleware is in api pipelines (though I guess many users do not actually customize pipeline these days).
On 8/29/25 12:49 AM, Chang Xue wrote:
Yes I have it in my neutron.conf but it's not working.
From openstack port list --debug, I could see `href":"http://openstack.bve.example.com:9696/v2.0/ports?fields=id&fields=name&fields=mac_address&fields=fixed_ips&fields=status&marker=...` in RESP BODY. Makes me think neutron is giving http in pagination href while the first call from the port list command is sending https.
Thanks for the reply. Yes we have http_proxy_to_wsgi middleware in api-paste.ini. I also tried changed HAproxy configs to add frontend and backend, to set headers for neutron/nova. Interesting thing is, after that I can get port list work and related logs is available in HAproxy logs, but not always. Sometimes it's still giving me the same error as before, and I don't see meaningful logs in HAproxy. So my struggling point is: 1. when I don't add HAproxy configs for neutron, why it returns http href link, although I'm sure CLI is sending neutron https request? 2. after adding HAproxy configs, why it only works intermittently?
On 8/29/25 2:12 AM, Chang Xue wrote:
Thanks for the reply. Yes we have http_proxy_to_wsgi middleware in api-paste.ini.
I also tried changed HAproxy configs to add frontend and backend, to set headers for neutron/nova. Interesting thing is, after that I can get port list work and related logs is available in HAproxy logs, but not always. Sometimes it's still giving me the same error as before, and I don't see meaningful logs in HAproxy.
So my struggling point is: 1. when I don't add HAproxy configs for neutron, why it returns http href link, although I'm sure CLI is sending neutron https request?
Unless the headers indicating that these requests were "forwarded", the api service does not know anything about your load balancer. It doesn't know the request a user sent to your loadbalancer. It knows only the request coming from your load balancer (which uses HTTP instead of HTTPS) So it makes the best guess based on the its own endpoint, These HTTP headers are pretty well common ones to tell applications that these requests come through forwarder and applications can "guess" the external url precisely.
2. after adding HAproxy configs, why it only works intermittently? I have no clear idea about this. You may want to ensure that the forwarded option is set at the level common for all backend servers. Also make sure that you reload haproxy so that it completely reflect the forward header options added. If you still see the same problem then you may check if the issue is seen with a specific backend server to see if there can be a misconfiguration in these servers.
So my confusion is, without my HAproxy config change, it always fails with sending http in href. With my change it just sometimes still sends http in href. I've been looking into my HAproxy config to see if I missed anything, but sometimes it's successful and related logs show up should mean it does take effect? I'm curious if it's been "guessing", why it always doesn't work without the change.
Just want to update, I figured out what is wrong with my HAproxy config file, and now it works. Meanwhile, when I test pagination on other services like Keystone, they don't have the similar issue. I think it's problem just for Neutron and Nova? Which makes me think the config in HAproxy might be just patching it, but the service itself should be returning https by default or so? Thanks, Chang
On 9/2/25 11:20 PM, Chang Xue wrote:
Just want to update, I figured out what is wrong with my HAproxy config file, and now it works.
Meanwhile, when I test pagination on other services like Keystone, they don't have the similar issue.I think it's problem just for Neutron and Nova? I don't really think this behavior is specific to some services but is common. You probably have different settings for these services defined in your haproxy. Note that keystone didn't support pagination in old releases and it was relatively recently added.
Which makes me think the config in HAproxy might be just patching it, but the service itself should be returning https by default or so?
IT SHOULD NOT, because you can't always assume that TLS is enabled. It might be common to use TLS for public endpoints but I've seen multiple deployments using non-TLS for internal endpoints. As I said if you terminate TLS by your loadbalancer then you should make sure that your load balancer tells that the actual public-facing endpoint has HTTPS enabled, to the api services behind it. (and that's what is achieved by the forwarder options).
Thanks, Chang
On 02/09/2025 15:33, Takashi Kajinami wrote:
On 9/2/25 11:20 PM, Chang Xue wrote:
Just want to update, I figured out what is wrong with my HAproxy config file, and now it works.
Meanwhile, when I test pagination on other services like Keystone, they don't have the similar issue.I think it's problem just for Neutron and Nova? I don't really think this behavior is specific to some services but is common. You probably have different settings for these services defined in your haproxy. Note that keystone didn't support pagination in old releases and it was relatively recently added.
i think the delta might be that nova and neutron return the fully qualified path including the protocol where as some other service only return relative paths and make the client construct the fully path themselves. so for nova and neutron to work properly you have ot properly configure your revers proxy assuming you are not doing tls termination in the wsgi server. i.e. uwsgi, grunicorn ectra.
Which makes me think the config in HAproxy might be just patching it, but the service itself should be returning https by default or so?
IT SHOULD NOT, because you can't always assume that TLS is enabled.
i agree unless the service is doing tls termination we cant no and must rely on the X-Forawded headers which upstream proxies are required to set if they are doing the termination.
It might be common to use TLS for public endpoints but I've seen multiple deployments using non-TLS for internal endpoints. As I said if you terminate TLS by your loadbalancer then you should make sure that your load balancer tells that the actual public-facing endpoint has HTTPS enabled, to the api services behind it. (and that's what is achieved by the forwarder options).
ya using tls for all endpoint is much more common now but we cant assume it. we recommend using tls for all endpoint to protect the keyston bareer token that is included with each requst but we cant force peopel to do that without breaking them.
Thanks, Chang
participants (3)
-
Chang Xue
-
Sean Mooney
-
Takashi Kajinami