Hey Christian, you found a deep rabbit hole, it took me 3 hours struggle through the traces to actually understand where this is happening. In the mentioned case an attempt to perform discovery was actually even happening multiple times, what is alone already very wrong. I was never seeing this particular case, because I was always forced to use endpoint_override for Swift anyway. - when the is no endpoint_override current SDK tries to perform version discovery for every service. In [1] I changed that to the similar logic used when endpoint_override is actually present - next time proxy is retrieved from connection SDK again invokes get_endpoint for it (what in the keystone ends in another discovery) - next request send in the proxy is again ending doing same (https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth1/s...) While first 2 cases are happening on the SDK side directly it is possible to get that addressed by using "skip_discovery" flag, but the 3rd one is not trivial. So I actually decided it may be reasonable to set endpoint_override in that case. In my test it helped, but please check this as well. One hint to your test source: simply having "project_connection.object_store.containers()" is doing actually nothing at all since all list operations in SDK return generators without going to the server. You need to consume them (at least "list(project_connection.object_store.containers())") [1] https://review.opendev.org/c/openstack/openstacksdk/+/900280 I hope this helps, but anyway - performing discovery in that case is useless and negatively influences performance, but it is not actually breaking the code (consume generators in your test and you will see it working) Artem On 11/6/23 19:06, Christian Rohmann wrote:
Hey Artem, all,
On 04.08.23 09:59, Artem Goncharov wrote:
Hi Christian,
It’s good that you pulled me explicitly.
So, the __bug__ is in https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/objec...
Explanation (reduced version): - normally all the service endpoints are consumed as announced by the service catalog (https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s in your case) - the URL above means that all of the API requests for swift must go to https://object-store.region.cloud.example.com/swift/v1/AUTH_f2bc4bd34567ddc3... - now the /info endpoint of the swift is explicitly NOT including the project_id (AUTH_XXX suffix). Current code calculates the URL to query as mentioned above by reconstructing it from the base domain. There are few similar cases in the SDK when service exposes functions under __non_standard__ url by doing dirty hacks. - Since so far nobody from us was deploying or dealing with cloud having additional element in the path we never seen that as a bug.
Solution: - either you remove /swift from your path when deploying swift - or we need to change the mentioned calculation logic by explicitly stripping last 2 elements of the path of the service catalog entry (what seems logical on one side is not guaranteed to be a proper solution either - I have seen wild examples)
Thanks again for your help ^^
With the bug (https://storyboard.openstack.org/#!/story/2010898) resolved and the bugfix to the SDK merged (https://review.opendev.org/c/openstack/openstacksdk/+/893062) and released with OpenStackSDK 2.0.x (https://docs.openstack.org/releasenotes/openstacksdk/unreleased.html#bug-fix...) I expected the issues with Swift URLs to be gone now.
But unfortunately it seems there is a similar issue about the Swift URL within keystoneauth1 as well. I wrote a little bit of code to reproduce it:
#!/bin/python
import openstack import logging
IDENTITY_API_VERSION = "3.14" CLOUD_NAME = "yourcloud-in-clouds.yml" PROJECT_NAME = "some-existing-project"
logging.basicConfig(format="%(asctime)s %(levelname)-8s %(pathname)s [%(lineno)s: %(funcName)s] %(message)s") logger = logging.getLogger(__name__) logger.setLevel(level='DEBUG')
try: print(f"Connecting to openstack cloud {CLOUD_NAME}") connection = openstack.connect( cloud=CLOUD_NAME, domain="Default", identity_api_version=IDENTITY_API_VERSION ) connection.object_store.containers()
print(f"Connecting to openstack cloud {CLOUD_NAME} as project {PROJECT_NAME}") project_connection = connection.connect_as_project(PROJECT_NAME) project_connection.object_store.containers()
except Exception as ex: print(f"Exception fetching containers: {ex}") finally: project_connection.close() connection.close()
Running this code produces the following output / warnings
# python ./swift_url_parsing_error__example.py
Connecting to openstack cloud mycloud 2023-11-06 18:56:48,255 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_9c8353142098445c8fad21254089f... for discovery. Fallback to using that endpoint as the base url. 2023-11-06 18:56:48,267 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_9c8353142098445c8fad21254089f... for discovery. Fallback to using that endpoint as the base url. Connecting to openstack cloud mycloud as project myproject 2023-11-06 18:56:48,907 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_4a794332aab049d0831692750c821... for discovery. Fallback to using that endpoint as the base url. 2023-11-06 18:56:48,959 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_4a794332aab049d0831692750c821... for discovery. Fallback to using that endpoint as the base url.
This is with Python 3.11 and an up to date venv:
# pip freeze certifi==2023.7.22 cffi==1.16.0 charset-normalizer==3.3.2 cryptography==41.0.5 decorator==5.1.1 dogpile.cache==1.2.2 idna==3.4 iso8601==2.1.0 jmespath==1.0.1 jsonpatch==1.33 jsonpointer==2.4 keystoneauth1==5.3.0 netifaces==0.11.0 openstacksdk==2.0.0 os-service-types==1.7.0 pbr==5.11.1 platformdirs==3.11.0 pycparser==2.21 PyYAML==6.0.1 requests==2.31.0 requestsexceptions==1.4.0 stevedore==5.1.0 urllib3==2.0.7
I suppose it comes down to the URL magic happening at https://github.com/openstack/keystoneauth/blob/44e477dfb5821cb535d880738a965... which ends up throwing the warning at https://github.com/openstack/keystoneauth/blob/44e477dfb5821cb535d880738a965....
Regards
Christian