On 03/08/2023 17:32, Jeremy Stanley wrote:

On 2023-08-03 17:09:29 +0200 (+0200), Christian Rohmann wrote:

...

we use Ceph RadosGW (Quincy release) to provide Swift as part of an OpenStack cloud.

[...]

While I don't have an answer for your question, I did want to clear up any potential misunderstanding. Please be aware that Ceph RadosGW is not Swift, it is a popular non-OpenStack alternative to using actual Swift, and supplies a partial replica of Swift's user-facing API (but with a substantially different backend implementation). Ceph RadosGW does not have feature-parity with Swift, lacking a number of Swift's features and sometimes having much different behaviors or performance even for the features it attempts to replicate.

So just to reiterate, Ceph RadosGW isn't Swift any more than Swift's S3-API is S3. It's merely a compatibility shim, and you shouldn't expect it to work the same way Swift does.

Thanks Jeremy for clarifying this. Honestly I was pretty aware of that fact, but did not want to NOT mention it.

Regarding the particular error I am seeing, this seems to then cause the OpenstackSDK to fail on discovering the "Swift caps" at https://opendev.org/openstack/openstacksdk/src/commit/88fc0c2cf6269dd2d3f862...

with the exception being: "No Info found for None: Client Error for url: https://object-store.region.cloud.example.com/info, Not Found" (I added a line to log the exception, which otherwise is just silently ignored). When issuing a GET to "https://object-store.region.cloud.example.com/swift/info" the caps are returned:

{"bulk_delete":{},"container_quotas":{},"swift":{"max_file_size":5368709120,"container_listing_limit":10000,"version":"17.2.6","policies":[{"default":true,"name":"default-placement"}],"max_object_name_size":1024,"strict_cors_mode":true,"max_container_name_length":255},"tempurl":{"methods":["GET","HEAD","PUT","POST","DELETE"]},"slo":{"max_manifest_segments":1000},"account_quotas":{},"staticweb":{},"tempauth":{"account_acls":true}}%

So to me it's not (yet) about Ceph vs. original Swift, but rather about the discovery logic in keystoneauth1 to find the correct base-URL from the endpoints, which is "/swift" instead of "/" in my case. But that could very well be also a possible setup for Swift, right? And isn't that what the endpoint list is all about? Telling clients the correct URLs for services? And why does this (using /swift) seem work without issue when doing other actions like "container list"?

I violently added Artem to CC, since he wrote the wrote the project cleanup code for object storage https://review.opendev.org/c/openstack/openstacksdk/+/853015

Regards (and sorry Artem for dragging you into this thread),

Christian

Hey Artem,

thanks a bunch for giving into this so quickly ...

On 04/08/2023 09:59, Artem Goncharov wrote:

Solution:

• either you remove /swift from your path when deploying swift

That's not possible, unfortunately, if you also want to support S3. See https://docs.ceph.com/en/latest/radosgw/config-ref/#confval-rgw_swift_url_pr...

One option might be to use dedicated instances of RGW just for Swift and others for S3. But this requires to also use different endpoints / hostnames. But even if only Swift was used, with "/swift" being the default prefix on Ceph RGWs, I am highly confident that this prefixed path therefore exists for quite a few clouds using Ceph to provide the object storage via the Swift protocol.

• or we need to change the mentioned calculation logic by explicitly

stripping last 2 elements of the path of the service catalog entry (what seems logical on one side is not guaranteed to be a proper solution either - I have seen wild examples)

Well statically stripping a certain number of elements does indeed not seem "proper.

If you look at https://docs.ceph.com/en/latest/radosgw/keystone/#ocata-and-later vs https://docs.ceph.com/en/latest/radosgw/keystone/#cross-project-tenant-acces... the existence of "AUTH_$(project_id)s" cannot always be expected. But maybe a rule / regex can be found to strip optional version and AUTH element?

Kinda like ...

'(.*?)(/v[0-9](/AUTH_.+)?)?$'

to get all the path elements until the (optional) "v"ersion and an (optional) "AUTH_" at the end.

Honestly I don't know how the swift endpoint URLs looks like when deploying OpenStack Swift instead of Ceph RGW. But I suppose the endpoint has some potential variance in path there as well?

Regards

Christian

On 04/08/2023 11:59, Artem Goncharov wrote:

Precisely. I always strongly disliked cases where REST methods are exposed under non service catalog provided base. This is an ugly grave full of corpses. Idea with regex sounds like a least bad for me. Eventually finding /v1 and getting one level above.

You mean using the path up until "v1" or "v[1-9]+"? But is v1 (or any version) always part of the endpoint url so it's possible to use this as a definitive delimiter?

In any case, could you then craft a patch for this one? Regards and thanks again,

Christian

Hey Artem, all,

On 04.08.23 09:59, Artem Goncharov wrote:

Hi Christian,

It’s good that you pulled me explicitly.

So, the __bug__ is in https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/objec...

Explanation (reduced version):

• normally all the service endpoints are consumed as announced by the

service catalog (https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s%C2... your case)

• the URL above means that all of the API requests for swift must go

to https://object-store.region.cloud.example.com/swift/v1/AUTH_f2bc4bd34567ddc3...

• now the /info endpoint of the swift is explicitly NOT including the

project_id (AUTH_XXX suffix). Current code calculates the URL to query as mentioned above by reconstructing it from the base domain. There are few similar cases in the SDK when service exposes functions under __non_standard__ url by doing dirty hacks.

• Since so far nobody from us was deploying or dealing with cloud

having additional element in the path we never seen that as a bug.

Solution:

• either you remove /swift from your path when deploying swift
• or we need to change the mentioned calculation logic by explicitly

stripping last 2 elements of the path of the service catalog entry (what seems logical on one side is not guaranteed to be a proper solution either - I have seen wild examples)

Thanks again for your help ^^

With the bug (https://storyboard.openstack.org/#!/story/2010898) resolved and the bugfix to the SDK merged (https://review.opendev.org/c/openstack/openstacksdk/+/893062) and released with OpenStackSDK 2.0.x (https://docs.openstack.org/releasenotes/openstacksdk/unreleased.html#bug-fix...) I expected the issues with Swift URLs to be gone now.

But unfortunately it seems there is a similar issue about the Swift URL within keystoneauth1 as well. I wrote a little bit of code to reproduce it:

#!/bin/python

import openstack import logging

IDENTITY_API_VERSION = "3.14" CLOUD_NAME = "yourcloud-in-clouds.yml" PROJECT_NAME = "some-existing-project"

logging.basicConfig(format="%(asctime)s %(levelname)-8s %(pathname)s [%(lineno)s: %(funcName)s] %(message)s") logger = logging.getLogger(__name__) logger.setLevel(level='DEBUG')

try:     print(f"Connecting to openstack cloud {CLOUD_NAME}")     connection = openstack.connect(         cloud=CLOUD_NAME, domain="Default", identity_api_version=IDENTITY_API_VERSION     )     connection.object_store.containers()

print(f"Connecting to openstack cloud {CLOUD_NAME} as project {PROJECT_NAME}")     project_connection = connection.connect_as_project(PROJECT_NAME)     project_connection.object_store.containers()

except Exception as ex:     print(f"Exception fetching containers: {ex}") finally:     project_connection.close()     connection.close()

Running this code produces the following output / warnings

# python ./swift_url_parsing_error__example.py

Connecting to openstack cloud mycloud 2023-11-06 18:56:48,255 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_9c8353142098445c8fad21254089f... for discovery. Fallback to using that endpoint as the base url. 2023-11-06 18:56:48,267 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_9c8353142098445c8fad21254089f... for discovery. Fallback to using that endpoint as the base url. Connecting to openstack cloud mycloud as project myproject 2023-11-06 18:56:48,907 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_4a794332aab049d0831692750c821... for discovery. Fallback to using that endpoint as the base url. 2023-11-06 18:56:48,959 WARNING /myvenvpath/lib/python3.11/site-packages/keystoneauth1/discover.py [1242: _run_discovery] Failed to contact the endpoint at https://object.cloud.example.com/swift/v1/AUTH_4a794332aab049d0831692750c821... for discovery. Fallback to using that endpoint as the base url.

This is with Python 3.11 and an up to date venv:

# pip freeze certifi==2023.7.22 cffi==1.16.0 charset-normalizer==3.3.2 cryptography==41.0.5 decorator==5.1.1 dogpile.cache==1.2.2 idna==3.4 iso8601==2.1.0 jmespath==1.0.1 jsonpatch==1.33 jsonpointer==2.4 keystoneauth1==5.3.0 netifaces==0.11.0 openstacksdk==2.0.0 os-service-types==1.7.0 pbr==5.11.1 platformdirs==3.11.0 pycparser==2.21 PyYAML==6.0.1 requests==2.31.0 requestsexceptions==1.4.0 stevedore==5.1.0 urllib3==2.0.7

I suppose it comes down to the URL magic happening at https://github.com/openstack/keystoneauth/blob/44e477dfb5821cb535d880738a965... which ends up throwing the warning at https://github.com/openstack/keystoneauth/blob/44e477dfb5821cb535d880738a965....

Regards

Christian

Hey Artem,

thanks for jumping right at my reported issue and please excuse the long delay in my response.

On 07.11.23 11:27, Artem Goncharov wrote:

you found a deep rabbit hole, it took me 3 hours struggle through the traces to actually understand where this is happening. In the mentioned case an attempt to perform discovery was actually even happening multiple times, what is alone already very wrong. I was never seeing this particular case, because I was always forced to use endpoint_override for Swift anyway.

• when the is no endpoint_override current SDK tries to perform

version discovery for every service. In [1] I changed that to the similar logic used when endpoint_override is actually present

• next time proxy is retrieved from connection SDK again invokes

get_endpoint for it (what in the keystone ends in another discovery)

• next request send in the proxy is again ending doing same

(https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth1/s...)

While first 2 cases are happening on the SDK side directly it is possible to get that addressed by using "skip_discovery" flag, but the 3rd one is not trivial. So I actually decided it may be reasonable to set endpoint_override in that case. In my test it helped, but please check this as well.

Excuse me for being slow to grasp all of this, I am still looking through the code of OpenStackSDK, keystoneauth1 and your change to understand the flows that happen ...

1) So for the first case do I understand you (and your change [1]) correctly, that you "check" if the proxy (of lowest version) has skip_discovery=True accordingly the new branch is used to explicitly override the endpoint to circumvent discovery?. That would be logical and straightforward.

The other two sentences I don't quite understand fully ...

2) The second case is about getting a fresh instance proxy, as when switching to project context in my case? Is there still an issue in this case or am I misreading you? Running debug I don't see multiple calls to. 3) Is there still a problem here? It's that what https://review.opendev.org/c/openstack/openstacksdk/+/900280/2/openstack/ser... does? Trying to explicitly set an endpoint override?

One hint to your test source: simply having "project_connection.object_store.containers()" is doing actually nothing at all since all list operations in SDK return generators without going to the server. You need to consume them (at least "list(project_connection.object_store.containers())")

[1] https://review.opendev.org/c/openstack/openstacksdk/+/900280

I hope this helps, but anyway - performing discovery in that case is useless and negatively influences performance, but it is not actually breaking the code (consume generators in your test and you will see it working)

Yeah the code was actually a stripped down version of some other tool we wrote. I wanted to reduce it as much as possible. I should have said that my report was "just" about the warning messages (about discovery) and NOT about it not working at all - because even with the warnings it works in the sense that data is returned.

To me your change works as in no more warnings and no more (multiple) discoveries for object storage. Would you want to add some reviewers to the change to get this reviewed and merged to the SDK then?

Thanks again,

Christian