Open Stack

On Thu, Jun 2, 2022, at 3:44 AM, Artem Goncharov wrote:
> Hey,
>
> https://review.opendev.org/c/openstack/openstacksdk/+/844414 should 
> address the issue. For R1 of SDK we want to ensure all interfaces of 
> SDK use same code branch and not 3 different like it was before. So 
> with the latest changes we now rely here on the regular mechanism that 
> actually watches how to handle each known parameter. All of the 
> arbitrary parameters in Swift are managed as container/object metadata 
> and should get X-Container-Meta- or X-Object-Meta- prefixes in order to 
> be considered and returned back by manila Swift. This is sadly not the 
> case for Rax as you rightfully mention.

Note that the docs [3]  still say that headers are passed through, which isn't quite the case in 0.99.0. Filtered headers for what the SDK thinks are valid are passed through. This change should probably update the docs too.

>
> So for now I have added header required by Rax into “known” exceptions 
> and added explicit unit test for that. Would be cool to find way how I 
> could test that really in Rax to ensure particular Zuul use case will 
> work. 

Probably the simplest thing is to have a test that checks the right headers are sent. Then if/when clouds change behavior we can update the tests to follow.

>
> Generally we would need to think whether SDK should continue “tell user 
> he is wrong before reaching out to the server” approach or to switch to 
> the “believe the user and let server reject” approach generally. This 
> here is precisely one of those cases. Alternatively we would need to 
> widen our concept of vendors and move all of the specific Rax logic 
> over there. We surely have lot of OpenStack based clouds, which do not 
> explicitly implement everything in the compatible way. So in order to 
> make SDK/CLI working with all of them we either drop most of the 
> validation logic in SDK or really start implementing cloud specific 
> overrides as part of “extensions”.

The problem with telling the user they are wrong before talking to the server is that it assumes that clouds don't do weird things. History has shown us that cloud definitely do weird things and this isn't related to any single cloud. Off the top of my head I can come up with a few examples and if we look through the git history of nodepool and opendev system-config I'm sure we'll find many many more occurences. This isn't a specific cloud problem either; they all do it one way or another.

It is my opinion that tools like openstacksdk should be forgiving in what they accept to enable users to use openstack in its many forms. Shade is a great example of this. Eventually shade was rolled into openstacksdk and doesn't really exist anymore. It feels like the spirit of shade has been lost though and we're regressing back to 2014 when you couldn't talk to two different clouds without 4 different client tools.

Similarly support for old versions of APIs have been ripped out of these tools. Unfortunately some clouds are still deploying old versions of clouds and ideally as a user I wouldn't need to think too hard about that. It isn't my fault this is the situation, I just want the software to work.

>
> Artem
>
>> On 1. Jun 2022, at 19:55, Clark Boylan <cboylan at sapwetik.org> wrote:
>> 
>> Hello,
>> 
>> Last week the OpenDev team upgraded Zuul which deployed zuul-executors with openstacksdk==0.99.0 installed. After this, test job logs uploaded to the rackspace swift location were no longer viewable in the Zuul dashboard. The reason for this is these objects did not have properly configured CORS headers any longer. The objects uploaded to OVH appear to have been fine (potentially because the headers are not required for OVH).
>> 
>> Today we downgraded the version of openstacksdk on the zuul-executors to 0.61.0 and the rackspace uploaded job log objects have functional CORS headers again. For this reason we're fairly confident that something about the 0.99.0 release is impacting the setting of these headers.
>> 
>> The code that does these uploads with openstacksdk can be found here [0]. I suspect that the problem is going to be related to this header set on line 227 [1]. In particular that appears to be rackspace specific and this issue is rackspace specific under 0.99.0. However, I wasn't able to find anything in openstacksdk that would have a problem with this. Does anyone else have ideas? I'm somewhat concerned that this represents a class of data loss (metadata is going missing) for swift object uploads.
>> 
>> Additionally, we rely on setting the x-delete-after header to timeout, expire, and prune our logs in swift [2]. If there is a general problem with 0.99.0 setting headers on objects it is possible that all of the objects we created using 0.99.0 do not have this metadata set and will leak in their swift containers.
>> 
>> [0] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py
>> [1] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L226-L227
>> [2] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L224
>>

[3] https://docs.openstack.org/openstacksdk/latest/user/connection.html#openstack.connection.Connection.create_object