Hi,

Thanks for the follow-up. I was able to find this cause in the IRC channel. I ultimately upgraded the other nodes to 20.04 in our test clusters and moved the quarantined objects back to where they belonged. From there the files were downloadable again.

We are going to try to create a new liberasurecode package 1.6.2 for 20.04 so we can set the environment variable to write legacy CRC headers until all the nodes in the cluster can be upgraded. It is hard to find the information about the bug pre-upgrade. I didn't see it in the release notes for 2.25.2 (well they don't exist) and I don't see anything about it in the main Ubuntu Release notes. This is why we have testing environments.

Reid
From: Clay Gerrard <clay.gerrard@gmail.com>
Sent: Sunday, October 2, 2022 17:41
To: Pete Zaitcev <zaitcev@redhat.com>
Cc: Reid Guyett <rguyett@datto.com>; openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>; Matthew Grinnell <mgrinnell@datto.com>
Subject: Re: [Swift][Ussuri] Erasure Coding Quarantines
 
KASEYA
Warning: Sender @clay​.gerrard@gmail​.com is not yet trusted by your organization.
Please be careful before replying or clicking on the URLs.
powered by Graphus®

[EXTERNAL]



On Fri, Sep 30, 2022 at 4:56 PM Pete Zaitcev < zaitcev@redhat.com> wrote:

Unfortunately, I'm not familiar with the exact details of this.
There was a window where depending on how linker worked, our
code could get linked with an incorrect zlib crc routine randomly.


# When upgrading from liberasurecode<=1.5.0, you may want to continue writing
# legacy CRCs until all nodes are upgraded and capabale of reading fragments
# with zlib CRCs. liberasurecode>=1.6.2 checks for the environment variable
# LIBERASURECODE_WRITE_LEGACY_CRC; if set (value doesn't matter), it will use
# its legacy CRC. Set this option to true or false to ensure the environment
# variable is or is not set. Leave the option blank or absent to not touch
# the environment (default). For more information, see
# https://bugs.launchpad.net/liberasurecode/+bug/1886088
# write_legacy_ec_crc =


set it in your object-server [DEFAULT] confs too

--
Clay Gerrard

Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at legal@kaseya.com, and then delete it. To opt-out of receiving emails Please click here. The term 'this e-mail' includes any and all attachments.