Hi folks, I am trying to test out Glance configured with Swift backend on 2026.1 prior to the release, and facing some issues. As we are super close to the release, I have decided to pop up with the ML, to ensure it's a "me" problem, and not a generic bug somewhere. So I am using: - Glance installed from SHA 07dd0a203b6883efa66486035c436b92c83da0f3 - glance-store 5.4.0 (according to U-C) - Swift installed from SHA 51b068309a626d2228ab69b5158ee857e96d101d Image upload works quite well [1], image is appearing in Glance and in Swift quite correctly. But whenever I attempt to create an instance from the image, or, basically, just download image with `openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008`,I get Glance WSGI crashing, resulting in 502: root@aio1-utility-container-20f6bf88:/# openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008 Unable to download image: checksum mismatch (sha512): 9a9bce0083a00939ec17c11febbfc767aa211aaa54f51e75c5a8b271a9b5637c77205a518b7a2007cb391d23cceb01e0e4e8d64832317151bc85b734b92a7be0 != a3b1ae48ab757e1e074c33c3159cb3bc749d0119677446b442 c5568eeaf2a4d741d794ca9750e13c3352b7db073d0f9101670f3cbede942df3f5fdb12024aaca <html><body><h1>502 Bad Gateway</h1> The server returned an invalid or incomplete response. </body></html> root@aio1-utility-container-20f6bf88:/# The actual crash and stack trace can be seen here [2]. With that, I can download the image with Swift client without any issues [3], and checksum matches. This result is pretty much reproducible, and I get it constantly in OpenStack-Ansible's CI jobs, for example [4] With that, I have the impression that something goes terribly wrong on the Glance side of things in combination with Swift, as CI jobs using Ceph or File backends are working normally. Given where we are release-wise, I'd love to ask for some help investigating this problem... [1] https://paste.openstack.org/show/bjdhtY2p33L4g0CTTdqH/ [2] https://paste.openstack.org/show/bRhFaEdoJotDngNXBjlB/ [3] https://paste.openstack.org/show/badVIJc0hwqbsy5X3Iv3/ [4] https://zuul.opendev.org/t/openstack/build/0be7ba6d8f854c0a88df9f9e33461d20/...
hi Dmitriy, The mentioned bug is reported in LP [1] and I have pushed a fix [2] for it. As a workaround you can set swift_store_retry_get_count to 0 in the config file. Let me know if mentioned fix [2] works for you. [1] https://bugs.launchpad.net/glance/+bug/2146432 [2] https://review.opendev.org/c/openstack/glance_store/+/982221 Thanks & Best Regards, Abhishek Kekane On Fri, Mar 27, 2026 at 8:50 PM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hi folks,
I am trying to test out Glance configured with Swift backend on 2026.1 prior to the release, and facing some issues. As we are super close to the release, I have decided to pop up with the ML, to ensure it's a "me" problem, and not a generic bug somewhere.
So I am using: - Glance installed from SHA 07dd0a203b6883efa66486035c436b92c83da0f3 - glance-store 5.4.0 (according to U-C) - Swift installed from SHA 51b068309a626d2228ab69b5158ee857e96d101d
Image upload works quite well [1], image is appearing in Glance and in Swift quite correctly. But whenever I attempt to create an instance from the image, or, basically, just download image with `openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008`,I get Glance WSGI crashing, resulting in 502: root@aio1-utility-container-20f6bf88:/# openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008 Unable to download image: checksum mismatch (sha512): 9a9bce0083a00939ec17c11febbfc767aa211aaa54f51e75c5a8b271a9b5637c77205a518b7a2007cb391d23cceb01e0e4e8d64832317151bc85b734b92a7be0 != a3b1ae48ab757e1e074c33c3159cb3bc749d0119677446b442 c5568eeaf2a4d741d794ca9750e13c3352b7db073d0f9101670f3cbede942df3f5fdb12024aaca
<html><body><h1>502 Bad Gateway</h1> The server returned an invalid or incomplete response. </body></html> root@aio1-utility-container-20f6bf88:/#
The actual crash and stack trace can be seen here [2]. With that, I can download the image with Swift client without any issues [3], and checksum matches.
This result is pretty much reproducible, and I get it constantly in OpenStack-Ansible's CI jobs, for example [4]
With that, I have the impression that something goes terribly wrong on the Glance side of things in combination with Swift, as CI jobs using Ceph or File backends are working normally.
Given where we are release-wise, I'd love to ask for some help investigating this problem...
[1] https://paste.openstack.org/show/bjdhtY2p33L4g0CTTdqH/ [2] https://paste.openstack.org/show/bRhFaEdoJotDngNXBjlB/ [3] https://paste.openstack.org/show/badVIJc0hwqbsy5X3Iv3/ [4] https://zuul.opendev.org/t/openstack/build/0be7ba6d8f854c0a88df9f9e33461d20/...
Hi Abhishek! Thanks for the prompt reply! While I think you're on spot with the diagnosis, I am not sure if the path is actually working. I tried to apply the patch, while having `swift_store_retry_get_count = 5` in the glance-api.conf, and I had exactly the same stack trace, as without the patch. However, setting swift_store_retry_get_count to 0 has solved the problem indeed. According to what I see in my test environment, your patch solves the issue only when swift_store_retry_get_count is not defined at all, but not when it's defined to non-zero value. пт, 27 мар. 2026 г. в 16:24, Abhishek Kekane <akekane@redhat.com>:
hi Dmitriy,
The mentioned bug is reported in LP [1] and I have pushed a fix [2] for it. As a workaround you can set swift_store_retry_get_count to 0 in the config file.
Let me know if mentioned fix [2] works for you.
[1] https://bugs.launchpad.net/glance/+bug/2146432 [2] https://review.opendev.org/c/openstack/glance_store/+/982221
Thanks & Best Regards,
Abhishek Kekane
On Fri, Mar 27, 2026 at 8:50 PM Dmitriy Rabotyagov < noonedeadpunk@gmail.com> wrote:
Hi folks,
I am trying to test out Glance configured with Swift backend on 2026.1 prior to the release, and facing some issues. As we are super close to the release, I have decided to pop up with the ML, to ensure it's a "me" problem, and not a generic bug somewhere.
So I am using: - Glance installed from SHA 07dd0a203b6883efa66486035c436b92c83da0f3 - glance-store 5.4.0 (according to U-C) - Swift installed from SHA 51b068309a626d2228ab69b5158ee857e96d101d
Image upload works quite well [1], image is appearing in Glance and in Swift quite correctly. But whenever I attempt to create an instance from the image, or, basically, just download image with `openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008`,I get Glance WSGI crashing, resulting in 502: root@aio1-utility-container-20f6bf88:/# openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008 Unable to download image: checksum mismatch (sha512): 9a9bce0083a00939ec17c11febbfc767aa211aaa54f51e75c5a8b271a9b5637c77205a518b7a2007cb391d23cceb01e0e4e8d64832317151bc85b734b92a7be0 != a3b1ae48ab757e1e074c33c3159cb3bc749d0119677446b442 c5568eeaf2a4d741d794ca9750e13c3352b7db073d0f9101670f3cbede942df3f5fdb12024aaca
<html><body><h1>502 Bad Gateway</h1> The server returned an invalid or incomplete response. </body></html> root@aio1-utility-container-20f6bf88:/#
The actual crash and stack trace can be seen here [2]. With that, I can download the image with Swift client without any issues [3], and checksum matches.
This result is pretty much reproducible, and I get it constantly in OpenStack-Ansible's CI jobs, for example [4]
With that, I have the impression that something goes terribly wrong on the Glance side of things in combination with Swift, as CI jobs using Ceph or File backends are working normally.
Given where we are release-wise, I'd love to ask for some help investigating this problem...
[1] https://paste.openstack.org/show/bjdhtY2p33L4g0CTTdqH/ [2] https://paste.openstack.org/show/bRhFaEdoJotDngNXBjlB/ [3] https://paste.openstack.org/show/badVIJc0hwqbsy5X3Iv3/ [4] https://zuul.opendev.org/t/openstack/build/0be7ba6d8f854c0a88df9f9e33461d20/...
Hi Dmitriy, Thank you for confirming, i will check and improve the code accordingly. Thanks & Best Regards, Abhishek Kekane On Fri, Mar 27, 2026 at 9:17 PM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hi Abhishek!
Thanks for the prompt reply! While I think you're on spot with the diagnosis, I am not sure if the path is actually working.
I tried to apply the patch, while having `swift_store_retry_get_count = 5` in the glance-api.conf, and I had exactly the same stack trace, as without the patch. However, setting swift_store_retry_get_count to 0 has solved the problem indeed.
According to what I see in my test environment, your patch solves the issue only when swift_store_retry_get_count is not defined at all, but not when it's defined to non-zero value.
пт, 27 мар. 2026 г. в 16:24, Abhishek Kekane <akekane@redhat.com>:
hi Dmitriy,
The mentioned bug is reported in LP [1] and I have pushed a fix [2] for it. As a workaround you can set swift_store_retry_get_count to 0 in the config file.
Let me know if mentioned fix [2] works for you.
[1] https://bugs.launchpad.net/glance/+bug/2146432 [2] https://review.opendev.org/c/openstack/glance_store/+/982221
Thanks & Best Regards,
Abhishek Kekane
On Fri, Mar 27, 2026 at 8:50 PM Dmitriy Rabotyagov < noonedeadpunk@gmail.com> wrote:
Hi folks,
I am trying to test out Glance configured with Swift backend on 2026.1 prior to the release, and facing some issues. As we are super close to the release, I have decided to pop up with the ML, to ensure it's a "me" problem, and not a generic bug somewhere.
So I am using: - Glance installed from SHA 07dd0a203b6883efa66486035c436b92c83da0f3 - glance-store 5.4.0 (according to U-C) - Swift installed from SHA 51b068309a626d2228ab69b5158ee857e96d101d
Image upload works quite well [1], image is appearing in Glance and in Swift quite correctly. But whenever I attempt to create an instance from the image, or, basically, just download image with `openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008`,I get Glance WSGI crashing, resulting in 502: root@aio1-utility-container-20f6bf88:/# openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008 Unable to download image: checksum mismatch (sha512): 9a9bce0083a00939ec17c11febbfc767aa211aaa54f51e75c5a8b271a9b5637c77205a518b7a2007cb391d23cceb01e0e4e8d64832317151bc85b734b92a7be0 != a3b1ae48ab757e1e074c33c3159cb3bc749d0119677446b442 c5568eeaf2a4d741d794ca9750e13c3352b7db073d0f9101670f3cbede942df3f5fdb12024aaca
<html><body><h1>502 Bad Gateway</h1> The server returned an invalid or incomplete response. </body></html> root@aio1-utility-container-20f6bf88:/#
The actual crash and stack trace can be seen here [2]. With that, I can download the image with Swift client without any issues [3], and checksum matches.
This result is pretty much reproducible, and I get it constantly in OpenStack-Ansible's CI jobs, for example [4]
With that, I have the impression that something goes terribly wrong on the Glance side of things in combination with Swift, as CI jobs using Ceph or File backends are working normally.
Given where we are release-wise, I'd love to ask for some help investigating this problem...
[1] https://paste.openstack.org/show/bjdhtY2p33L4g0CTTdqH/ [2] https://paste.openstack.org/show/bRhFaEdoJotDngNXBjlB/ [3] https://paste.openstack.org/show/badVIJc0hwqbsy5X3Iv3/ [4] https://zuul.opendev.org/t/openstack/build/0be7ba6d8f854c0a88df9f9e33461d20/...
Hi Dmitriy, Could you please share the stack trace after setting swift_store_retry_get_count to a non-zero value. It will help me to debug further. Thanks & Best Regards, Abhishek Kekane On Fri, Mar 27, 2026 at 9:17 PM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hi Abhishek!
Thanks for the prompt reply! While I think you're on spot with the diagnosis, I am not sure if the path is actually working.
I tried to apply the patch, while having `swift_store_retry_get_count = 5` in the glance-api.conf, and I had exactly the same stack trace, as without the patch. However, setting swift_store_retry_get_count to 0 has solved the problem indeed.
According to what I see in my test environment, your patch solves the issue only when swift_store_retry_get_count is not defined at all, but not when it's defined to non-zero value.
пт, 27 мар. 2026 г. в 16:24, Abhishek Kekane <akekane@redhat.com>:
hi Dmitriy,
The mentioned bug is reported in LP [1] and I have pushed a fix [2] for it. As a workaround you can set swift_store_retry_get_count to 0 in the config file.
Let me know if mentioned fix [2] works for you.
[1] https://bugs.launchpad.net/glance/+bug/2146432 [2] https://review.opendev.org/c/openstack/glance_store/+/982221
Thanks & Best Regards,
Abhishek Kekane
On Fri, Mar 27, 2026 at 8:50 PM Dmitriy Rabotyagov < noonedeadpunk@gmail.com> wrote:
Hi folks,
I am trying to test out Glance configured with Swift backend on 2026.1 prior to the release, and facing some issues. As we are super close to the release, I have decided to pop up with the ML, to ensure it's a "me" problem, and not a generic bug somewhere.
So I am using: - Glance installed from SHA 07dd0a203b6883efa66486035c436b92c83da0f3 - glance-store 5.4.0 (according to U-C) - Swift installed from SHA 51b068309a626d2228ab69b5158ee857e96d101d
Image upload works quite well [1], image is appearing in Glance and in Swift quite correctly. But whenever I attempt to create an instance from the image, or, basically, just download image with `openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008`,I get Glance WSGI crashing, resulting in 502: root@aio1-utility-container-20f6bf88:/# openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008 Unable to download image: checksum mismatch (sha512): 9a9bce0083a00939ec17c11febbfc767aa211aaa54f51e75c5a8b271a9b5637c77205a518b7a2007cb391d23cceb01e0e4e8d64832317151bc85b734b92a7be0 != a3b1ae48ab757e1e074c33c3159cb3bc749d0119677446b442 c5568eeaf2a4d741d794ca9750e13c3352b7db073d0f9101670f3cbede942df3f5fdb12024aaca
<html><body><h1>502 Bad Gateway</h1> The server returned an invalid or incomplete response. </body></html> root@aio1-utility-container-20f6bf88:/#
The actual crash and stack trace can be seen here [2]. With that, I can download the image with Swift client without any issues [3], and checksum matches.
This result is pretty much reproducible, and I get it constantly in OpenStack-Ansible's CI jobs, for example [4]
With that, I have the impression that something goes terribly wrong on the Glance side of things in combination with Swift, as CI jobs using Ceph or File backends are working normally.
Given where we are release-wise, I'd love to ask for some help investigating this problem...
[1] https://paste.openstack.org/show/bjdhtY2p33L4g0CTTdqH/ [2] https://paste.openstack.org/show/bRhFaEdoJotDngNXBjlB/ [3] https://paste.openstack.org/show/badVIJc0hwqbsy5X3Iv3/ [4] https://zuul.opendev.org/t/openstack/build/0be7ba6d8f854c0a88df9f9e33461d20/...
Sorry for the delay, Friday evening went too well :) So, I have tested a new iteration of your patch [1] and it looks like it does the trick for me now. The only thing I see in logs is actually a transaction error in SQL [2], as I have just `rm /var/lib/glance/cache/*` to ensure that cache is not used, but that is a completely different topic. And, I guess, the transaction is being retried, as the image is returned without issues. So thanks a lot for such a quick response and patch! [1] https://review.opendev.org/c/openstack/glance_store/+/982221/1..2 [2] https://paste.openstack.org/show/bHz8WqPQt0Xzv4yJkGVl/ пт, 27 мар. 2026 г. в 17:53, Abhishek Kekane <akekane@redhat.com>:
Hi Dmitriy,
Could you please share the stack trace after setting swift_store_retry_get_count to a non-zero value. It will help me to debug further.
Thanks & Best Regards,
Abhishek Kekane
On Fri, Mar 27, 2026 at 9:17 PM Dmitriy Rabotyagov < noonedeadpunk@gmail.com> wrote:
Hi Abhishek!
Thanks for the prompt reply! While I think you're on spot with the diagnosis, I am not sure if the path is actually working.
I tried to apply the patch, while having `swift_store_retry_get_count = 5` in the glance-api.conf, and I had exactly the same stack trace, as without the patch. However, setting swift_store_retry_get_count to 0 has solved the problem indeed.
According to what I see in my test environment, your patch solves the issue only when swift_store_retry_get_count is not defined at all, but not when it's defined to non-zero value.
пт, 27 мар. 2026 г. в 16:24, Abhishek Kekane <akekane@redhat.com>:
hi Dmitriy,
The mentioned bug is reported in LP [1] and I have pushed a fix [2] for it. As a workaround you can set swift_store_retry_get_count to 0 in the config file.
Let me know if mentioned fix [2] works for you.
[1] https://bugs.launchpad.net/glance/+bug/2146432 [2] https://review.opendev.org/c/openstack/glance_store/+/982221
Thanks & Best Regards,
Abhishek Kekane
On Fri, Mar 27, 2026 at 8:50 PM Dmitriy Rabotyagov < noonedeadpunk@gmail.com> wrote:
Hi folks,
I am trying to test out Glance configured with Swift backend on 2026.1 prior to the release, and facing some issues. As we are super close to the release, I have decided to pop up with the ML, to ensure it's a "me" problem, and not a generic bug somewhere.
So I am using: - Glance installed from SHA 07dd0a203b6883efa66486035c436b92c83da0f3 - glance-store 5.4.0 (according to U-C) - Swift installed from SHA 51b068309a626d2228ab69b5158ee857e96d101d
Image upload works quite well [1], image is appearing in Glance and in Swift quite correctly. But whenever I attempt to create an instance from the image, or, basically, just download image with `openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008`,I get Glance WSGI crashing, resulting in 502: root@aio1-utility-container-20f6bf88:/# openstack image save a024ca39-42bc-4a0c-bd34-26209bda4008 Unable to download image: checksum mismatch (sha512): 9a9bce0083a00939ec17c11febbfc767aa211aaa54f51e75c5a8b271a9b5637c77205a518b7a2007cb391d23cceb01e0e4e8d64832317151bc85b734b92a7be0 != a3b1ae48ab757e1e074c33c3159cb3bc749d0119677446b442 c5568eeaf2a4d741d794ca9750e13c3352b7db073d0f9101670f3cbede942df3f5fdb12024aaca
<html><body><h1>502 Bad Gateway</h1> The server returned an invalid or incomplete response. </body></html> root@aio1-utility-container-20f6bf88:/#
The actual crash and stack trace can be seen here [2]. With that, I can download the image with Swift client without any issues [3], and checksum matches.
This result is pretty much reproducible, and I get it constantly in OpenStack-Ansible's CI jobs, for example [4]
With that, I have the impression that something goes terribly wrong on the Glance side of things in combination with Swift, as CI jobs using Ceph or File backends are working normally.
Given where we are release-wise, I'd love to ask for some help investigating this problem...
[1] https://paste.openstack.org/show/bjdhtY2p33L4g0CTTdqH/ [2] https://paste.openstack.org/show/bRhFaEdoJotDngNXBjlB/ [3] https://paste.openstack.org/show/badVIJc0hwqbsy5X3Iv3/ [4] https://zuul.opendev.org/t/openstack/build/0be7ba6d8f854c0a88df9f9e33461d20/...
participants (2)
-
Abhishek Kekane
-
Dmitriy Rabotyagov