[nova][glance] nova-lvm job constantly fails on glance image upload
Hi, I recently spotted a constant failure on the nova-lvm job [1] which seems to occur since July-19 at least, and which prevents us to merge all changes touching the nova-libvirt code section [2]
Evidences show a glance image upload HTTP 502 failure when we want to snapshot [3] which corresponds to the Glance API getting issues with staged image data not found [4]
Until we figure out how to solve this, recheck against nova changes touching nova/virt/libvirt/ module are useless. Any help would be appreciated on debugging the Glance failure, tho.
Thanks, -Sylvain
[1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master [2] https://zuul.openstack.org/job/nova-lvm [3] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contro... [4] https://zuul.openstack.org/build/a04b0d2e184a4d3e9c39bd6110b454ea/log/contro...
On Tue, Aug 3, 2021 at 10:21 AM Sylvain Bauza sbauza@redhat.com wrote:
Evidences show a glance image upload HTTP 502 failure when we want to snapshot [3] which corresponds to the Glance API getting issues with staged image data not found [4]
To be clear, and as I said in the #openstack-nova channel, correlation isn't consequence. We are sure that the job is failing *because* Glance image upload fails, but we don't know yet *why*.
In the meantime, I filed a bug https://bugs.launchpad.net/nova/+bug/1938765
On 03-08-21 10:48:21, Sylvain Bauza wrote:
On Tue, Aug 3, 2021 at 10:21 AM Sylvain Bauza sbauza@redhat.com wrote:
Evidences show a glance image upload HTTP 502 failure when we want to snapshot [3] which corresponds to the Glance API getting issues with staged image data not found [4]
To be clear, and as I said in the #openstack-nova channel, correlation isn't consequence. We are sure that the job is failing *because* Glance image upload fails, but we don't know yet *why*.
In the meantime, I filed a bug https://bugs.launchpad.net/nova/+bug/1938765
Looks like a simple case of us needing to increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL to accommodate the RAW snapshots created by tests within the job. I've posted the following:
zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm https://review.opendev.org/c/openstack/nova/+/803322
Cheers,
On 03-08-21 10:14:23, Lee Yarwood wrote:
On 03-08-21 10:48:21, Sylvain Bauza wrote:
On Tue, Aug 3, 2021 at 10:21 AM Sylvain Bauza sbauza@redhat.com wrote:
Evidences show a glance image upload HTTP 502 failure when we want to snapshot [3] which corresponds to the Glance API getting issues with staged image data not found [4]
To be clear, and as I said in the #openstack-nova channel, correlation isn't consequence. We are sure that the job is failing *because* Glance image upload fails, but we don't know yet *why*.
In the meantime, I filed a bug https://bugs.launchpad.net/nova/+bug/1938765
Looks like a simple case of us needing to increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL to accommodate the RAW snapshots created by tests within the job. I've posted the following:
zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm https://review.opendev.org/c/openstack/nova/+/803322
This is now blocked by the following grenade fix if any cores have time:
Unblock Gate: create encryption type with right params https://review.opendev.org/c/openstack/grenade/+/803317
Thanks in advance!
participants (2)
-
Lee Yarwood
-
Sylvain Bauza