Hello,
most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].
I suspect it might be an issue with the new gabbi package - there has been an update recently. Could you please have a look.
[1] https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/824832 [2] https://review.opendev.org/c/openstack/heat-tempest-plugin/+/823794
Thank you,
I've looked into this but it seems the following error was actually caused by the latest release of gabbi(2.5.0). TypeError: __init__() got an unexpected keyword argument 'server_hostname'
I've reported that issue to gabbi in [1] but if my observation is correct the problem should be fixed in urllib3 which gabbi is dependent on. [1] https://github.com/cdent/gabbi/issues/309
On Tue, Jan 18, 2022 at 2:34 AM Martin Kopec mkopec@redhat.com wrote:
Hello,
most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].
I suspect it might be an issue with the new gabbi package - there has been an update recently. Could you please have a look.
[1] https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/824832 [2] https://review.opendev.org/c/openstack/heat-tempest-plugin/+/823794
Thank you,
Martin Kopec Senior Software Quality Engineer Red Hat EMEA
I have opened an issue for urllib3[1], too, and created a PR to discuss a potential fix. [1] https://github.com/urllib3/urllib3/issues/2534
Because it'd take some time until we get feedback from these two communities, I've proposed a change to pin gabbi to 2.4.0[2]. [2] https://review.opendev.org/c/openstack/requirements/+/825044
The issue might affect other projects using gabbi as well, unless https, instead of http, is used for endpoint access.
On Tue, Jan 18, 2022 at 11:42 AM Takashi Kajinami tkajinam@redhat.com wrote:
I've looked into this but it seems the following error was actually caused by the latest release of gabbi(2.5.0). TypeError: __init__() got an unexpected keyword argument 'server_hostname'
I've reported that issue to gabbi in [1] but if my observation is correct the problem should be fixed in urllib3 which gabbi is dependent on. [1] https://github.com/cdent/gabbi/issues/309
On Tue, Jan 18, 2022 at 2:34 AM Martin Kopec mkopec@redhat.com wrote:
Hello,
most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].
I suspect it might be an issue with the new gabbi package - there has been an update recently. Could you please have a look.
[1] https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/824832 [2] https://review.opendev.org/c/openstack/heat-tempest-plugin/+/823794
Thank you,
Martin Kopec Senior Software Quality Engineer Red Hat EMEA
Thank you Takashi for looking into this.
Should we consider using only https for endpoint access in the future?
On Tue, 18 Jan 2022 at 09:35, Takashi Kajinami tkajinam@redhat.com wrote:
I have opened an issue for urllib3[1], too, and created a PR to discuss a potential fix. [1] https://github.com/urllib3/urllib3/issues/2534
Because it'd take some time until we get feedback from these two communities, I've proposed a change to pin gabbi to 2.4.0[2]. [2] https://review.opendev.org/c/openstack/requirements/+/825044
The issue might affect other projects using gabbi as well, unless https, instead of http, is used for endpoint access.
On Tue, Jan 18, 2022 at 11:42 AM Takashi Kajinami tkajinam@redhat.com wrote:
I've looked into this but it seems the following error was actually caused by the latest release of gabbi(2.5.0). TypeError: __init__() got an unexpected keyword argument 'server_hostname'
I've reported that issue to gabbi in [1] but if my observation is correct the problem should be fixed in urllib3 which gabbi is dependent on. [1] https://github.com/cdent/gabbi/issues/309
On Tue, Jan 18, 2022 at 2:34 AM Martin Kopec mkopec@redhat.com wrote:
Hello,
most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].
I suspect it might be an issue with the new gabbi package - there has been an update recently. Could you please have a look.
[1] https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/824832 [2] https://review.opendev.org/c/openstack/heat-tempest-plugin/+/823794
Thank you,
Martin Kopec Senior Software Quality Engineer Red Hat EMEA
The issue triggered by gabbi 2.5.0 was temporarily resolved by pinning it to 2.4.0 . The gate is unblocked, I believe (We found an issue with a functional job for stable/train which is being fixed now). We can remove that pin once the issue with urllib3 is resolved.
Should we consider using only https for endpoint access in the future?
Maybe ? But this is a separate topic, IMHO, and I don't have any strong opinion about this. There are several devstack plugins (like heat, aodh, ...) which don't support setting up tls-proxy for https endpoints yet and we need to fix each plugin if we take this direction.
On Tue, Jan 18, 2022 at 10:21 PM Martin Kopec mkopec@redhat.com wrote:
Thank you Takashi for looking into this.
Should we consider using only https for endpoint access in the future?
On Tue, 18 Jan 2022 at 09:35, Takashi Kajinami tkajinam@redhat.com wrote:
I have opened an issue for urllib3[1], too, and created a PR to discuss a potential fix. [1] https://github.com/urllib3/urllib3/issues/2534
Because it'd take some time until we get feedback from these two communities, I've proposed a change to pin gabbi to 2.4.0[2]. [2] https://review.opendev.org/c/openstack/requirements/+/825044
The issue might affect other projects using gabbi as well, unless https, instead of http, is used for endpoint access.
On Tue, Jan 18, 2022 at 11:42 AM Takashi Kajinami tkajinam@redhat.com wrote:
I've looked into this but it seems the following error was actually caused by the latest release of gabbi(2.5.0). TypeError: __init__() got an unexpected keyword argument 'server_hostname'
I've reported that issue to gabbi in [1] but if my observation is correct the problem should be fixed in urllib3 which gabbi is dependent on. [1] https://github.com/cdent/gabbi/issues/309
On Tue, Jan 18, 2022 at 2:34 AM Martin Kopec mkopec@redhat.com wrote:
Hello,
most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].
I suspect it might be an issue with the new gabbi package - there has been an update recently. Could you please have a look.
[1] https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/824832 [2] https://review.opendev.org/c/openstack/heat-tempest-plugin/+/823794
Thank you,
Martin Kopec Senior Software Quality Engineer Red Hat EMEA
-- Martin
I confirm the pin helped, the tests are passing now: https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/825183 Although I see a strange error in the master job ^^, 2 tests are failing to create a stack. We noticed this some time ago and the tests have been failing consistently since. It's strange that only master job is affected. Any idea what could be causing that? https://zuul.opendev.org/t/openstack/build/c9568ac2e4f140f8b2ff461e2b1f5030
On Wed, 19 Jan 2022 at 10:17, Takashi Kajinami tkajinam@redhat.com wrote:
The issue triggered by gabbi 2.5.0 was temporarily resolved by pinning it to 2.4.0 . The gate is unblocked, I believe (We found an issue with a functional job for stable/train which is being fixed now). We can remove that pin once the issue with urllib3 is resolved.
Should we consider using only https for endpoint access in the future?
Maybe ? But this is a separate topic, IMHO, and I don't have any strong opinion about this. There are several devstack plugins (like heat, aodh, ...) which don't support setting up tls-proxy for https endpoints yet and we need to fix each plugin if we take this direction.
On Tue, Jan 18, 2022 at 10:21 PM Martin Kopec mkopec@redhat.com wrote:
Thank you Takashi for looking into this.
Should we consider using only https for endpoint access in the future?
On Tue, 18 Jan 2022 at 09:35, Takashi Kajinami tkajinam@redhat.com wrote:
I have opened an issue for urllib3[1], too, and created a PR to discuss a potential fix. [1] https://github.com/urllib3/urllib3/issues/2534
Because it'd take some time until we get feedback from these two communities, I've proposed a change to pin gabbi to 2.4.0[2]. [2] https://review.opendev.org/c/openstack/requirements/+/825044
The issue might affect other projects using gabbi as well, unless https, instead of http, is used for endpoint access.
On Tue, Jan 18, 2022 at 11:42 AM Takashi Kajinami tkajinam@redhat.com wrote:
I've looked into this but it seems the following error was actually caused by the latest release of gabbi(2.5.0). TypeError: __init__() got an unexpected keyword argument 'server_hostname'
I've reported that issue to gabbi in [1] but if my observation is correct the problem should be fixed in urllib3 which gabbi is dependent on. [1] https://github.com/cdent/gabbi/issues/309
On Tue, Jan 18, 2022 at 2:34 AM Martin Kopec mkopec@redhat.com wrote:
Hello,
most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].
I suspect it might be an issue with the new gabbi package - there has been an update recently. Could you please have a look.
[1] https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/824832 [2] https://review.opendev.org/c/openstack/heat-tempest-plugin/+/823794
Thank you,
Martin Kopec Senior Software Quality Engineer Red Hat EMEA
-- Martin
On Wed, Jan 19, 2022 at 3:49 PM Martin Kopec mkopec@redhat.com wrote:
I confirm the pin helped, the tests are passing now:
https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/825183 Although I see a strange error in the master job ^^, 2 tests are failing to create a stack. We noticed this some time ago and the tests have been failing consistently since. It's strange that only master job is affected. Any idea what could be causing that? https://zuul.opendev.org/t/openstack/build/c9568ac2e4f140f8b2ff461e2b1f5030
AFAICT it could not build instances with below failures in nova/neutron logs.
Jan 19 00:41:45.839386 ubuntu-focal-ovh-bhs1-0028063966 nova-compute[111088]: ERROR nova.compute.manager [instance: 862d3350-f712-466c-87d2-b82299b06755] nova.exception.PortBindingFailed: Binding failed for port 7dcc5c11-3e32-491c-8615-044ae78c1507, please check neutron logs for more information.
Jan 19 00:41:45.006133 ubuntu-focal-ovh-bhs1-0028063966 neutron-server[105341]: ERROR neutron.plugins.ml2.managers [req-2a879dd1-4da2-435d-a4f9-39d91cde831e req-c57b35fb-525f-4311-9dca-8b0cee02b1d9 service neutron] Port 7dcc5c11-3e32-491c-8615-044ae78c1507 does not have an IP address assigned and there are no driver with 'connectivity' = 'l2'. The port cannot be bound.
These tests are running fine in the heat gate and the only difference I could see is that you've tls_proxy enabled in devstack [1] (used with ovn) which we don't[2]. Probably some configuration issue that neutron folks could help.
[1]
ovn_sb_connection = ssl:158.69.73.104:6642 ovn_nb_connection = ssl:158.69.73.104:6641
[2]
ovn_sb_connection = tcp:158.69.74.7:6642 ovn_nb_connection = tcp:158.69.74.7:6641
On Wed, 19 Jan 2022 at 10:17, Takashi Kajinami tkajinam@redhat.com wrote:
The issue triggered by gabbi 2.5.0 was temporarily resolved by pinning it to 2.4.0 . The gate is unblocked, I believe (We found an issue with a functional job for stable/train which is being fixed now). We can remove that pin once the issue with urllib3 is resolved.
Should we consider using only https for endpoint access in the future?
Maybe ? But this is a separate topic, IMHO, and I don't have any strong opinion about this. There are several devstack plugins (like heat, aodh, ...) which don't support setting up tls-proxy for https endpoints yet and we need to fix each plugin if we take this direction.
On Tue, Jan 18, 2022 at 10:21 PM Martin Kopec mkopec@redhat.com wrote:
Thank you Takashi for looking into this.
Should we consider using only https for endpoint access in the future?
On Tue, 18 Jan 2022 at 09:35, Takashi Kajinami tkajinam@redhat.com wrote:
I have opened an issue for urllib3[1], too, and created a PR to discuss a potential fix. [1] https://github.com/urllib3/urllib3/issues/2534
Because it'd take some time until we get feedback from these two communities, I've proposed a change to pin gabbi to 2.4.0[2]. [2] https://review.opendev.org/c/openstack/requirements/+/825044
The issue might affect other projects using gabbi as well, unless https, instead of http, is used for endpoint access.
On Tue, Jan 18, 2022 at 11:42 AM Takashi Kajinami tkajinam@redhat.com wrote:
I've looked into this but it seems the following error was actually caused by the latest release of gabbi(2.5.0). TypeError: __init__() got an unexpected keyword argument 'server_hostname'
I've reported that issue to gabbi in [1] but if my observation is correct the problem should be fixed in urllib3 which gabbi is dependent on. [1] https://github.com/cdent/gabbi/issues/309
On Tue, Jan 18, 2022 at 2:34 AM Martin Kopec mkopec@redhat.com wrote:
Hello,
most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].
I suspect it might be an issue with the new gabbi package - there has been an update recently. Could you please have a look.
[1] https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/824832 [2] https://review.opendev.org/c/openstack/heat-tempest-plugin/+/823794
Thank you,
Martin Kopec Senior Software Quality Engineer Red Hat EMEA
-- Martin
-- Martin
participants (3)
-
Martin Kopec
-
Rabi Mishra
-
Takashi Kajinami