On Wed, Jan 19, 2022 at 3:49 PM Martin Kopec <mkopec@redhat.com> wrote:
I confirm the pin helped, the tests are passing now:
https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/825183
Although I see a strange error in the master job ^^, 2 tests are failing to create a stack. We noticed this some time ago and the tests have been failing consistently since. It's strange that only master job is affected. Any idea what could be causing that?
https://zuul.opendev.org/t/openstack/build/c9568ac2e4f140f8b2ff461e2b1f5030


AFAICT it could not build  instances with below failures in nova/neutron logs. 

Jan 19 00:41:45.839386 ubuntu-focal-ovh-bhs1-0028063966 nova-compute[111088]: ERROR nova.compute.manager [instance: 862d3350-f712-466c-87d2-b82299b06755] nova.exception.PortBindingFailed: Binding failed for port 7dcc5c11-3e32-491c-8615-044ae78c1507, please check neutron logs for more information.

Jan 19 00:41:45.006133 ubuntu-focal-ovh-bhs1-0028063966 neutron-server[105341]: ERROR neutron.plugins.ml2.managers [req-2a879dd1-4da2-435d-a4f9-39d91cde831e req-c57b35fb-525f-4311-9dca-8b0cee02b1d9 service neutron] Port 7dcc5c11-3e32-491c-8615-044ae78c1507 does not have an IP address assigned and there are no driver with 'connectivity' = 'l2'. The port cannot be bound.


These tests are running fine in the heat gate and the only difference I could see is that you've tls_proxy enabled in devstack [1] (used with ovn) which we don't[2]. Probably some configuration issue that neutron folks could help.

[1]
ovn_sb_connection = ssl:158.69.73.104:6642
ovn_nb_connection = ssl:158.69.73.104:6641

 [2]
ovn_sb_connection = tcp:158.69.74.7:6642
ovn_nb_connection = tcp:158.69.74.7:6641

On Wed, 19 Jan 2022 at 10:17, Takashi Kajinami <tkajinam@redhat.com> wrote:
The issue triggered by gabbi 2.5.0 was temporarily resolved by pinning it to 2.4.0 .
The gate is unblocked, I believe (We found an issue with a functional job
for stable/train which is being fixed now).
We can remove that pin once the issue with urllib3 is resolved.

> Should we consider using only https for endpoint access in the future?
Maybe ? But this is a separate topic, IMHO, and I don't have any strong
opinion about this.
There are several devstack plugins (like heat, aodh, ...) which don't support
setting up tls-proxy for https endpoints yet and we need to fix each plugin
if we take this direction.


On Tue, Jan 18, 2022 at 10:21 PM Martin Kopec <mkopec@redhat.com> wrote:
Thank you Takashi for looking into this.

Should we consider using only https for endpoint access in the future?

On Tue, 18 Jan 2022 at 09:35, Takashi Kajinami <tkajinam@redhat.com> wrote:
I have opened an issue for urllib3[1], too, and created a PR to discuss a potential fix.

Because it'd take some time until we get feedback from these two communities,
I've proposed a change to pin gabbi to 2.4.0[2].

The issue might affect other projects using gabbi as well, unless https, instead of http,
is used for endpoint access.


On Tue, Jan 18, 2022 at 11:42 AM Takashi Kajinami <tkajinam@redhat.com> wrote:
I've looked into this but it seems the following error was actually caused by the latest release of gabbi(2.5.0).
 TypeError: __init__() got an unexpected keyword argument 'server_hostname'

I've reported that issue to gabbi in [1] but if my observation is correct the problem should be
fixed in urllib3 which gabbi is dependent on.

On Tue, Jan 18, 2022 at 2:34 AM Martin Kopec <mkopec@redhat.com> wrote:
Hello,

most of the tests of heat-tempest-plugin have started failing. We noticed that in interop [1], however, we reproduced that in the project's gates as well [2].

I suspect it might be an issue with the new gabbi package - there has been an update recently.
Could you please have a look.


Thank you,
--
Martin Kopec
Senior Software Quality Engineer
Red Hat EMEA





--
Martin


--
Martin


--
Regards,
Rabi Mishra