Yes, I've seen a failure with keystone as well today, and in some of my CI runs devstack itself has been OK. But in the context of a whole devstack setup, it seems there's currently a flake which is very likely to prevent the setup as a whole from succeeding.

On Fri, Sep 30, 2022 at 12:07 PM Lajos Katona katonalala@gmail.com wrote:

Hi, I had the same with keystone, and pulling Neutron, but strangely pulling neutron-lib (and networking-bagpipe also) worked...

Lajos

Artem Goncharov artem.goncharov@gmail.com ezt írta (időpont: 2022. szept. 30., P, 12:48):

...

I experience similar (but not same) error cloning freshly SDK repo

[linux@demo ~]$ git clone https://opendev.org/openstack/openstacksdk Cloning into 'openstacksdk'... remote: Enumerating objects: 22680, done. remote: Counting objects: 100% (22680/22680), done. remote: Compressing objects: 100% (4690/4690), done. error: RPC failed; curl 18 transfer closed with outstanding read data remaining fatal: early EOF fatal: fetch-pack: invalid index-pack output

Artem

On 30. Sep 2022, at 12:09, Neil Jerram neil@tigera.io wrote:

Anyone else seeing this?

neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output

Best wishes, Neil

-- Neil Jerram Principal Software Engineer Tigera neil@tigera.io | @neiljerram http://twitter.com/neiljerram Follow Tigera: Blog https://blog.tigera.io/ | Twitter https://twitter.com/tigeraio | LinkedIn https://www.linkedin.com/company/tigera/

Security and observability for containers, Kubernetes, and cloud https://tigera.io/

For me it seems to be still ongoing. I have been seeing these errors in CI runs from <subdomain>.semaphoreci.com. There have been 6 occurrences in the last 5 hours, all when cloning either devstack or keystone, and all with an error like "error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received." I've just kicked off a new run just now, and it has got past cloning devstack - I'll write again about how that goes.

On Fri, Sep 30, 2022 at 1:53 PM Jeremy Stanley fungi@yuggoth.org wrote:

On 2022-09-30 11:09:27 +0100 (+0100), Neil Jerram wrote:

...

Anyone else seeing this?

neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output

[...]

I've tested each of our 8 backends individually by cloning devstack from them directly, and am not seeing any errors. If there is a problem with one (or more) of them, it's not consistent. Our cacti graphs show no significant spikes in activity nor resource consumption on the backends or the load balancer around the time the first message about this was posted.

Is the problem ongoing at this point? Persistent or inconsistent? Are there any commonalities for the people/sites where the errors are being observed? -- Jeremy Stanley

On Fri, Sep 30, 2022 at 2:05 PM Neil Jerram neil@tigera.io wrote:

For me it seems to be still ongoing. I have been seeing these errors in CI runs from <subdomain>.semaphoreci.com. There have been 6 occurrences in the last 5 hours, all when cloning either devstack or keystone, and all with an error like "error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received." I've just kicked off a new run just now, and it has got past cloning devstack - I'll write again about how that goes.

That test actually passed.

But then I kicked off two more, and the first of those failed with

Cloning into '/opt/stack/neutron'... error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. fatal: early EOF fatal: fetch-pack: invalid index-pack output

The second is still running, unusually slowly. It appears to be in the middle of cloning Nova - possibly hung there.

Best wishes, Neil

On Fri, Sep 30, 2022 at 1:53 PM Jeremy Stanley fungi@yuggoth.org wrote:

...

On 2022-09-30 11:09:27 +0100 (+0100), Neil Jerram wrote:

...

Anyone else seeing this?

neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output

[...]

I've tested each of our 8 backends individually by cloning devstack from them directly, and am not seeing any errors. If there is a problem with one (or more) of them, it's not consistent. Our cacti graphs show no significant spikes in activity nor resource consumption on the backends or the load balancer around the time the first message about this was posted.

Is the problem ongoing at this point? Persistent or inconsistent? Are there any commonalities for the people/sites where the errors are being observed? -- Jeremy Stanley

On Fri, Sep 30, 2022 at 3:05 PM Jeremy Stanley fungi@yuggoth.org wrote:

On 2022-09-30 14:59:21 +0100 (+0100), Neil Jerram wrote: [...]

...

That test actually passed.

But then I kicked off two more, and the first of those failed with

Cloning into '/opt/stack/neutron'... error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. fatal: early EOF fatal: fetch-pack: invalid index-pack output

The second is still running, unusually slowly. It appears to be in the middle of cloning Nova - possibly hung there.

[...]

I ran repeated devstack clones in a tight loop for roughly half an hour from my home network and never had a single failure. Is it possible all the people experiencing this issue may be coming from the same Internet provider or the same region of the World?

For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as in my report that started this thread. My ISP is Shell Energy.

I just tried again from my laptop, and now the clone has succeeded.

On Fri, Sep 30, 2022 at 4:09 PM Jeremy Stanley fungi@yuggoth.org wrote:

On 2022-09-30 15:53:22 +0100 (+0100), Neil Jerram wrote: [...]

...

For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as

in

...

my report that started this thread. My ISP is Shell Energy.

I just tried again from my laptop, and now the clone has succeeded.

Our server farm for Git is generously hosted within VEXXHOST's San Jose USA public cloud region, so it's possible something is causing disconnects reaching it from some parts of the 'net and not others. I'm connecting to it from Charter Cable in the USA, who peers with Zayo (formerly AboveNet) in Atlanta, and then VEXXHOST peers with them in San Jose. Not exactly next door, but I'm only traversing one backbone provider to get there (and with access to the server I've confirmed the return route is mostly symmetrical though comes back through a Zayo/Charter peering in Washington DC).

To rule out IP protocol differences, I've done repeated git clone tests over both IPv4 and IPv6 too, with no observable issues.

Thanks for thinking about and investigating this, Jeremy. Apparently my team's Semaphore servers are based in Germany, so there could be a transatlantic factor here. Perhaps we just need to wait now for any further reports.

Best wishes, Neil

-- Jeremy Stanley

My trace route shows that once my traffic enters Zayo network it becomes terribly unstable and most rechecks are even showing different routes on their side. Apparently EU-US connection is going now this way and is therefore affected.

On 30. Sep 2022, at 17:07, Jeremy Stanley fungi@yuggoth.org wrote:

On 2022-09-30 15:53:22 +0100 (+0100), Neil Jerram wrote: [...]

...

For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as in my report that started this thread. My ISP is Shell Energy.

I just tried again from my laptop, and now the clone has succeeded.

Our server farm for Git is generously hosted within VEXXHOST's San Jose USA public cloud region, so it's possible something is causing disconnects reaching it from some parts of the 'net and not others. I'm connecting to it from Charter Cable in the USA, who peers with Zayo (formerly AboveNet) in Atlanta, and then VEXXHOST peers with them in San Jose. Not exactly next door, but I'm only traversing one backbone provider to get there (and with access to the server I've confirmed the return route is mostly symmetrical though comes back through a Zayo/Charter peering in Washington DC).

To rule out IP protocol differences, I've done repeated git clone tests over both IPv4 and IPv6 too, with no observable issues. -- Jeremy Stanley