TLS error when cloning https://opendev.org/openstack/devstack
Anyone else seeing this? neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output Best wishes, Neil -- Neil Jerram Principal Software Engineer Tigera neil@tigera.io | @neiljerram <http://twitter.com/neiljerram> Follow Tigera: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/> Security and observability for containers, Kubernetes, and cloud <https://tigera.io/>
I experience similar (but not same) error cloning freshly SDK repo [linux@demo ~]$ git clone https://opendev.org/openstack/openstacksdk Cloning into 'openstacksdk'... remote: Enumerating objects: 22680, done. remote: Counting objects: 100% (22680/22680), done. remote: Compressing objects: 100% (4690/4690), done. error: RPC failed; curl 18 transfer closed with outstanding read data remaining fatal: early EOF fatal: fetch-pack: invalid index-pack output Artem
On 30. Sep 2022, at 12:09, Neil Jerram <neil@tigera.io> wrote:
Anyone else seeing this?
neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack <https://opendev.org/openstack/devstack> Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output
Best wishes, Neil
-- Neil Jerram Principal Software Engineer Tigera neil@tigera.io <mailto:neil@tigera.io> | @neiljerram <http://twitter.com/neiljerram> Follow Tigera: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/> Security and observability for containers, Kubernetes, and cloud <https://tigera.io/>
Hi, I had the same with keystone, and pulling Neutron, but strangely pulling neutron-lib (and networking-bagpipe also) worked... Lajos Artem Goncharov <artem.goncharov@gmail.com> ezt írta (időpont: 2022. szept. 30., P, 12:48):
I experience similar (but not same) error cloning freshly SDK repo
[linux@demo ~]$ git clone https://opendev.org/openstack/openstacksdk Cloning into 'openstacksdk'... remote: Enumerating objects: 22680, done. remote: Counting objects: 100% (22680/22680), done. remote: Compressing objects: 100% (4690/4690), done. error: RPC failed; curl 18 transfer closed with outstanding read data remaining fatal: early EOF fatal: fetch-pack: invalid index-pack output
Artem
On 30. Sep 2022, at 12:09, Neil Jerram <neil@tigera.io> wrote:
Anyone else seeing this?
neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output
Best wishes, Neil
-- Neil Jerram Principal Software Engineer Tigera neil@tigera.io | @neiljerram <http://twitter.com/neiljerram> Follow Tigera: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/>
Security and observability for containers, Kubernetes, and cloud <https://tigera.io/>
Yes, I've seen a failure with keystone as well today, and in some of my CI runs devstack itself has been OK. But in the context of a whole devstack setup, it seems there's currently a flake which is very likely to prevent the setup as a whole from succeeding. On Fri, Sep 30, 2022 at 12:07 PM Lajos Katona <katonalala@gmail.com> wrote:
Hi, I had the same with keystone, and pulling Neutron, but strangely pulling neutron-lib (and networking-bagpipe also) worked...
Lajos
Artem Goncharov <artem.goncharov@gmail.com> ezt írta (időpont: 2022. szept. 30., P, 12:48):
I experience similar (but not same) error cloning freshly SDK repo
[linux@demo ~]$ git clone https://opendev.org/openstack/openstacksdk Cloning into 'openstacksdk'... remote: Enumerating objects: 22680, done. remote: Counting objects: 100% (22680/22680), done. remote: Compressing objects: 100% (4690/4690), done. error: RPC failed; curl 18 transfer closed with outstanding read data remaining fatal: early EOF fatal: fetch-pack: invalid index-pack output
Artem
On 30. Sep 2022, at 12:09, Neil Jerram <neil@tigera.io> wrote:
Anyone else seeing this?
neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output
Best wishes, Neil
-- Neil Jerram Principal Software Engineer Tigera neil@tigera.io | @neiljerram <http://twitter.com/neiljerram> Follow Tigera: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/>
Security and observability for containers, Kubernetes, and cloud <https://tigera.io/>
On 2022-09-30 11:09:27 +0100 (+0100), Neil Jerram wrote:
Anyone else seeing this?
neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output [...]
I've tested each of our 8 backends individually by cloning devstack from them directly, and am not seeing any errors. If there is a problem with one (or more) of them, it's not consistent. Our cacti graphs show no significant spikes in activity nor resource consumption on the backends or the load balancer around the time the first message about this was posted. Is the problem ongoing at this point? Persistent or inconsistent? Are there any commonalities for the people/sites where the errors are being observed? -- Jeremy Stanley
For me it seems to be still ongoing. I have been seeing these errors in CI runs from <subdomain>.semaphoreci.com. There have been 6 occurrences in the last 5 hours, all when cloning either devstack or keystone, and all with an error like "error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received." I've just kicked off a new run just now, and it has got past cloning devstack - I'll write again about how that goes. On Fri, Sep 30, 2022 at 1:53 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-09-30 11:09:27 +0100 (+0100), Neil Jerram wrote:
Anyone else seeing this?
neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output [...]
I've tested each of our 8 backends individually by cloning devstack from them directly, and am not seeing any errors. If there is a problem with one (or more) of them, it's not consistent. Our cacti graphs show no significant spikes in activity nor resource consumption on the backends or the load balancer around the time the first message about this was posted.
Is the problem ongoing at this point? Persistent or inconsistent? Are there any commonalities for the people/sites where the errors are being observed? -- Jeremy Stanley
Hi, For me it seems things are ok now. Thanks for checking. Lajos Neil Jerram <neil@tigera.io> ezt írta (időpont: 2022. szept. 30., P, 15:16):
For me it seems to be still ongoing. I have been seeing these errors in CI runs from <subdomain>.semaphoreci.com. There have been 6 occurrences in the last 5 hours, all when cloning either devstack or keystone, and all with an error like "error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received." I've just kicked off a new run just now, and it has got past cloning devstack - I'll write again about how that goes.
On Fri, Sep 30, 2022 at 1:53 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-09-30 11:09:27 +0100 (+0100), Neil Jerram wrote:
Anyone else seeing this?
neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output [...]
I've tested each of our 8 backends individually by cloning devstack from them directly, and am not seeing any errors. If there is a problem with one (or more) of them, it's not consistent. Our cacti graphs show no significant spikes in activity nor resource consumption on the backends or the load balancer around the time the first message about this was posted.
Is the problem ongoing at this point? Persistent or inconsistent? Are there any commonalities for the people/sites where the errors are being observed? -- Jeremy Stanley
On Fri, Sep 30, 2022 at 2:05 PM Neil Jerram <neil@tigera.io> wrote:
For me it seems to be still ongoing. I have been seeing these errors in CI runs from <subdomain>.semaphoreci.com. There have been 6 occurrences in the last 5 hours, all when cloning either devstack or keystone, and all with an error like "error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received." I've just kicked off a new run just now, and it has got past cloning devstack - I'll write again about how that goes.
That test actually passed. But then I kicked off two more, and the first of those failed with Cloning into '/opt/stack/neutron'... error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. fatal: early EOF fatal: fetch-pack: invalid index-pack output The second is still running, unusually slowly. It appears to be in the middle of cloning Nova - possibly hung there. Best wishes, Neil
On Fri, Sep 30, 2022 at 1:53 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-09-30 11:09:27 +0100 (+0100), Neil Jerram wrote:
Anyone else seeing this?
neil@glaurung:~/tt$ git clone https://opendev.org/openstack/devstack Cloning into 'devstack'... remote: Enumerating objects: 28589, done. remote: Counting objects: 100% (28589/28589), done. remote: Compressing objects: 100% (9638/9638), done. error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. fatal: early EOF fatal: fetch-pack: invalid index-pack output [...]
I've tested each of our 8 backends individually by cloning devstack from them directly, and am not seeing any errors. If there is a problem with one (or more) of them, it's not consistent. Our cacti graphs show no significant spikes in activity nor resource consumption on the backends or the load balancer around the time the first message about this was posted.
Is the problem ongoing at this point? Persistent or inconsistent? Are there any commonalities for the people/sites where the errors are being observed? -- Jeremy Stanley
On 2022-09-30 14:59:21 +0100 (+0100), Neil Jerram wrote: [...]
That test actually passed.
But then I kicked off two more, and the first of those failed with
Cloning into '/opt/stack/neutron'... error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. fatal: early EOF fatal: fetch-pack: invalid index-pack output
The second is still running, unusually slowly. It appears to be in the middle of cloning Nova - possibly hung there. [...]
I ran repeated devstack clones in a tight loop for roughly half an hour from my home network and never had a single failure. Is it possible all the people experiencing this issue may be coming from the same Internet provider or the same region of the World? -- Jeremy Stanley
On Fri, Sep 30, 2022 at 3:05 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-09-30 14:59:21 +0100 (+0100), Neil Jerram wrote: [...]
That test actually passed.
But then I kicked off two more, and the first of those failed with
Cloning into '/opt/stack/neutron'... error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. fatal: early EOF fatal: fetch-pack: invalid index-pack output
The second is still running, unusually slowly. It appears to be in the middle of cloning Nova - possibly hung there. [...]
I ran repeated devstack clones in a tight loop for roughly half an hour from my home network and never had a single failure. Is it possible all the people experiencing this issue may be coming from the same Internet provider or the same region of the World?
For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as in my report that started this thread. My ISP is Shell Energy. I just tried again from my laptop, and now the clone has succeeded. --
Jeremy Stanley
On 2022-09-30 15:53:22 +0100 (+0100), Neil Jerram wrote: [...]
For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as in my report that started this thread. My ISP is Shell Energy.
I just tried again from my laptop, and now the clone has succeeded.
Our server farm for Git is generously hosted within VEXXHOST's San Jose USA public cloud region, so it's possible something is causing disconnects reaching it from some parts of the 'net and not others. I'm connecting to it from Charter Cable in the USA, who peers with Zayo (formerly AboveNet) in Atlanta, and then VEXXHOST peers with them in San Jose. Not exactly next door, but I'm only traversing one backbone provider to get there (and with access to the server I've confirmed the return route is mostly symmetrical though comes back through a Zayo/Charter peering in Washington DC). To rule out IP protocol differences, I've done repeated git clone tests over both IPv4 and IPv6 too, with no observable issues. -- Jeremy Stanley
On Fri, Sep 30, 2022 at 4:09 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-09-30 15:53:22 +0100 (+0100), Neil Jerram wrote: [...]
For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as in my report that started this thread. My ISP is Shell Energy.
I just tried again from my laptop, and now the clone has succeeded.
Our server farm for Git is generously hosted within VEXXHOST's San Jose USA public cloud region, so it's possible something is causing disconnects reaching it from some parts of the 'net and not others. I'm connecting to it from Charter Cable in the USA, who peers with Zayo (formerly AboveNet) in Atlanta, and then VEXXHOST peers with them in San Jose. Not exactly next door, but I'm only traversing one backbone provider to get there (and with access to the server I've confirmed the return route is mostly symmetrical though comes back through a Zayo/Charter peering in Washington DC).
To rule out IP protocol differences, I've done repeated git clone tests over both IPv4 and IPv6 too, with no observable issues.
Thanks for thinking about and investigating this, Jeremy. Apparently my team's Semaphore servers are based in Germany, so there could be a transatlantic factor here. Perhaps we just need to wait now for any further reports. Best wishes, Neil
-- Jeremy Stanley
On 2022-09-30 16:15:36 +0100 (+0100), Neil Jerram wrote: [...]
Thanks for thinking about and investigating this, Jeremy. Apparently my team's Semaphore servers are based in Germany, so there could be a transatlantic factor here. Perhaps we just need to wait now for any further reports.
That mostly confirms what we're finding on the load balancer. Haproxy records numerous prematurely disconnected sessions involving a small number of clients, many of which are from an IP address in Germany, and also some others from Europe. -- Jeremy Stanley
My trace route shows that once my traffic enters Zayo network it becomes terribly unstable and most rechecks are even showing different routes on their side. Apparently EU-US connection is going now this way and is therefore affected.
On 30. Sep 2022, at 17:07, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-09-30 15:53:22 +0100 (+0100), Neil Jerram wrote: [...]
For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as in my report that started this thread. My ISP is Shell Energy.
I just tried again from my laptop, and now the clone has succeeded.
Our server farm for Git is generously hosted within VEXXHOST's San Jose USA public cloud region, so it's possible something is causing disconnects reaching it from some parts of the 'net and not others. I'm connecting to it from Charter Cable in the USA, who peers with Zayo (formerly AboveNet) in Atlanta, and then VEXXHOST peers with them in San Jose. Not exactly next door, but I'm only traversing one backbone provider to get there (and with access to the server I've confirmed the return route is mostly symmetrical though comes back through a Zayo/Charter peering in Washington DC).
To rule out IP protocol differences, I've done repeated git clone tests over both IPv4 and IPv6 too, with no observable issues. -- Jeremy Stanley
I also see the traffic entering Zayo to cross the ocean, but I've been so far unable to get a failure after cloning devstack in a loop. On Fri, Sep 30, 2022 at 5:31 PM Artem Goncharov <artem.goncharov@gmail.com> wrote:
My trace route shows that once my traffic enters Zayo network it becomes terribly unstable and most rechecks are even showing different routes on their side. Apparently EU-US connection is going now this way and is therefore affected.
On 30. Sep 2022, at 17:07, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-09-30 15:53:22 +0100 (+0100), Neil Jerram wrote: [...]
For me it would be coming from wherever Semaphore's machines are. But I also tried from my own laptop, based in Cambridge UK, and that failed as in my report that started this thread. My ISP is Shell Energy.
I just tried again from my laptop, and now the clone has succeeded.
Our server farm for Git is generously hosted within VEXXHOST's San Jose USA public cloud region, so it's possible something is causing disconnects reaching it from some parts of the 'net and not others. I'm connecting to it from Charter Cable in the USA, who peers with Zayo (formerly AboveNet) in Atlanta, and then VEXXHOST peers with them in San Jose. Not exactly next door, but I'm only traversing one backbone provider to get there (and with access to the server I've confirmed the return route is mostly symmetrical though comes back through a Zayo/Charter peering in Washington DC).
To rule out IP protocol differences, I've done repeated git clone tests over both IPv4 and IPv6 too, with no observable issues. -- Jeremy Stanley
-- Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross
participants (5)
-
Artem Goncharov
-
Dmitry Tantsur
-
Jeremy Stanley
-
Lajos Katona
-
Neil Jerram