[opendev] git clone fails: "The TLS connection was non-properly terminated"
There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem? This happens around midnight between July 22 and 23 UTC. $ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded /git.openstack.org/ strings with the equivalent /github.com/ URL, but I am not sure if I can trust that all Github repos are up to date. Bernd.
On Mon, Jul 22, 2019, at 5:09 PM, Bernd Bausch wrote:
There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem?
This happens around midnight between July 22 and 23 UTC.
$ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated.
Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded *git.openstack.org* strings with the equivalent *github.com* URL, but I am not sure if I can trust that all Github repos are up to date.
Bernd.
I think this may have happened because because one of the backend servers was removed from the load balancer rotation. And while we thought we had removed that server from the haproxy config [0] that didn't happen because our ansible doesn't know how to gracefully restart haproxy in a container. At this point haproxy should be aware that the server is gone and it will prevent any new connections from going to it. All that to say I believe this should be a one off. We'll have to be more careful to ensure backends are rotated off gracefully when replacing them. If you see this persist that information would be very useful. Thank you. [0] https://review.opendev.org/#/c/672083/
On 2019-07-22 17:17:08 -0700 (-0700), Clark Boylan wrote:
There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem?
This happens around midnight between July 22 and 23 UTC.
$ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. [...] I think this may have happened because because one of the backend servers was removed from the load balancer rotation. And while we
On Mon, Jul 22, 2019, at 5:09 PM, Bernd Bausch wrote: thought we had removed that server from the haproxy config [0] that didn't happen because our ansible doesn't know how to gracefully restart haproxy in a container. At this point haproxy should be aware that the server is gone and it will prevent any new connections from going to it. [...]
Yep, sorry, we commented gitea01 out of the load balancer and I subsequently deleted its server instance at 2019-07-22 23:25 UTC, but I neglected to double-check that haproxy had actually stopped distributing connections to it. That one's on me, seems we still have some work to do making sure haproxy config changes are applied automatically there. -- Jeremy Stanley
Thanks much, gentlemen. I did hope this was a glitch on the server side, and indeed, I successfully cloned Devstack about 12 hours ago. Bernd On 7/23/2019 9:53 AM, Jeremy Stanley wrote:
On 2019-07-22 17:17:08 -0700 (-0700), Clark Boylan wrote:
There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem?
This happens around midnight between July 22 and 23 UTC.
$ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. [...] I think this may have happened because because one of the backend servers was removed from the load balancer rotation. And while we
On Mon, Jul 22, 2019, at 5:09 PM, Bernd Bausch wrote: thought we had removed that server from the haproxy config [0] that didn't happen because our ansible doesn't know how to gracefully restart haproxy in a container. At this point haproxy should be aware that the server is gone and it will prevent any new connections from going to it. [...]
Yep, sorry, we commented gitea01 out of the load balancer and I subsequently deleted its server instance at 2019-07-22 23:25 UTC, but I neglected to double-check that haproxy had actually stopped distributing connections to it. That one's on me, seems we still have some work to do making sure haproxy config changes are applied automatically there.
On 2019-07-24 20:20:24 +0900 (+0900), Bernd Bausch wrote:
Thanks much, gentlemen. I did hope this was a glitch on the server side, and indeed, I successfully cloned Devstack about 12 hours ago. [...]
Yes, sorry, we also subsequently discovered that haproxy was not configured with health checks for any of the backends, so it never removed the missing server from the relevant pools until I manually set the entries for it to maintenance mode. We have since merged fixes both to automatically apply haproxy configuration changes and to perform proper health checks of the Gitea servers, so this should hopefully not happen again. (We also updated how we restart Gitea itself so that each backend server is done sequentially and should provide ample alternatives for haproxy to roll through during updates now.) Thanks for letting us know! -- Jeremy Stanley
Please ignore; this seems to be a problem on my side. On 7/23/2019 9:09 AM, Bernd Bausch wrote:
There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem?
This happens around midnight between July 22 and 23 UTC.
$ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated.
Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded /git.openstack.org/ strings with the equivalent /github.com/ URL, but I am not sure if I can trust that all Github repos are up to date.
Bernd.
Hi all, Once it works well on my laptop, I'm still facing the same issue on the fresh ubuntu18.04-based VM. Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Tue, Jul 23, 2019 at 4:02 AM Bernd Bausch <berndbausch@gmail.com> wrote:
Please ignore; this seems to be a problem on my side. On 7/23/2019 9:09 AM, Bernd Bausch wrote:
There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem?
This happens around midnight between July 22 and 23 UTC.
$ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated.
Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded *git.openstack.org <http://git.openstack.org>* strings with the equivalent *github.com <http://github.com>* URL, but I am not sure if I can trust that all Github repos are up to date.
Bernd.
Looks like some CIs are affected too: http://logs.openstack.org/88/577388/23/check/grenade-vitrage/13de30a/logs/gr... Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Tue, Jul 23, 2019 at 3:36 PM Ivan Kolodyazhny <e0ne@e0ne.info> wrote:
Hi all,
Once it works well on my laptop, I'm still facing the same issue on the fresh ubuntu18.04-based VM.
Regards, Ivan Kolodyazhny, http://blog.e0ne.info/
On Tue, Jul 23, 2019 at 4:02 AM Bernd Bausch <berndbausch@gmail.com> wrote:
Please ignore; this seems to be a problem on my side. On 7/23/2019 9:09 AM, Bernd Bausch wrote:
There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem?
This happens around midnight between July 22 and 23 UTC.
$ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated.
Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded *git.openstack.org <http://git.openstack.org>* strings with the equivalent *github.com <http://github.com>* URL, but I am not sure if I can trust that all Github repos are up to date.
Bernd.
participants (4)
-
Bernd Bausch
-
Clark Boylan
-
Ivan Kolodyazhny
-
Jeremy Stanley