Galera and haproxy fail; keystone error 'Lost connection to MySQL server during query'
If I run galera without haproxy and point everything to 1 server, it seems to work fine, but we want to load-balance it with haproxy. After setting up haproxy we had lots of errors in nova, neutron and keystone, but I got rid of those by setting haproxy timeout values to 10 minutes. The remaining errors are in keystone. In /var/log/keystone/keystone-wsgi-public.log I see "Loaded 2 Fernet keys" every 5 minutes and this is frequently accompanied by a mysql error 'Lost connection to MySQL server during query' I tried changing various config items in haproxy, galera and keystone, but nothing seems to help. How can I fix these errors? Errors: https://f.perl.bot/p/pbq26k https://f.perl.bot/p/1tnh78 https://f.perl.bot/p/fuxmwo Haproxy config: https://f.perl.bot/p/gu2lil Mysql timeout values: https://f.perl.bot/p/i6l7tn Keystone config (minus commented lines): https://f.perl.bot/p/o6fdht
Can anyone help with this galera issue? If I run galera without haproxy and point everything to 1 server, it seems to work fine, but we want to load-balance it with haproxy. After setting up haproxy we had lots of errors in nova, neutron and keystone, but I got rid of those by setting haproxy timeout values to 10 minutes. The remaining errors are in keystone. In /var/log/keystone/keystone-wsgi-public.log I see "Loaded 2 Fernet keys" every 5 minutes and this is frequently accompanied by a mysql error 'Lost connection to MySQL server during query' I tried changing various config items in haproxy, galera and keystone, but nothing seems to help. How can I fix these errors? Errors: https://f.perl.bot/p/pbq26k https://f.perl.bot/p/1tnh78 https://f.perl.bot/p/fuxmwo Haproxy config: https://f.perl.bot/p/gu2lil Mysql timeout values: https://f.perl.bot/p/i6l7tn Keystone config (minus commented lines): https://f.perl.bot/p/o6fdht
Database Connections are kept open for one hour. Timeouts on haproxy needs to reflect that. Set timeout to at least 60 minutes. Sent from my iPhone On Feb 10, 2020, at 9:04 AM, Albert Braden <Albert.Braden@synopsys.com> wrote: Can anyone help with this galera issue? If I run galera without haproxy and point everything to 1 server, it seems to work fine, but we want to load-balance it with haproxy. After setting up haproxy we had lots of errors in nova, neutron and keystone, but I got rid of those by setting haproxy timeout values to 10 minutes. The remaining errors are in keystone. In /var/log/keystone/keystone-wsgi-public.log I see “Loaded 2 Fernet keys” every 5 minutes and this is frequently accompanied by a mysql error 'Lost connection to MySQL server during query' I tried changing various config items in haproxy, galera and keystone, but nothing seems to help. How can I fix these errors? Errors: https://f.perl.bot/p/pbq26k<https://urldefense.com/v3/__https://f.perl.bot/p/pbq26k__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-YBgt797g$> https://f.perl.bot/p/1tnh78<https://urldefense.com/v3/__https://f.perl.bot/p/1tnh78__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-axKsd7Vw$> https://f.perl.bot/p/fuxmwo<https://urldefense.com/v3/__https://f.perl.bot/p/fuxmwo__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-ZpWo8aJw$> Haproxy config: https://f.perl.bot/p/gu2lil<https://urldefense.com/v3/__https://f.perl.bot/p/gu2lil__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-bBbKxEwg$> Mysql timeout values: https://f.perl.bot/p/i6l7tn<https://urldefense.com/v3/__https://f.perl.bot/p/i6l7tn__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-YO341YRQ$> Keystone config (minus commented lines): https://f.perl.bot/p/o6fdht<https://urldefense.com/v3/__https://f.perl.bot/p/o6fdht__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-bsHPT2GA$>
Erik, thank you! I did this, and made some other changes based on your other email, and my errors are gone. I’m still working on isolating the specific changes that fixed the problem, but it definitely was one of your ideas. Many thanks! From: Erik Olof Gunnar Andersson <eandersson@blizzard.com> Sent: Monday, February 10, 2020 9:54 AM To: Albert Braden <albertb@synopsys.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: Galera and haproxy fail; keystone error 'Lost connection to MySQL server during query' Database Connections are kept open for one hour. Timeouts on haproxy needs to reflect that. Set timeout to at least 60 minutes. Sent from my iPhone On Feb 10, 2020, at 9:04 AM, Albert Braden <Albert.Braden@synopsys.com<mailto:Albert.Braden@synopsys.com>> wrote: Can anyone help with this galera issue? If I run galera without haproxy and point everything to 1 server, it seems to work fine, but we want to load-balance it with haproxy. After setting up haproxy we had lots of errors in nova, neutron and keystone, but I got rid of those by setting haproxy timeout values to 10 minutes. The remaining errors are in keystone. In /var/log/keystone/keystone-wsgi-public.log I see “Loaded 2 Fernet keys” every 5 minutes and this is frequently accompanied by a mysql error 'Lost connection to MySQL server during query' I tried changing various config items in haproxy, galera and keystone, but nothing seems to help. How can I fix these errors? Errors: https://f.perl.bot/p/pbq26k<https://urldefense.com/v3/__https:/f.perl.bot/p/pbq26k__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-YBgt797g$> https://f.perl.bot/p/1tnh78<https://urldefense.com/v3/__https:/f.perl.bot/p/1tnh78__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-axKsd7Vw$> https://f.perl.bot/p/fuxmwo<https://urldefense.com/v3/__https:/f.perl.bot/p/fuxmwo__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-ZpWo8aJw$> Haproxy config: https://f.perl.bot/p/gu2lil<https://urldefense.com/v3/__https:/f.perl.bot/p/gu2lil__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-bBbKxEwg$> Mysql timeout values: https://f.perl.bot/p/i6l7tn<https://urldefense.com/v3/__https:/f.perl.bot/p/i6l7tn__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-YO341YRQ$> Keystone config (minus commented lines): https://f.perl.bot/p/o6fdht<https://urldefense.com/v3/__https:/f.perl.bot/p/o6fdht__;!!Ci6f514n9QsL8ck!x4V2c3Ar0WBo_5ECE5wOFh-JHypGv45o5SAQWiCOQZPtHMKHHLMvAI92Q-bsHPT2GA$>
participants (2)
-
Albert Braden
-
Erik Olof Gunnar Andersson