Open Stack

Hi all,

Sometimes when I create a bunch of VMs (40) at once, one of them gets 
stuck in BUILD status.

 From logs I found that this happens due to SQL error in nova-conductor: 
DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost 
connection to MySQL server during query').

There is an attempt to schedule the VM:

....

nova-scheduler"sending reply msg_id: 593a990393d5439580870d8412ab49f3 
reply queue: reply_a0992fe3f9924cb1b45bfa8b27057580 time elapsed: 
1.49198414385s"

And right after that the SQL error happens:

nova-conductor "Exception during message handling" "DBConnectionError: 
(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server 
during query') [SQL: u'SELECT instance_id_mappings.created_at AS 
instance_id_mappings_created ......"

And after that the VM stays in BUILD status forever.


My setup is Galera cluster behind Haproxy load balancer.

I checked various DB server timeouts - they are totally fine. Ones in 
MySQL correspond ones in nova.conf and haproxy.cfg.

In the MySQL logs there are messages like:

[Warning] Aborted connection 17798924 to db: 'nova' user: 'nova' host: 
'xxx.xxx.xxx.xxx' (Got an error reading communication packets)

In the Haproxy logs there are messages like:

<134>2023-08-05T08:44:34.325338+00:00 dbs01 haproxy[1275]: 
xxx.xxx.xxx.xxx:44880 [05/Aug/2023:08:42:44.441] mysql_cluster_nova_conn 
mysql_cluster_nova_conn/dbs02 1/0/109884 11438 CD 4125/1884/1884/1884/0 0/0

"CD" means "The client unexpectedly aborted during data transfer"

So the Haproxy believes the connection is closed at client side. I 
checked the network interface counters - they are also fine (no errors, 
drops etc).


I found some discussions in the Internet. People say, when the DB is 
behind load balancer - such 'Lost connection' events are pretty normal 
and not harmful to the Openstack components.

I would agree with them. I think, Nova should be able to recover from 
accidental DB connection lost. Or to set the ERROR status to the VM if 
that's impossible.

My Nova version is 17.0.13 (Queens).

May be someone encountered such issue and knows how to fix?

Or that's a bug in Nova and I need to upgrade it to a more recent version?

Thanks in advance


----

Best regards,

Pavlo Bychikhin