[kolla] restart mariadb container on host 1

Gregory Orange gregory.orange at pawsey.org.au
Thu Jun 9 11:45:20 UTC 2022


Right, thank you - I thought and hoped that was the intent.

I'm just doing:
docker stop mariadb
docker start mariadb

I've just done the same with host 3, and while stopped grastate.dat 
changed to `safe_to_bootstrap: 1` on host 2, and seqno: appeared in host 
3's grastate.dat. Starting it again worked fine.

docker inspect mariadb differs on the hosts, 1 has:
             "Env": [ ... "BOOTSTRAP_ARGS=--wsrep-new-cluster",

while the other two don't.

/var/log/kolla/mariadb/mariadb.log
https://termbin.com/px15 with safe_to_bootstrap: 0, container stops again

https://termbin.com/meli after removing 
/var/lib/docker/volumes/mariadb/_data/<all but mysql/>, separate cluster 
results


I've just tested a dev cluster with the same steps, and it doesn't have 
the same problem - no Env entry, no problems restarting.
I'm now wondering if it relates to the database migration and restore we 
did a while back, coming from another cluster with rsync replication and 
switching over to mariabackup.
Next, I deleted the container in the dev cluster, and ran 
mariadb_recovery play - worked fine. I was hesitant to do this in the 
test cluster I'm working on because of other settings we have 
handcrafted for now, but after a few trial runs, I'm more confident.
...

Yep, that fixed it - docker stop/start mariadb now works. BOOTSTRAP_ARGS 
is gone from the Env in the container too.

Thank you for giving me the confidence to prod it further.

Greg.


On 9/6/22 16:53, Mark Goddard wrote:
> Hi Greg,
> 
> It should only have that option set when bootstrapping the cluster. How 
> are you stopping and starting the container? Could you provide logs?
> 
> Mark
> 
> On Thu, 9 Jun 2022 at 09:17, Gregory Orange 
> <gregory.orange at pawsey.org.au <mailto:gregory.orange at pawsey.org.au>> wrote:
> 
>     Hi all,
> 
>     I wanted to temporarily change a mariadb config in
>     /etc/kolla/mariadb/galera.cnf so I intended to stop mariadb on host 1,
>     make the change, start it again, then repeat in turn for hosts 2 and 3.
> 
>     Upon starting it on host 1, it doesn't rejoin, but creates its own
>     separate cluster. Dutiful really, since logs show:
> 
>     WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster
> 
>     How do I avoid that? General advice on handling this kind of activity
>     would also be welcome.
> 
>     Thank you,
>     Greg.
> 

-- 
Gregory Orange

Cloud System Administrator
Scientific Platforms Team building representative
Pawsey Supercomputing Centre, CSIRO



More information about the openstack-discuss mailing list