Open Stack

Hello,

I made such modifications today in my infra and generally it looks 
better now. I don't see deadlocks. But I have one more problem with 
that: generally it works fine when main node is active but in situation 
when this node is down, haproxy connect to one of backup nodes. Still 
all is ok but problem is when main node is up again - all new 
connections are made to main node but active connections which was made 
to backup node still are active and neutron (or nova) are using 
connections to two servers and then there are problems with deadlock 
again.
Do You know how to prevent such situation?

--
Best regards
Slawek Kaplonski
slawek at kaplonski.pl

W dniu 2014-09-18 18:52, Simon McCartney napisał(a):
> We're using haproxy in front of a 3 node galera cluster, and we see
> the lost connection stuff pretty often, but that's normal behaviour,
> MySQL will tear down idle connections, SQLA handles reconnecting fine.
> 
> As for deadlocks, if you are distributing writes across your nodes,
> I'd expect that, we solved it by using haproxy in a primary/backup
> mode, all traffic flows to one node by default, and fails over to one
> of the other nodes in the event of a failure, this means that writes
> are concentrated in a single node & the deadlock & roll back doesn't
> bite you.
> 
> Also - I think you need to be running 3 galera nodes to be safe, as if
> a partition occurs, you can't cleanly reconnect a node as you can't
> achieve quorum. (I am not a DBA...)
> 
> We're not running Neutron, so I can't comment on that, sorry.
> 
> Simon.
> 
> On 18 September 2014 at 17:35:18, Sławek Kapłoński
> (slawek at kaplonski.pl) wrote:
> 
>> Hello,
>> 
>> Is anyone here using openstack with mysql galera and haproxy? Have
>> You got any
>> problems with that?
>> I was today installed such ha infra for database (two mysql servers
>> in galera
>> cluster and haproxy on controller and neutron node, this haproxy is
>> connecting
>> to one of galera servers with round robin algorithm). Generally all
>> is working
>> fine but I have few problems:
>> 1. I have a lot of messages like:
>> WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
>> server
>> has gone away: (2006, 'MySQL server has gone away')
>> 2. I have (most on neutron) many errors like:
>> OperationalError: (OperationalError) (2013, 'Lost connection to
>> MySQL server
>> during query') 'UPDATE ml2_port_bindings SET vif_type=%s, driver=%s,
>> 
>> segment=%s WHERE ml2_port_bindings.port_id =
>> 3. Also errors:
>> StaleDataError: UPDATE statement on table 'ports' expected to update
>> 1 row(s);
>> 0 were matched.
>> 4. and errors:
>> DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to
>> get lock;
>> try restarting transaction') 'UPDATE ipavailabilityranges SET
>> first_ip=%s WHERE
>> ipavailabilityranges.allocation_pool_id =
>> 
>> Sql queries in examples are "accidental" and same problem is with
>> other
>> queries also (like deleting ports).
>> Strange think is that those problems are not happend when I have one
>> mysql
>> server and all was connecting to that one server. Do You have maybe
>> same
>> problems? Do You know what can be a reason and solution for it?
>> 
>> ---
>> Best regards
>> Sławek Kapłoński
>> slawek at kaplonski.pl
>> -------------------------
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 
Pozdrawiam
Sławek Kapłoński
slawek at kaplonski.pl