[openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
florian.engelmann at everyware.ch
Tue Oct 9 15:32:02 UTC 2018
Am 10/9/18 um 1:23 PM schrieb Jay Pipes:
> On 10/09/2018 06:34 AM, Florian Engelmann wrote:
>> Am 10/9/18 um 11:41 AM schrieb Jay Pipes:
>>> On 10/09/2018 04:34 AM, Christian Berendt wrote:
>>>>> On 8. Oct 2018, at 19:48, Jay Pipes <jaypipes at gmail.com> wrote:
>>>>> Why not send all read and all write traffic to a single haproxy
>>>>> endpoint and just have haproxy spread all traffic across each
>>>>> Galera node?
>>>>> Galera, after all, is multi-master synchronous replication... so it
>>>>> shouldn't matter which node in the Galera cluster you send traffic to.
>>>> Probably because of MySQL deadlocks in Galera:
>>>> Galera cluster has known limitations, one of them is that it uses
>>>> cluster-wide optimistic locking. This may cause some transactions to
>>>> rollback. With an increasing number of writeable masters, the
>>>> transaction rollback rate may increase, especially if there is write
>>>> contention on the same dataset. It is of course possible to retry
>>>> the transaction and perhaps it will COMMIT in the retries, but this
>>>> will add to the transaction latency. However, some designs are
>>>> deadlock prone, e.g sequence tables.
>>> Have you seen the above in production?
>> Yes of course. Just depends on the application and how high the
>> workload gets.
>> Please read about deadloks and nova in the following report by Intel:
> I have read the above. It's a synthetic workload analysis, which is why
> I asked if you'd seen this in production.
> For the record, we addressed much of the contention/races mentioned in
> the above around scheduler resource consumption in the Ocata and Pike
> releases of Nova.
> I'm aware that the report above identifies the quota handling code in
> Nova as the primary culprit of the deadlock issues but again, it's a
> synthetic workload that is designed to find breaking points. It doesn't
> represent a realistic production workload.
> You can read about the deadlock issue in depth on my blog here:
> That explains where the source of the problem comes from (it's the use
> of SELECT FOR UPDATE, which has been removed from Nova's quota-handling
> code in the Rocky release).
Thank you very much for your link. Great article!!! Took my a while to
read it and understand everything but helped a lot!!!
>> If just Nova is affected we could also create an additional HAProxy
>> listener using all Galera nodes with round-robin for all other services?
> I fail to see the point of using Galera with a single writer. At that
> point, why bother with Galera at all? Just use a single database node
> with a single slave for backup purposes.
From my point of view Galera is easy to manage and great for HA. Having
to handle a manual failover in production with mysql master/slave never
Indeed writing to a single node and not using the other nodes (even for
read, like it is done in kolla-ansible) is not the best solution. Galera
is slower than a standalone MySQL.
Using ProxySQL would enable us to use caching and read/write split to
speed up database queries while HA and management are still good.
>> Anyway - proxySQL would be a great extension.
> I don't disagree that proxySQL is a good extension. However, it adds yet
> another services to the mesh that needs to be deployed, configured and
True. I guess we will start with an external MySQL installation to
collect some experience.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 5210 bytes
Desc: not available
More information about the OpenStack-dev