Open Stack

Thanks Clay, very helpful!


On 14/12/16 13:06, Clay Gerrard wrote:
> I strongly prefer to configure rsyncd with a module per disk:
>
> https://github.com/openstack/swift/blob/c0640f87107d84d262c20bdc1250b805ae8f9482/etc/rsyncd.conf-sample#L25
>
> and then tune the per-disk connection limit to 2-4
>
> There's not really hard and fast rule, in some sense it's related to 
> replicator concurrency (the producer of the rsync's that *want* those 
> connections) - targeting roughly:
>
>    replicator-concurrency * #nodes ~== (max_conenction * nodes or 
> max_connection_per_module_per_disk * #disks)
>
> But max connections is about enforcing a sane limit, if you hit that 
> limit *a lot* it may be limiting throughput (and can lead to other 
> inefficiencies).  But having a few max connections fire now and again 
> is probably a good thing, since there's no guarantee that every 
> replicator coro in the cluster might not be thinking it should be 
> talking to a partition that just so happen to all have a replica on 
> this one disk at this one time.  You generally don't want too many 
> connections hammering a single spindle because of the await it 
> introduces for client requests that might need to hit that disk (but 
> there's auditor configuration management as well, and the new ionice 
> options for tuning might be an option as well).
>
> -Clay
>
> On Tue, Dec 13, 2016 at 3:11 PM, Mark Kirkwood 
> <mark.kirkwood at catalyst.net.nz <mailto:mark.kirkwood at catalyst.net.nz>> 
> wrote:
>
>     Any thoughts on this one guys?
>
>
>
>     On 06/12/16 09:56, Mark Kirkwood wrote:
>
>         Hi,
>
>         Is there a way to calculate rsync max connection (i.e based on
>         number of hosts, devices, object server workers etc)?
>
>         Some context: we have recently gone live with Swift 2.7.0
>         cluster (2 regions, 6 hosts overall, each with 16 cores/32
>         threads and 4 drives). We have rsync max connections to 25
>         (which I think is being set by puppet-swift.., is the default
>         value). We are seeing errors in the rsync log: 'max
>         connections (25) reached'. I'd like to be able to say to the
>         operations guys: set 'max connections to x, because <some
>         rules for calculating it>'.
>
>
>         Cheers
>
>
>         Mark
>
>
>         _______________________________________________
>         Mailing list:
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
>         Post to     : openstack at lists.openstack.org
>         <mailto:openstack at lists.openstack.org>
>         Unsubscribe :
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
>
>
>
>     _______________________________________________
>     Mailing list:
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
>     Post to     : openstack at lists.openstack.org
>     <mailto:openstack at lists.openstack.org>
>     Unsubscribe :
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
>
>