[Openstack] Swift generating thousands of rsyncs per second on a small cluster

Diogo Vieira dfv at eurotux.com
Tue Jul 1 09:50:15 UTC 2014


> The object servers should only be talking to each other during replication.  They should not talk to the proxy, and probably not the load balancer.  Can you provide the output of "swift-ring-builder /etc/swift/object.builder" and more details on the network configuration of this system.

First please allow me to correct my first email. I found out that most of the traffic is REPLICATE requests and not rsyncs between storage nodes.

So here is the output of the ring builder:

# swift-ring-builder /etc/swift/object.builder 
/etc/swift/object.builder, build version 5
262144 partitions, 3.000000 replicas, 1 regions, 5 zones, 5 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  region  zone      ip address  port  replication ip  replication port      name weight partitions balance meta
             0       1     5      10.10.6.16  6000      10.10.6.16              6000   device4  10.00     157287    0.00 
             1       1     3      10.10.6.14  6000      10.10.6.14              6000   device2  10.00     157286   -0.00 
             2       1     1      10.10.6.17  6000      10.10.6.17              6000   device5  10.00     157286   -0.00 
             3       1     2      10.10.4.21  6000      10.10.4.21              6000   device1  10.00     157287    0.00 
             4       1     4      10.10.6.15  6000      10.10.6.15              6000   device3  10.00     157286   -0.00 

Well, as far as network configuration goes, I'm running Openstack on two blades. The second one has only a compute node and Swift installed (1 Proxy Node and 1 Storage Node). Both machines are on the same vlan. The others Storage Nodes are on a different vlan as guest virtual machines of the first two. Between the two vlans there's a firewall where I logged the traffic.

It came to my attention that this should be the object-replicator doing its work, but the fact is that I get about a thousand or 2 requests per second between vlans - (the real machine running a Proxy and a Storage Node) and the virtual machines (all Storage Nodes only) - and it consumes a lot of resources (mainly CPU) seems to be a bit of a waste because the cluster is not being used for more than 2 weeks - nothing added or deleted from it.

Can you tell me if this is normal behaviour? If so, how will this scale when I add more objects? Will it keep getting more and more CPU usage?

> Generally the proxies are configured with multiple interfaces as the object servers and other backend storage nodes should not be accessible to clients directly.

Yes, I believe this is the case.

Thank you for your answer

Diogo Vieira <dfv at eurotux.com>
Programador 
Eurotux Informática, S.A. | www.eurotux.com
(t) +351 253 680 300
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140701/b94997dc/attachment.html>


More information about the Openstack mailing list