[Openstack] [Swift] PUT requests sensitive to latency?
Hua ZZ Zhang
zhuadl at cn.ibm.com
Thu Jun 26 06:39:16 UTC 2014
Shrinand,
I have done some tracing work here and you maybe interested to try.
This new patch can help you to record timing of each hop of request if you
enable it in log options.
This might give you more insights and help you to diagnose network latency
problem.
https://review.openstack.org/#/c/93677/
Any feedback is welcomed!
-Edward Zhang
Shrinand
Javadekar
<shrinand at maginat To
ics.com> Hua ZZ Zhang/China/IBM at IBMCN
cc
2014-06-26 下午 "openstack at lists.openstack.org"
01:57 <openstack at lists.openstack.org>
Subject
Re: [Openstack] [Swift] PUT
requests sensitive to latency?
Hey Edward,
Thanks for the explanation. I had a discussion regarding this on the
openstack-swift irc channel yesterday. Sam Merrit mentioned pretty much the
points that you mention here.
To confirm this, I actually bumped up the client_chunk_size
and object_chunk_size in the proxy-server.conf as well as disk_chunk_size
and network_chunk_size in the object-server.conf and ran the same
experiment. As expected these did not change the results. The client which
is farther still results in slower PUT times compared to the closer one.
Also, I know that a single node swift cluster can have multiple replicas. I
meant to say that my setup has a single node Swift instance with a single
replica.
Just to summarize:
* In my setup, the object server is writing the data to SSDs. So the IO
cost of writing to the storage is less.
* The proxy-server and object-server are co-located. So communication
between them is fast.
* Therefore the dominating factor affecting the time taken by PUT requests
is the network latency between client and swift proxy-server.
* Optimizing this would require tweaking either the TCP options on the
client and server machines or changing the socket options appropriately in
the application code.
-Shri
On Tue, Jun 24, 2014 at 8:17 PM, Hua ZZ Zhang <zhuadl at cn.ibm.com> wrote:
I don't think so. Since you are doing PUT object request, it will have
network I/O operations b/w client/proxy server/object server
and disk I/O operations on object server to save the data. The latency of
sending data from client to proxy server should be affected
by their network latency. So the object server might wait a very little
more time to get those data transmitted from proxy server.
Plus object server need to disk I/O to save object data. Ultimately the
network latency can be ignored due to disk operation become
the new bottleneck if your network latency improve a lot.
Single node Swift cluster don't mean you could have only one replica. It
can be more by configuring your install script. You should check it in
your
Swift Rings. Even you changed the network and disk chuck size, it is only
used by Swift object server as buffer size to read and write data from
network or disk. It does not change the default TCP/IP packet size which
means you can't send the object data in one packet. The network latency
will have impact on your object PUT request.
-Edward
Inactive hide details for Shrinand Javadekar ---2014-06-25 上午
01:14:07---Shrinand Javadekar <shrinand at maginatics.com>Shrinand Javadekar
---2014-06-25 上午 01:14:07---Shrinand Javadekar <shrinand at maginatics.com
>
Shrinand Javadekar <
shrinand at maginatics.com>
To
2014-06-25 上午 01:14
Hua ZZ
Zhang/China/IBM at IBMCN
cc
"
openstack at lists.openstac
k.org" <
openstack at lists.openstac
k.org>
Subject
Re: [Openstack] [Swift]
PUT requests sensitive
to latency?
Communication between proxy and object servers shouldn't be affected by
the latency between the proxy server and the client, right? Also, I'm
using a single node Swift cluster. So there should be only 1 copy of the
object (along with any other I/Os required for the container and accound
DBs).
Everything that happens on the Swift side should be the same (if there is
no back-n-forth between the Swift server and client) irrespective of how
much time it takes for communication between the Swift cluster and the
client.
I had made one mistake when experimenting with the network_chunk_size
and disk_chunk_size config options. These are supposed to go to the
object-server.conf and not the proxy-server.conf. I made that change and
restarted all the swift servers. However, I don't see any improvement. My
current object-server.conf looks like this:
http://pastie.org/private/exjiho1cbl80mbruythama
What do you think?
-Shri
On Tue, Jun 24, 2014 at 12:29 AM, Hua ZZ Zhang <zhuadl at cn.ibm.com> wrote:
My guess is the object data need to be transmitted to Swift cluster
before the status code returned.
It can't be returned immediately before 2/3 I/O completed.
Otherwise it is not consistent to tell client
it succeed.
-Edward Zhang
Inactive hide details for Shrinand Javadekar ---2014-06-24 下午
03:12:14---Shrinand Javadekar <shrinand at maginatics.com>Shrinand
Javadekar ---2014-06-24 下午 03:12:14---Shrinand Javadekar <
shrinand at maginatics.com>
Shrinand Javadekar <
shrinand at maginatics.com>
To
2014-06-24 下午 03:05
"
openstack at lists.openstac
k.org" <
openstack at lists.openstac
k.org>
cc
Subject
[Openstack] [Swift] PUT
requests sensitive to
latency?
Hi,
I have a single node swift cluster. I measured the time taken to
complete a PUT request that originated from three different client
machines. Each client was writing a single 256K byte object.
Note that the time measured was only the time taken on the Swift
cluster itself. I started the timer after the request was received
by
the swift proxy-server process and stopped it when that method
returned an HTTP status to the client. This is not the time on the
client side and therefore *ideally* should not be affected by the
latency between the client and the Swift cluster.
However, it appears that the above is not true. The time required
to
complete the request is related to the latency between the client
and
swift cluster.
Here are the results:
* Client 1:
Ping time 28ms
PUT request time: ~180 ms
* Client 2:
Ping time 4 ms
PUT request time: ~35 ms
* Client 3:
Ping time 0.04 ms
PUT request time: ~10 ms
Details about the experiment:
* This is a single node Swift installation (not devstack) and uses
SSDs to store metadata as well as data. This is just a test setup.
In
production, we won't have SSDs for storing data.
* The above numbers are average of 50 PUT requests.
* The Swift cluster was not being used for anything else during the
experiment.
* The client used was the jclouds library written in java. I had
disable a config option that used the Expect 100-Continue header;
i.e.
the requests were not using the Expect 100-Continue header.
* I tried increasing the size of the following options in the
proxy-server.conf and restarting Swift.
disk_chunk_size = 262144
network_chunk_size = 262144
...
[app:proxy-server]
object_chunk_size = 262144
client_chunk_size = 262144
However, this didn't show any improvement in the time required for
PUT requests.
Am I missing anything? Does Swift require an extra round trip from
the
client for completing PUT requests? Any ways of avoiding that?
Thanks in advance.
-Shri
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
(See attached file: pic09941.gif)
(See attached file: pic08688.gif)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140626/618b687d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140626/618b687d/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pic25830.gif
Type: image/gif
Size: 1255 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140626/618b687d/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140626/618b687d/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pic08688.gif
Type: image/gif
Size: 1255 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140626/618b687d/attachment-0003.gif>
More information about the Openstack
mailing list
