<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">The guys from Zmanda presented some
evaluation of swift at the summit, might be useful here<br>
<br>
<a class="moz-txt-link-freetext" href="http://www.zmanda.com/blogs/?p=947">http://www.zmanda.com/blogs/?p=947</a> they've written a blog but it
doesn't have all the findings which they presented at the summit.<br>
<br>
Maybe Chander would be willing to share? I've CC'd him in.<br>
<div class="moz-signature"><font face="Helvetica" size="2"><br>
--<br>
<b>Sina Sadeghi</b><br>
Lead Cloud Engineer<br>
<img src="cid:part1.00030603.00050706@aptira.com"><br>
<b>Aptira Pty Ltd</b><br>
1800 APTIRA<br>
<a href="http://www.aptira.com">aptira.com</a><br>
<a href="https://twitter.com/#/aptira">Follow @aptira</a><br>
</font><br>
</div>
On 25/10/12 08:03, Alejandro Comisario wrote:<br>
</div>
<blockquote
cite="mid:CAMrG31wFtyPcAO6f7Ex2+9t9Lxma_oeTdE1=tG=fLc-Z26CjKA@mail.gmail.com"
type="cite"><font color="#000000"><font><font face="courier
new,monospace">Wow nice, i think we have a lot to look at
guys.</font></font></font>
<div><font face="courier new, monospace">I'll get back to you as
soon as we have more metrics to share regarding this matter.</font></div>
<div><font face="courier new, monospace">Basically, we are going
to try to add more proxies, since indeed, the requests are to
small (20K not 20MB)</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<font face="courier new, monospace">Thanks guys !</font></div>
<div><font face="courier new, monospace">-----------</font></div>
<div><font face="courier new, monospace">Alejandrito</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div>
<div class="gmail_quote">On Wed, Oct 24, 2012 at 5:49 PM, John
Dickinson <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:me@not.mn" target="_blank">me@not.mn</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Smaller requests, of course, will have a higher percentage
overhead for each request, so you will need more proxies
for many small requests than the same number of larger
requests (all other factors being equal).<br>
<br>
If most of the requests are reads, then you probably won't
have to worry about keystone keeping up.<br>
<br>
You may want to look at tuning the object server config
variable "keep_cache_size". This variable is the maximum
size of an object to keep in the buffer cache for publicly
requested objects. So if you tuned it to be 20K
(20971520)--by default it is 5424880--you should be able
to serve most of your requests without needing to do a
disk seek, assuming you have enough RAM on the object
servers. Note that background processes on the object
servers end up using the cache for storing the filesystem
inodes, so lots of RAM will be a very good thing in your
use case. Of course, the usefulness of this caching is
dependent on how frequently a given object is accessed.
You may consider an external caching system (anything from
varnish or squid to a CDN provider) if the direct public
access becomes too expensive.<br>
<br>
One other factor to consider is that since swift stores 3
replicas of the data, there are 3 servers that can serve a
request for a given object, regardless of how many storage
nodes you have. This means that if all 3500 req/sec are to
the same object, only 3 object servers are handling that.
However, if the 3500 req/sec are spread over many objects,
the full cluster will be utilized. Some of us have talked
about how to improve swift's performance for concurrent
access to a single object, but those improvements have not
been coded yet.<br>
<span class="HOEnZb"><font color="#888888"><br>
--John<br>
</font></span>
<div class="HOEnZb">
<div class="h5"><br>
<br>
<br>
On Oct 24, 2012, at 1:20 PM, Alejandro Comisario <<a
moz-do-not-send="true"
href="mailto:alejandro.comisario@mercadolibre.com">alejandro.comisario@mercadolibre.com</a>>
wrote:<br>
<br>
> Thanks Josh, and Thanks John.<br>
> I know it was an exciting Summit! Congrats to
everyone !<br>
><br>
> John, let me give you extra data and something
that i've already said, that might me wrong.<br>
><br>
> First, the request size that will compose the
90.000RPM - 200.000 RPM will be from 90% 20K objects,
and 10% 150/200K objects.<br>
> Second, all the "GET" requests, are going to be
"public", configured through ACL, so, if the GET
requests are public (so, no X-Auth-Token is passed)
why should i be worried about the keystone middleware
?<br>
><br>
> Just to clarify, because i really want to
understand what my real metrics are so i can know
where to tune in case i need to.<br>
> Thanks !<br>
><br>
> ---<br>
> Alejandrito<br>
><br>
><br>
> On Wed, Oct 24, 2012 at 3:28 PM, John Dickinson
<<a moz-do-not-send="true" href="mailto:me@not.mn">me@not.mn</a>>
wrote:<br>
> Sorry for the delay. You've got an interesting
problem, and we were all quite busy last week with the
summit.<br>
><br>
> First, the standard caveat: Your performance is
going to be highly dependent on your particular
workload and your particular hardware deployment. 3500
req/sec in two different deployments may be very
different based on the size of the requests, the
spread of the data requested, and the type of
requests. Your experience may vary, etc, etc.<br>
><br>
> However, for an attempt to answer your
question...<br>
><br>
> 6 proxies for 3500 req/sec doesn't sound
unreasonable. It's in line with other numbers I've
seen from people and what I've seen from other large
scale deployments. You are basically looking at about
600 req/sec/proxy.<br>
><br>
> My first concern is not the swift workload, but
how keystone handles the authentication of the tokens.
A quick glance at the keystone source seems to
indicate that keystone's auth_token middleware is
using a standard memcached module that may not play
well with concurrent connections in eventlet.
Specifically, sockets cannot be reused concurrently by
different greenthreads. You may find that the token
validation in the auth_token middleware fails under
any sort of load. This would need to be verified by
your testing or an examination of the memcache module
being used. An alternative would be to look at the way
swift implements it's memcache connections in an
eventlet-friendly way (see
swift/common/memcache.py:_get_conns() in the swift
codebase).<br>
><br>
> --John<br>
><br>
><br>
><br>
> On Oct 11, 2012, at 4:28 PM, Alejandro Comisario
<<a moz-do-not-send="true"
href="mailto:alejandro.comisario@mercadolibre.com">alejandro.comisario@mercadolibre.com</a>>
wrote:<br>
><br>
> > Hi Stackers !<br>
> > This is the thing, today we have a 24
datanodes (3 copies, 90TB usables) each datanode has 2
intel hexacores CPU with HT and 96GB of RAM, and 6
Proxies with the same hardware configuration, using
swift 1.4.8 with keystone.<br>
> > Regarding the networking, each proxy /
datanodes has a dual 1Gb nic, bonded in LACP mode 4,
each of the proxies are behind an F5 BigIP Load
Balancer ( so, no worries over there ).<br>
> ><br>
> > Today, we are receiving 5000 RPM ( Requests
per Minute ) with 660 RPM per Proxies, i know its low,
but now ... with a new product migration, soon (
really soon ) we are expecting to receive about a
total of 90.000 RPM average ( 1500 req / s ) with
weekly peaks of 200.000 RPM ( 3500 req / s ) to the
swift api, witch will be 90% public gets ( no keystone
auth ) and 10% authorized PUTS (keystone in the
middle, worth to know that we have a 10 keystone vms
pool, connected to a 5 nodes galera mysql cluster, so
no worries there either )<br>
> ><br>
> > So, 3500 req/s divided by 6 proxy nodes
doesnt sounds too much, but well, its a number that we
cant ignore.<br>
> > What do you think about this numbers? does
this 6 proxies sounds good, or we should double or
triple the proxies ? Does anyone has this size of
requests and can share their configs ?<br>
> ><br>
> > Thanks a lot, hoping to ear from you guys !<br>
> ><br>
> > -----<br>
> > alejandrito<br>
> >
_______________________________________________<br>
> > Mailing list: <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
> > Post to : <a moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
> > Unsubscribe : <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
> > More help : <a moz-do-not-send="true"
href="https://help.launchpad.net/ListHelp"
target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Mailing list: <a class="moz-txt-link-freetext" href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a>
Post to : <a class="moz-txt-link-abbreviated" href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a>
Unsubscribe : <a class="moz-txt-link-freetext" href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a>
More help : <a class="moz-txt-link-freetext" href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a>
</pre>
</blockquote>
<br>
</body>
</html>