<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8">
</head>
<body>
<div style="font-family:sans-serif"><div style="white-space:normal">
<p dir="auto">On 14 Nov 2016, at 13:57, gordon chung wrote:</p>
<p dir="auto"></p></div>
<div style="white-space:pre-wrap"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><div dir="auto">hi,
</div><div dir="auto">
</div><div dir="auto">one issue i've noticed with our 'backlog' scheduling is that we register
</div><div dir="auto">all our new measures in a single folder/filestore object. this folder or
</div><div dir="auto">object in most production cases can grow quite large (tens/hundreds of
</div><div dir="auto">thousands). so we don't load it all into memory, the drivers will only
</div><div dir="auto">grab the first x items and process them. unfortunately, we don't control
</div><div dir="auto">the ordering of the returned items so it is dependent on the ordering
</div><div dir="auto">the backend returns. for Ceph, it returns in what i guess is some
</div><div dir="auto">alphanumeric order. the file driver i believe returns based on how the
</div><div dir="auto">filesystem indexes files. i have no idea how swift ordering behaves. the
</div></blockquote></div>
<div style="white-space:normal">
<p dir="auto">Listings in Swift are lexicographically sorted by the object name.</p>
<p dir="auto"></p></div>
<div style="white-space:pre-wrap"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><div dir="auto">result of this is that we may starve some new measures from being
</div><div dir="auto">processed because they keep getting pushed back by more recent measures
</div><div dir="auto">if less agents are deployed.
</div><div dir="auto">
</div><div dir="auto">with that said, this isn't a huge issue because measures can be
</div><div dir="auto">processed on demand using refresh parameter but it's not ideal.
</div><div dir="auto">
</div><div dir="auto">i was thinking, to better handle processing while minimising the effects
</div><div dir="auto">of a driver's natural indexing, we can hash our new measures into
</div><div dir="auto">buckets based on metric_id. Gnocchi would hash all incoming metrics into
</div><div dir="auto">100? buckets and metricd agents would divide up these buckets and loop
</div><div dir="auto">through them. this would ensure we have smaller buckets to deal with and
</div><div dir="auto">therefore less chance, metrics get pushed back and starved. that said it
</div><div dir="auto">will add additional requirements of 100? folders/filestore objects
</div><div dir="auto">rather than 1. it will also mean we may be making significantly more
</div><div dir="auto">smaller fetches vs single (possibly) giant fetch.
</div><div dir="auto">
</div><div dir="auto">to extend this, we could also hash into project_id groups and thus allow
</div><div dir="auto">some projects to have more workers and thus more performant queries?
</div><div dir="auto">this might be too product tailored. :)
</div></blockquote></div>
<div style="white-space:normal">
<p dir="auto">I'm stepping in to an area here (gnocchi) that I know very little about, so please forgive me where I mess up.</p>
<p dir="auto">First, as a practical note, stuff in Swift will be <em>much</em> better when you spread it across the entire namespace. It's a lot better to data stored in many containers instead of putting all data into just one container. Spreading the data out takes advantages of Swift's scaling characteristics and makes users and ops happier.</p>
<p dir="auto">Second, at the risk of overgeneralizing[1], you may want to consider using the consistent hashing code from Ironic and Nova (and is being discussed as a new oslo library). Consistent hashing gives you the nice property of being able to change the number of buckets you're hashing data into without having to rehash most of the existing data. Think of it this way, if you hash into two buckets and use even/odd (i.e. last bit) to determine which bucket data goes into, then when you need a third bucket you have to switch to MOD 3 and two-thirds of your existing data will move into a different bucket. That's bad, and it gets even worse as you add more and more buckets. With consistent hashing, you can get the property that if you add 1% more buckets, you'll only move about 1% of the existing data to a different bucket.</p>
<p dir="auto">So going even further into my ignorance of the gnocchi problem space, I could imagine that there may be some benefit of being able to change the number of hash buckets over time based on how many items, how many workers are processing them, rate of new metrics, etc. If there's benefit to changing the number of hash buckets over time, then looking at consistent hashing is probably worth the time. If there is no benefit to changing the number of hash buckets over time, then a simple <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">hash(data) >> (hash_len - log2(num_hash_buckets) + 1)</code> or <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">hash(data) MOD num_hash_buckets</code> is probably sufficient.</p>
<p dir="auto">--John</p>
<p dir="auto">[1] I've got a nice hammer. Your problem sure looks like a nail to me.</p>
<p dir="auto"></p></div>
<div style="white-space:pre-wrap"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><div dir="auto">
</div><div dir="auto">thoughts?
</div><div dir="auto">
</div><div dir="auto">cheers,
</div><div dir="auto">--
</div><div dir="auto">gord
</div><div dir="auto">__________________________________________________________________________
</div><div dir="auto">OpenStack Development Mailing List (not for usage questions)
</div><div dir="auto">Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
</div><div dir="auto"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="color:#777">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</div></blockquote></div>
<div style="white-space:normal">
</div>
</div>
</body>
</html>