Open Stack

On 02/03/17 09:52 AM, Julien Danjou wrote:
>> using hashring idea, the buckets will be distributed among all the
>> > active metricd agents. the metricd agents will loop through all the
>> > assigned buckets based on processing interval. the actual processing of
>> > each bucket will be similar to what we have now: grab metrics, queue it
>> > for processing workers. the only difference is instead of just grabbing
>> > first x metrics and stopping, we keep grabbing until bucket is 'clear'.
>> > this will help us avoid the current issue where some metrics are never
>> > scheduled because the return order puts it at the end.
> It does not change that much the current issue IIUC. The only difference
> is that now we have 1 bucket and N metricd trying to empty it, whereas
> now we would have M buckets with N metricd trying spread so there's M/N
> metricd per bucket trying to empty each bucket. :)
>
> At some scale (larger than currently) it will improve things but it does
> not seem to be a drastic change.
>
> (I am also not saying that I have a better solution :)
>

one of the issues we can't effectively partition the single bucket is 
partly because we have multiple agents on a single bucket. in theory we 
can use markers to partition the single bucket but because of multiple 
workers, the marker has a very high chance of disappearing.

in this case, only one agents is ever working on a bucket so it should 
minimise the chance of a marker disappearing and therefore let us go 
deeper into bucket.

-- 
gord