Open Stack

hey,

so as we transition to the bucket/shard/sack framework for incoming 
writes, we've set up locks on the sacks so we only have one process 
handling any given sack. this allows us to remove the per-metric locking 
we had previously.

the issue i've notice now is that if we only have per-sack locking, 
metric based actions can affect sack level processing. for example:

scenario 1:
1. delete metric, locks sack to delete single metric,
2. metricd processor attempts to process entire sack but can't, so skips.

OR

scenario 2:
1. API request passes in 'refresh' param so they want all unaggregated 
metrics to be processed on demand and returned.
2. API locks 1 or more sacks to process 1 or more metrics
3. metricd processor attempts to process entire sack but can't, so 
skips. potentially multiple sacks unprocessed in currently cycle.

scenario 3
same as scenario 2 but metricd processor locks first, and either blocks
API process OR  doesn't allow API to guarantee 'all measures processed'.

i imagine these scenarios are not critical unless a very large 
processing interval is defined or if for some unfortunate reason, the 
metric-based actions are perfectly timed to lock out background processing.

alternatively, this could be solved by keeping per-metric locks in 
addition to per-sack locks. this would effectively double the number of 
active locks we have so instead of each metricd worker having a single 
per-sack lock, it will also have a per-metric lock for whatever metric 
it may be publishing at the time.


cheers,
-- 
gord