Open Stack

On Thu, Jul 28 2016, gordon chung wrote:

> this is probably something to discuss on ops list as well eventually but 
> what do you think about shrinking the max size of timeseries chunks from 
> 14400 to something smaller? i'm curious to understand what the length of 
> the typical timeseries is. my main reason for bringing this up is that 
> even our default 'high' policy doesn't reach 14400 limit so it at most 
> will only split into two, partially filled objects. as we look to make a 
> more efficient storage format for v3(?) seems like this may be an 
> opportunity to change size as well (if necessary)

1 minute granularity over a year: 525600 points, 27 splits.
Even in that case, which is pretty precise, that's not a lot of split
I'd say.

> 14400 points roughly equals 128KB object which is cool but maybe we 
> should target something smaller? 7200points aka 64KB? 3600 points aka 
> 32KB? just for reference our biggest default series is 10080 points 
> (1min granularity over week).

It's 128 Kb if you don't compress, but if you do, it's usually way less.

> that said 128KB (at most) might not be that bad from read/write pov and 
> maybe it's ok to keep it at 14400? i know from the test i did earlier, 
> the time requirement to read/write increases linearly (7200 point object 
> takes roughly half time of 14400 point object)[1]. i think the main item 
> is we don't want it too small that we're updating multiple objects at a 
> time.

Best way is probably to do some bench… but I think it really depends on
the use cases here. The interest of having many small splits is that you
can parallelize the read.

Considering the compression ratio we have, I think we should split in
smaller files. I'd pick 3600 and give it a try.

-- 
Julien Danjou
// Free Software hacker
// https://julien.danjou.info
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 800 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160729/b69eaa3d/attachment.pgp>