Open Stack

On 2017-10-12 16:28:33 +1100 (+1100), Ian Wienand wrote:
[...]
> - logs issues
> 
> Should be behind us.  The logs partition ran out of inodes, causing
> log upload failures.  Pruning jobs should have rectified this.
[...]

Just getting caught up now that I have caffeine, but this problem
seems to have returned. The pruning passes are still underway (and
likely take days to traverse the ~0.8 billion inodes on that 12TB
volume so that much is expected), but it's apparent we're saving log
files from jobs faster than we can find and delete old ones at this
point. I expect our only recourse for now is to further reduce log
retention to something like 2 weeks instead of 4 until we can get
this under control (if we were already on Zuul v3 we could fold in
draft documentation publishing and combine those volumes to buy us
some breathing room, but such is life).
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: Digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171012/afad9881/attachment.sig>