Open Stack

Excerpts from Jeremy Stanley's message of 2017-10-12 14:18:38 +0000:
> On 2017-10-12 16:28:33 +1100 (+1100), Ian Wienand wrote:
> [...]
> > - logs issues
> > 
> > Should be behind us.  The logs partition ran out of inodes, causing
> > log upload failures.  Pruning jobs should have rectified this.
> [...]
> 
> Just getting caught up now that I have caffeine, but this problem
> seems to have returned. The pruning passes are still underway (and
> likely take days to traverse the ~0.8 billion inodes on that 12TB
> volume so that much is expected), but it's apparent we're saving log
> files from jobs faster than we can find and delete old ones at this
> point. I expect our only recourse for now is to further reduce log
> retention to something like 2 weeks instead of 4 until we can get
> this under control (if we were already on Zuul v3 we could fold in
> draft documentation publishing and combine those volumes to buy us
> some breathing room, but such is life).

Yesterday jeblair asked about deleting some of the docs-old content,
which I took to be about space but if it's about inodes and the log
pruning isn't helping then we could do some partial cleanup if it would
help.

What I'm hoping to save for a little longer is the content from
openstack-manuals up to mitaka (newer stuff is relatively easy to
rebuild, but some of that old stuff isn't so much). So, if we can
preserve the diablo, essex, folsom, grizzly, havana, icehouse, juno,
kilo, liberty, and mitaka directories under /afs/openstack.org/docs-old
then anything else can be removed.

Doug