Open Stack

Hi,

It seems like the opensuse mirror has been on a bit of a growth spurt
[1].  Monitoring alerted me that the volume had not released for
several days, which lead me to look at the logs.

The rsync is failing with "File too large (27)" as it goes through
the tumbleweed sync.

As it turns out, AFS has a hard limit on the combined size of the
file names within a directory.  There are a couple of threads [2]
around from people who have found this out in pretty much the same way
as me ... when it starts failing :)

So you have 64k slots per directory, and file metadata+name takes up
slots per the formula:

 /* Find out how many entries are required to store a name. */
 int
 afs_dir_NameBlobs(char *name)
 {
     int i;
     i = strlen(name) + 1;
     return 1 + ((i + 15) >> 5);
 }

This means we have a problem with the large opensuse
tumbleweed/repo/oss/x86_64 directory, which has a lot of files with
quite long names.  Please, check my command/math, but if you run the
following command:

 $ rsync --list-only rsync://mirrors.rit.edu/opensuse/tumbleweed/repo/oss/x86_64/ \
   | awk '
     function slots(x) {
       i = length(x)+1;
       return 1 + rshift((i+15), 5)
     }
     { n += slots($5) }
     END {print n}
    '

I come out with 82285, which is significantly more than the 64k slots
available.

I don't know what to do here, and it's really going to be up to people
interested in opensuse.  The most immediate thing is unnecessary
packages could be pruned from tumbleweed/repo/oss/x86_64/ during the
rsync.  Where unnecessary is in the eye of the beholder ... :)
See my note below, but it may have to be quite under 64k.

If we have any sway with upstream, maybe they could shard this
directory; similar to debian [3] or fedora [4] (that said, centos does
the same thing [5], but due to less packages and shorter names etc
it's only about 40% allocated).

Note that (open)AFS doesn't hard-link across directories, so some sort
of "rsync into smaller directories then hardlink tree" doesn't really
work.

Ideas, suggestions, reviews welcome :)

-- ps

There's an additional complication in that the slots fragment over
time and file names must be contiguous.  This means in practice, you do
get even less.

There is potential to "defrag" (I bet post Windows 95 you never
thought you'd hear that again :) by rebuilding the directories with
the salvager [6].  However, there are additional complications
again...

To simply do this we have to run a complete salvage of the *entire*
partition.  Although I have added "-salvagedirs" to afs01's
salvageserver (via [7]) in an attempt to do this for just one volume,
it turns out this is not obeyed until after [8] which is not in the
Xenial AFS version we use.  I really do not want to salvage all the
other volumes, most of which are huge.  The other option is to create
a new AFS server, move the volume to that so it's the only thing on
the partition, and run it there, then move it back [9].

I actually suspect an rm -rf * might also do it, and probably be faster
because we'd only move the data down once from the remote mirror,
rather than to a new server and back

But defragging is rather secondary if the directory is oversubscribed
anyway.

-i

[1] http://grafana02.openstack.org/d/ACtl1JSmz/afs?orgId=1&from=now-7d&to=now&panelId=28&fullscreen
[2] https://lists.openafs.org/pipermail/openafs-info/2016-July/041859.html
[3] http://mirror.iad.rax.openstack.org/debian/pool/main/
[4] http://mirror.iad.rax.openstack.org/fedora/releases/28/Everything/x86_64/os/Packages/
[5] http://mirror.iad.rax.openstack.org/centos/7/os/x86_64/Packages/
[6] http://docs.openafs.org/Reference/8/salvager.html
[7] https://docs.openstack.org/infra/system-config/afs.html#updating-settings
[8] https://gerrit.openafs.org/#/c/12461/