Open Stack

On Mon, Jun 18, 2018, at 6:58 AM, Ian Wienand wrote:
> Hi,
> 
> It seems like the opensuse mirror has been on a bit of a growth spurt
> [1].  Monitoring alerted me that the volume had not released for
> several days, which lead me to look at the logs.
> 
> The rsync is failing with "File too large (27)" as it goes through
> the tumbleweed sync.
> 
> As it turns out, AFS has a hard limit on the combined size of the
> file names within a directory.  There are a couple of threads [2]
> around from people who have found this out in pretty much the same way
> as me ... when it starts failing :)
> 
> So you have 64k slots per directory, and file metadata+name takes up
> slots per the formula:
> 
>  /* Find out how many entries are required to store a name. */
>  int
>  afs_dir_NameBlobs(char *name)
>  {
>      int i;
>      i = strlen(name) + 1;
>      return 1 + ((i + 15) >> 5);
>  }
> 
> This means we have a problem with the large opensuse
> tumbleweed/repo/oss/x86_64 directory, which has a lot of files with
> quite long names.  Please, check my command/math, but if you run the
> following command:
> 
>  $ rsync --list-only 
> rsync://mirrors.rit.edu/opensuse/tumbleweed/repo/oss/x86_64/ \
>    | awk '
>      function slots(x) {
>        i = length(x)+1;
>        return 1 + rshift((i+15), 5)
>      }
>      { n += slots($5) }
>      END {print n}
>     '
> 
> I come out with 82285, which is significantly more than the 64k slots
> available.
> 
> I don't know what to do here, and it's really going to be up to people
> interested in opensuse.  The most immediate thing is unnecessary
> packages could be pruned from tumbleweed/repo/oss/x86_64/ during the
> rsync.  Where unnecessary is in the eye of the beholder ... :)
> See my note below, but it may have to be quite under 64k.

It looks like the repo carries at least two versions of each package. We could start by only syncing the latest, which would about halve the number of files.

Colleen

> 
> If we have any sway with upstream, maybe they could shard this
> directory; similar to debian [3] or fedora [4] (that said, centos does
> the same thing [5], but due to less packages and shorter names etc
> it's only about 40% allocated).
> 
> Note that (open)AFS doesn't hard-link across directories, so some sort
> of "rsync into smaller directories then hardlink tree" doesn't really
> work.
> 
> Ideas, suggestions, reviews welcome :)
> 
> -- ps
> 
> There's an additional complication in that the slots fragment over
> time and file names must be contiguous.  This means in practice, you do
> get even less.
> 
> There is potential to "defrag" (I bet post Windows 95 you never
> thought you'd hear that again :) by rebuilding the directories with
> the salvager [6].  However, there are additional complications
> again...
> 
> To simply do this we have to run a complete salvage of the *entire*
> partition.  Although I have added "-salvagedirs" to afs01's
> salvageserver (via [7]) in an attempt to do this for just one volume,
> it turns out this is not obeyed until after [8] which is not in the
> Xenial AFS version we use.  I really do not want to salvage all the
> other volumes, most of which are huge.  The other option is to create
> a new AFS server, move the volume to that so it's the only thing on
> the partition, and run it there, then move it back [9].
> 
> I actually suspect an rm -rf * might also do it, and probably be faster
> because we'd only move the data down once from the remote mirror,
> rather than to a new server and back
> 
> But defragging is rather secondary if the directory is oversubscribed
> anyway.
> 
> -i
> 
> [1] 
> http://grafana02.openstack.org/d/ACtl1JSmz/afs?orgId=1&from=now-7d&to=now&panelId=28&fullscreen
> [2] 
> https://lists.openafs.org/pipermail/openafs-info/2016-July/041859.html
> [3] http://mirror.iad.rax.openstack.org/debian/pool/main/
> [4] 
> http://mirror.iad.rax.openstack.org/fedora/releases/28/Everything/x86_64/os/Packages/
> [5] http://mirror.iad.rax.openstack.org/centos/7/os/x86_64/Packages/
> [6] http://docs.openafs.org/Reference/8/salvager.html
> [7] 
> https://docs.openstack.org/infra/system-config/afs.html#updating-settings
> [8] https://gerrit.openafs.org/#/c/12461/
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra