Open Stack

(moving to infra)

On 10/12/2017 04:28 PM, Ian Wienand wrote:
> mirrors provide, leading apt to great confusion.  Some debugging notes
> on reprepro at [1], but I have to conclude the .db files are corrupt
> and I have no idea how to recreate these other than to start again.

I ran the reprepro under strace, and the last thing that comes out is

 3170  pread(6, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 90521600) = 4096

then it just stops with cpu at 100%.  lsof tells us

 reprepro 3170 root    6u   REG   0,25  90628096 2537568 /afs/.openstack.org/mirror/ubuntu/db/checksums.db

so, that db seems as likely as any to be causing the problems

pabelanger pointed to some recovery instructions at [1] previously.

I tried this in order, firstly recreating references.db (didn't help)
and so I have started the checksums.db recreation.  This is now
running; I just moved the old one out of the way

  root at mirror-update:/afs/.openstack.org/mirror/ubuntu/db# ls -lh
  total 1.1G
  -rw-r--r-- 1 10004 root 1.6M Oct 12 06:38 checksums.db
  -rw-r--r-- 1 10004 root  87M Oct 12 02:59 checksums.db.old

This started at about 06:30, meaning ~5 minutes/mb so I think around 6
hours till this is finished, hopefully (it's dragging everything
across afs).

Please take any of this over; it's running on mirror-update:

  screen(9683)─┬─bash(9684)───su(9917)───bash(9918)
          │    ├─bash(10466)───k5start(3755)───bash(3758)─┬─find(3996)
               │                                          └─reprepro(3997)

note i'm holding the cron lock with 

 root     10957  9918  0 06:46 pts/0    00:00:00 flock -n /var/run/reprepro/ubuntu.lock bash -c while true; do sleep 1000; done

(ps, I think we need to make those dirs on reboot:
https://review.openstack.org/511380)

-i

[1] https://github.com/esc/reprepro/blob/master/docs/recovery