Open Stack

On Mon, 2012-12-17 at 23:47 +0800, Thomas Goirand wrote:
> On 12/04/2012 09:06 PM, Thierry Carrez wrote:
> > A bit of history here:
> > 
> > We used to rely on MANIFEST.in to list all files, but people routinely
> > forgot to add new files to it. Apparently "every Python developer"
> > doesn't know (or care) about this. The end result was that we
> > discovered very late (sometimes after the release itself) that we
> > built incomplete tarballs. As a quick search[1] shows, I have
> > personally filed 27 bugs so far on the subject, so it's not a corner case.
> > 
> > [1] http://bit.ly/TDim7U
> > 
> > Relying on setuptools_git instead allows to avoid that issue
> > altogether. The projects that adopted it became a non-issue. The
> > projects that didn't adopt it yet are still a problem. I was about to
> > push setuptools_git support to projects that don't use it yet.
> > 
> > In summary, I would hate it if we went back to the previous situation.
> > I'm not personally attached to setuptools_git, but any proposed
> > replacement solution should keep its simplicity.
> 
> With my Debian Developer in charge of Openstack, I have to tell that
> this was a real problem for us, in Debian, as well.
> 
> On Debian, we don't use released tarballs like they do on Ubuntu. We use
> git-buildpackage, and do our packaging exclusively with Git (no bzr at
> all here...). All of our packaging work is stored on alioth.debian.org,
> on the pkg-openstack git repository. We don't even store the master
> branch, it is picked-up doing:
> 
> ./debian/rules get-vcs-source
> 
> which on top of that, creates a .orig.tar.xz out of the upstream source
> code. (yes! Because xz is so much nicer than gzip...)
> 
> The problem isn't to use setuptools_git for building, you're fine to do
> that, the problem is having the Git repositories not holding the correct
> MANIFEST.in file, and fooling the people doing packaging using Git. This
> means that absolutely all of our packages have to embed a patch in
> debian/patches to "fix" the "wrong" MANIFEST.in.
> 
> We've spent quite some time on that. Or rather, should I say: it's a
> real time waster.
> 
> While I do agree that the MANIFEST.in should be generated automatically,
> I don't think it should be stored in a "wrong" way on github.
> 
> The magic solution could be to have some automatic tools to take care of
> it. Like, for example, jenkins, or maybe a git hook? I'm not sure, how,
> but I'm quite convince it should be workable.
> 
> Also, we've been having troubles with clean targets accessing the
> network at build time, because they see a ".git" folder. That's wrong
> too, when I do "git-buildpackage", I do have a .git folder, though I
> don't want to have any network access, even less a "git fetch origin" or
> something like this. This can be mitigated by doing:
> 
> echo "cleaner = true" >>debian/gbp.conf
> 
> I've done that, but that's quite annoying to have to disable the clean
> before the build (note: it's only not cleaning my git repo, but it's
> really calling clean later on at build time in ../build-area).

I have to say - I'm quite sympathetic about this issue and would like to
see it solved, but I haven't yet seen a clear explanation of the build
process you're using and where setuptools_git is causing an issue. There
are a lot of gaps in my understanding of the problem that I'm having to
use guesswork to fill in.

Let's see if I can explain my current understanding and where the gaps
are.

  1) You don't build packages from the tarballs produced by upstream. I 
     don't understand why you don't use tarballs, but I'm willing to 
     assume it's not something you can (or want to) change.

  2) Instead you use git-buildpackage which I know nothing about.

  3) You work from git repositories forked from upstream e.g.

       http://anonscm.debian.org/gitweb/?p=openstack/python-novaclient.git;a=shortlog;h=refs/heads/debian/experimental

     which AFAICT just add a debian/ directory to the source tree

  4) 'get-vcs-source' somehow generates a tarball from this tree. I'm 
     guessing it does 'git archive' rather than 'python setup.py sdist' 
     but I'm not sure. Some more digging turns up:

       http://anonscm.debian.org/gitweb/?p=openstack/openstack-pkg-tools.git;a=blob_plain;f=pkgos.make;hb=HEAD

     and it seems that yes, you're using 'git archive'

  5) I'm guessing there are two issues with these 'git archive' 
     generated tarballs - (a) there's no versioninfo file so setuptools 
     doesn't know have a version number and (b) there's no .git 
     directory so setuptools doesn't have an accurate way of building a 
     manifest

     I'm not completely clear what setuptools commands are failing 
     because of issue (b)

  6) However, you seem to be saying that issue (b) isn't an issue but 
     rather inaccurate MANIFEST.in files are the problem. How exactly 
     are they causing a problem. If we delete those files from upstream 
     git, does that work for you?

  7) You're also describing some issue where 'clean targets' (I don't 
      know what they are? Similar to 'make clean'?) are causing 
      commands like 'git fetch origin' to be run? What exactly is going 
      on here? Is this because the versioninfo file gets deleted by the 
      cleanup and our setup.py logic attempts to recreate it? If we had 
      a way of disabling network access during the build (e.g. 
      OS_SETUP_NONET=1) would that solve the problem?

Honestly, I've just spent a good amount of time trying to figure out
your problems and I think I'm even more confused.

If you want upstream to change its build process to reduce your
packaging pain, then you really need to take the time to very clearly
explain what you need in a way that doesn't require upstream folks to
know anything about your packaging process.

For example, you could say ....

  I effectively need to be able to do:

    $> git clone git://.../python-novaclient.git
    $> cd python-novaclient
    $> git archive .... > ../python-novaclient-2.10.0.tar.gz
    $> tar -xvzf python-nova-client-2.10.0
    $> cd python-nova-client-2.10.0
    $> python setup.py install

  But I right now that fails because of foo, bar and blaa.

  Instead I'm doing:

    ....

  But that sucks because ....

Cheers,
Mark.