Open Stack

On 12/18/2012 08:54 AM, Mark McLoughlin wrote:
> On Wed, 2012-12-19 at 00:22 +0800, Thomas Goirand wrote:
>> On 12/18/2012 07:50 PM, Mark McLoughlin wrote:
> 
>>>   5) I'm guessing there are two issues with these 'git archive'
>>>      generated tarballs - (a) there's no versioninfo file so
>>>      setuptools doesn't know have a version number and (b) there's
>>>      no .git directory so setuptools doesn't have an accurate way of
>>>      building a manifest
>>>
>>>      I'm not completely clear what setuptools commands are failing
>>>      because of issue (b)
>>
>> All the above is exactly right!
>>
>> One of the reasons we are happy to use git archive is that this produces
>> a Debian orig.tar.xz (notice the xz, and not just gz) from *any* commit,
>> if we want to. Let me explain how it works.
>>
>> Let's say I want to make a snapshot release of Ceilometer for the commit
>> hash 23ff2f9bbfc14e435c4c04ddddfba473cf2a829b (this is an actual real
>> life example, in fact...). Then, to do that, I just do:
>>
>> git checkout master
>> git log # find out what's the name of the tag...
>> git tag 2013.1_g0.4+23ff2f9bbf 23ff2f9bbfc14e435c4c04ddddfba473cf2a829b
>> git checkout debian/experimental
>> git merge -X theirs 23ff2f9bbfc14e435c4c04ddddfba473cf2a829b
>>
>> Then I edit my debian/changelog so it matches the tag:
>>
>> <example debian/changelog>
>> ceilometer (2013.1~g0.4+23ff2f9bbf-1) experimental; urgency=low
>>
>>   * Initial release (Closes: #693406).
>>
>>  -- Thomas Goirand <zigo at debian.org>  Wed, 14 Nov 2012 14:41:52 +0000
>> <end of example debian/changelog>
>>
>> (the important bit is of course the version number above)
>>
>> Then I generate the orig.tar.xz:
>>
>> ./debian/rules get-vcs-source
>> cp ../ceilometer_2013.1~g0.4+23ff2f9bbf.orig.tar.xz ../build-area
>>
>> Then I just build:
>> git-buildpackage
>>
>> Another reason why git is convenient, is that we can cherry pick -x
>> stuff from upstream, do branching, etc. Well, do I really have to
>> convince people of this list that git is convenient? :) Probably not.
> 
> The main thing I don't understand is why generate the tarball with 'git
> archive' and not 'python setup.py sdist' ?
> 
> With a setuptools generated tarball, we'd at least have the versioninfo
> file generated and also the potential to generate a correct manifest
> from git.

I agree. We go through a LOT of trouble to make tarballs properly for
you. If you just download and use them as your upstream source, you
don't have to deal with this.

Either way - we do not and will not support git archive being a valid
source of tarballs. If you want a tarball from a random git sha, grab
the repo, run "python setup.py sdist" (or even, tox -evenv python
setup.py sdist) then use that tarball.

>>>   6) However, you seem to be saying that issue (b) isn't an issue but
>>>      rather inaccurate MANIFEST.in files are the problem. How exactly
>>>      are they causing a problem. If we delete those files from
>>>      upstream git, does that work for you?
>>
>> Well, it be better if the MANIFEST.in could be stored correctly in
>> upstream Git repositories, so we wouldn't have to deal with them.
>> Currently, we embed a patch in debian/patches to fix them.
> 
> How exactly is an incorrect MANIFEST.in is more trouble for you than no
> MANIFEST.in? Does it break setuptools in some way?

How do you patch MANIFEST.in?

>>>   7) You're also describing some issue where 'clean targets' (I don't
>>>       know what they are? Similar to 'make clean'?) are causing
>>>       commands like 'git fetch origin' to be run? What exactly is
>>>       going on here? Is this because the versioninfo file gets
>>>       deleted by the cleanup and our setup.py logic attempts to
>>>       recreate it? If we had a way of disabling network access during
>>>       the build (e.g. OS_SETUP_NONET=1) would that solve the problem?
>>
>> Let me give an example from Ceilometer. Without "cleaner = true",
>> git-buildpackage attemps to run "./debian/rules clean", which calls
>> setup.py clean -a. The shell output is then:
>>
>> zigo at GPLHost:dom0.node4407>_
>> ~/openstack-auto-builder/sources/ceilometer/ceilometer$ git-buildpackage
>> dh clean  --with python2
>>    dh_testdir
>>    debian/rules override_dh_auto_clean
>> make[1]: Entering directory
>> `/home/zigo/openstack-auto-builder/sources/ceilometer/ceilometer'
>> dh_auto_clean
>> Could not create directory '/root/.ssh'.
>> The authenticity of host 'git.debian.org (217.196.43.140)' can't be
>> established.
>> RSA key fingerprint is 8c:c0:b8:9f:0a:79:ee:1c:77:c4:b8:a1:70:55:b7:31.
>> Are you sure you want to continue connecting (yes/no)?
> 
> What setuptools command is being called here? 'setup.py clean' ?
> 
> Is this setuptools command being called from within the tarball or the
> git checkout?
> 
> If a versioninfo file exists, do you still see this issue?
> 
> Would 'git clean -f -x -d' do what you want without hitting setuptools?
> 
>> because in ceilometer/openstack/common/setup.py, there's a "git fetch
>> origin". Openstack common (or Oslo...) makes the wrong assumption that
>> "origin" is the repository on github, which isn't true in my case (for
>> us, origin is alioth.debian.org). Worse: it's trying to do this as root,
>> because of the use of the debian tool "fakeroot" to build.
> 
> I agree that you should be able to avoid hitting the network during
> builds.

Yes. We are working on that.
> 
>> I have discussed this problem already, and Julien Danjou made an attempt
>> at fixing it (but I'm not sure if his patch has been approved already or
>> not in gerrit).
>>
>> Of course, I can add "cleaner = true" so that no "setup.py clean -a" is
>> called before the build, but I think this is both suboptimal and
>> potentially dangerous.
> 
> Ok, so it is 'setup.py clean' you're doing. My other questions above
> still stand.

I will investigate what effects setup.py clean have. I don't believe I
ever run that command.

>>> Honestly, I've just spent a good amount of time trying to figure out
>>> your problems
>>
>> Thanks a lot!
>>
>>> If you want upstream to change its build process to reduce your
>>> packaging pain, then you really need to take the time to very clearly
>>> explain what you need in a way that doesn't require upstream folks to
>>> know anything about your packaging process.
>>
>> I'm trying ... :)
>> I hope that with this mail, it's going to be more clear. Let me know if
>> it's not.
>>
>> What would be great, for us in Debian, would be if every (PGP signed,
>> please... ;) ) tag upstream would also include the generation of a
>> correct MANIFEST.in and versioninfo files.
> 
> I think it's pretty clear - upstream is determined to come up with a
> solution that avoids MANIFEST.in and versioninfo being checked into git.
> You need to focus on helping us figure out how to achieve that while
> enabling your desired build process.

++

Every signed tag upstream will result in a proper tarball on
tarballs.openstack.org. So I don't think that there is anything gained
from generating those files as part of signed tag revisions. I'm fairly
against going back to static MANIFEST.in files, because they just plain
don't work. But I agree with Mark - I definitely want to solve your
problems, and I'll be investigating the elements of this that are
causing you pain.

Monty