[Openstack] Blueprint proposal: Drop setuptools_git for including data/config files

Sascha Peilicke saschpe at suse.de
Wed Dec 5 08:56:01 UTC 2012

On 12/04/2012 11:01 PM, Nah, Zhongyue wrote:
> Is it possible to generate a MANIFEST.in file using setuptools-git? Would that solve simplicity and efficiency?

Please join the discussion on openstack-dev at . It was my mistake sending
this technical proposal to this ML initially.

> Sent from my iPhone
> On Dec 4, 2012, at 9:07 PM, "Thierry Carrez" <thierry at openstack.org> wrote:
>> Sascha Peilicke wrote:
>>> Currently, the majority of OpenStack components make use of the
>>> Python module "setuptools_git" in order to install additional
>>> configuration files. This is basically the same functionality that
>>> the MANIFEST.in file (setuptools/distribute) provides, but
>>> automatic.
>> Note: This is a development topic, it should (have) be(en) posted to
>> openstack-dev to reach the appropriate audience. Please follow-up there.
>>> However, we recently discovered that this approach has issues from
>>> a packaging perspective. We weren't getting all the data/config
>>> files that the python package usually gets even though we were
>>> running the same commands:
>>> $ python setup.py build
>>> followed by:
>>> $ python setup.py install --skip-build
>>> We are building RPM packages from release tarballs (such as [1]),
>>> which of course don't include the .git directory. Therefore the
>>> setuptools_git approach can't do its magic, thus our package builds
>>> get wrong results. Having OpenStack components rely on
>>> setuptools_git at build time means we have to distribute the whole
>>> git repository along with the source code tarball. Of course this
>>> makes no sense, since it would increase the size of release
>>> tarballs dramatically and won't get shipped in distributions
>>> anyway.Therefore, we (and potentially other distribution vendors)
>>> would have to track these files manually in our RPM spec files. 
>>> Some reviews have already been opened on the topic (admittedly
>>> before we discovered the real issue). Given the different outcome
>>> of each review it seems that not everybody is aware that
>>> setuptools_git is used or of what it does.
>>> https://review.openstack.org/#/c/17122/ (ceilometer) - this one
>>> got accepted before we knew what was going on
>>> https://review.openstack.org/#/c/17347/ (cinder) - abandoned until
>>> the situation is clarified
>>> https://review.openstack.org/#/c/17355/ (nova) - rejected
>>> So the better solution would be to stop using setuptools_git and
>>> just include all the data/config files that are meant to be
>>> distributed in the MANIFEST.in file. This is what every Python
>>> developer should know about and has the benefit of increased
>>> transparency about what gets installed and what not. We created a
>>> blueprint to track this [2].
>>> Thoughts?
>> A bit of history here:
>> We used to rely on MANIFEST.in to list all files, but people routinely
>> forgot to add new files to it. Apparently "every Python developer"
>> doesn't know (or care) about this. The end result was that we
>> discovered very late (sometimes after the release itself) that we
>> built incomplete tarballs. As a quick search[1] shows, I have
>> personally filed 27 bugs so far on the subject, so it's not a corner case.
>> [1] http://bit.ly/TDim7U
>> Relying on setuptools_git instead allows to avoid that issue
>> altogether. The projects that adopted it became a non-issue. The
>> projects that didn't adopt it yet are still a problem. I was about to
>> push setuptools_git support to projects that don't use it yet.
>> In summary, I would hate it if we went back to the previous situation.
>> I'm not personally attached to setuptools_git, but any proposed
>> replacement solution should keep its simplicity.
>> -- 
>> Thierry Carrez (ttx)
>> Release Manager, OpenStack
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

With kind regards,
Sascha Peilicke
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend├Ârffer HRB 16746 (AG N├╝rnberg)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20121205/669f5297/attachment.sig>

More information about the Openstack mailing list