Open Stack

Hello

I recently submitted the following pull request in order to allow using git
submodules with PBR modules: https://review.openstack.org/#/c/524421/

To give a little bit of context, in my team, all our python modules are
following almos the same pattern:
- python 3 only
- setup.py/setuptools/distutil stuff completely handled by PBR
- dependencies handled by Pipfile (see my first draft to support it with
PBR here: https://review.openstack.org/#/c/524436/)
- "active dependencies", basically shared modules we may want to update
quickly are injected in the project using Git Submodules, and installed
with `pipenv install ./deps/mydependency` (equivalent in pip: `pip install
-e deps/mydepedency`).
- all on gitlab (so using the github/gitlab "pull request"/"mergerequest"
mecanism).
- Python applications (that used PBR) are deployed in docker

This is a very, very cool architecture, allowing us to do change very
quickly in depedencies and reference it directly even if it has been merged
into the dependency yet. A whole cycle or review might takes a few days,
change may be rejected, unit tests might be requested before merge,... So
during all this time, in the "normal" way of releasing a change on a
dependency (mergerequest + git tag (thanks PBR) + upload to our internal
pypi), the change might not be available in the parent module that needs it.
So injecting dependencies with git submodules is very handy, even if all
dependencies, at the end, once stabilized, will end up in a clean pypi repo.

Now, why Pipfile? Because it is also much more convinient to use that a
requirements.txt (and if you want to accurately handle the versions, you
need a requirements.txt.in + pip-tools to generate requirements.txt). Same
for the development requirements. So, pipfile handles all of that for us,
and Pipfile will eventually become the defacto standard for dependency
declaration, replacing requirements*.txt.
Pipfile recently moved to Pypa, becoming a little bit more official:
https://github.com/pypa/pipfile.

So, back to my first patch (https://review.openstack.org/#/c/524421/). It
fixes an issue when trying to use everything all together. In our Gitlab
CI, PBR is happy and almost does the right thing, even it does not handle
Pipfile directly (see https://review.openstack.org/#/c/524436/ for the
limitations).

But once in the docker build, PBR complains about not getting access to
upstream GIT. And it is true: the git tree (with dependencies) has been
checkedout by gitlab and the docker build (running in a gitlab runner) does
not give access to the repository, because it is not required.

Why PBR does that? Because:
- git submodules dependencies does not provided pkg metadata, because they
are not distribution package (they are installed from source)
- the method that finds the version from the git tree fails with this
upstream access error probably because it expects to be able to fetch
something from the remote, and the environment does not give this access)

So my simple and dirty solution is to use the PBR_VERSION environment
variable to skip this check, that is not required on dependencies coming
from git submodules since the "version matching" is actually handled by git
sumodules. I first only moved PBR_VERSION handling a few lines to avoid
affecting other packages that uses PBR:
https://review.openstack.org/#/c/524421/1/pbr/packaging.py

There are a bit of discussion on my patch, thanks for the reviewers by the
way. I see basically 3 solutions for making PBR happy with git submodules
and docker, all requiring developer to set an envvar in their Dockerfile:

   - developers set the environment variable PBR_VERSION to freeze the
   version of dependencies injected by git submodules. This only requires my
   first patchset
   - same but with another variable (PBR_FALLBACK_VERSION ?).
   - same but with an environment variable with the name of the package in
   it. if the name can take non alphanumeric character, I can strip them out
   with a regex for example (see documentation proposal here:
   https://review.openstack.org/#/c/524421/3/doc/source/user/packagers.rst)

The other solution might be to allow pbr to NOT have git upstream access
in _get_version_from_git.

Sorry for the long context description. Hope this will help you being
interested by my patches.

PS: about Pipfile support in PBR, I will open another thread on this ML if
you agree, when my unit tests will pass, but i will not be able to do it
until net year.
Also, I maintain a fork a pbr "pbrlgs" (
https://github.com/Stibbons/pbr/tree/pbrlgs) that makes PBR happy with git
submodules. I had to in order to use it in our internal build system.

Thanks

-----
Gaetan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171219/be733283/attachment.html>