Open Stack

On Tue, Dec 19, 2017, at 10:18 AM, Gaetan wrote:
> Hello
> 
> I recently submitted the following pull request in order to allow using git
> submodules with PBR modules: https://review.openstack.org/#/c/524421/
> 
> To give a little bit of context, in my team, all our python modules are
> following almos the same pattern:
> - python 3 only
> - setup.py/setuptools/distutil stuff completely handled by PBR
> - dependencies handled by Pipfile (see my first draft to support it with
> PBR here: https://review.openstack.org/#/c/524436/)
> - "active dependencies", basically shared modules we may want to update
> quickly are injected in the project using Git Submodules, and installed
> with `pipenv install ./deps/mydependency` (equivalent in pip: `pip install
> -e deps/mydepedency`).
> - all on gitlab (so using the github/gitlab "pull request"/"mergerequest"
> mecanism).
> - Python applications (that used PBR) are deployed in docker
> 
> This is a very, very cool architecture, allowing us to do change very
> quickly in depedencies and reference it directly even if it has been merged
> into the dependency yet. A whole cycle or review might takes a few days,
> change may be rejected, unit tests might be requested before merge,... So
> during all this time, in the "normal" way of releasing a change on a
> dependency (mergerequest + git tag (thanks PBR) + upload to our internal
> pypi), the change might not be available in the parent module that needs it.
> So injecting dependencies with git submodules is very handy, even if all
> dependencies, at the end, once stabilized, will end up in a clean pypi repo.
> 
> Now, why Pipfile? Because it is also much more convinient to use that a
> requirements.txt (and if you want to accurately handle the versions, you
> need a requirements.txt.in + pip-tools to generate requirements.txt). Same
> for the development requirements. So, pipfile handles all of that for us,
> and Pipfile will eventually become the defacto standard for dependency
> declaration, replacing requirements*.txt.
> Pipfile recently moved to Pypa, becoming a little bit more official:
> https://github.com/pypa/pipfile.
> 
> So, back to my first patch (https://review.openstack.org/#/c/524421/). It
> fixes an issue when trying to use everything all together. In our Gitlab
> CI, PBR is happy and almost does the right thing, even it does not handle
> Pipfile directly (see https://review.openstack.org/#/c/524436/ for the
> limitations).
> 
> But once in the docker build, PBR complains about not getting access to
> upstream GIT. And it is true: the git tree (with dependencies) has been
> checkedout by gitlab and the docker build (running in a gitlab runner) does
> not give access to the repository, because it is not required.

Is the issue here that submodules somehow cause PBR to need access to upstream git repositories or is the installation process not including the .git directories for repositories and only including their checked out state? PBR should work with local git repository checkouts without needing to talk to any remotes/upstreams.

> 
> Why PBR does that? Because:
> - git submodules dependencies does not provided pkg metadata, because they
> are not distribution package (they are installed from source)
> - the method that finds the version from the git tree fails with this
> upstream access error probably because it expects to be able to fetch
> something from the remote, and the environment does not give this access)
> 
> So my simple and dirty solution is to use the PBR_VERSION environment
> variable to skip this check, that is not required on dependencies coming
> from git submodules since the "version matching" is actually handled by git
> sumodules. I first only moved PBR_VERSION handling a few lines to avoid
> affecting other packages that uses PBR:
> https://review.openstack.org/#/c/524421/1/pbr/packaging.py
> 
> There are a bit of discussion on my patch, thanks for the reviewers by the
> way. I see basically 3 solutions for making PBR happy with git submodules
> and docker, all requiring developer to set an envvar in their Dockerfile:
> 
>    - developers set the environment variable PBR_VERSION to freeze the
>    version of dependencies injected by git submodules. This only requires my
>    first patchset
>    - same but with another variable (PBR_FALLBACK_VERSION ?).
>    - same but with an environment variable with the name of the package in
>    it. if the name can take non alphanumeric character, I can strip them out
>    with a regex for example (see documentation proposal here:
>    https://review.openstack.org/#/c/524421/3/doc/source/user/packagers.rst)
> 
> The other solution might be to allow pbr to NOT have git upstream access
> in _get_version_from_git.

At least without submodules this shouldn't be required. Without submodules you only need access to the local git repository and not any remotes. I think this behavior is desireable and if we can have that behavior with submodules as well that is probably a good improvement to make.

> 
> Sorry for the long context description. Hope this will help you being
> interested by my patches.
> 
> PS: about Pipfile support in PBR, I will open another thread on this ML if
> you agree, when my unit tests will pass, but i will not be able to do it
> until net year.

Adding pipfile support seems like an excellent thing to add considering people are trying to use pipfile and pbr together now.

> Also, I maintain a fork a pbr "pbrlgs" (
> https://github.com/Stibbons/pbr/tree/pbrlgs) that makes PBR happy with git
> submodules. I had to in order to use it in our internal build system.

Are these improvements something that should be upstreamed as well?

> 
> Thanks

Thank you!

Clark