Open Stack

On 04/08/2018 04:10 AM, Gaetan wrote:
> Hello OpenStack dev community,
> 
> I am currently working on the support of Pipfile for PBR ([1]), and I 
> also follow actively the work on pipenv, which is now in officially 
> supported by PyPA.

Awesome - welcome! This is a fun topic ...

> There have been recently an intense discussion on the difficulties about 
> Python libraries development, and how to spread good practices [2] on 
> the pipenv community and enhance its documentation.
> 
> As a user of PBR, and big fan of it, I try to bridge the link between 
> pbr and pipenv (with [1]) but I am interested in getting the feedback of 
> Python developers of OpenStack that may have much more experience using 
> PBR and more generally packaging python libraries than me.

Great - I'll comment more on this a little later.

> The main point is that packaging an application is quite easy or at 
> least understandable by newcomers, using `requirements.txt` or 
> `Pipfile`+ `Pipfile.lock` with pipenv. At least it is easily "teachable".
> Packaging a library is harder, and require to explain why by default 
> `requirements.txt`(or `Pipfile`) does not work. Some "advanced" 
> documentation exists but it still hard to understand why Python ended up 
> with something complex for libraries ([3]).
> One needs to ensure `install_requires`declares the dependencies to that 
> pip can find them during transitive dependencies installation (that is, 
> installing the dependencies of a given dependency). PBR helps on this 
> point but some does not want its other features.

In general, as you might imagine, pbr has a difference of opinion with 
the pypa community about requirements.txt and install_requires. I'm 
going to respond from my POV about how things should work - and how I 
believe they MUST work for a project such as OpenStack to be able to 
operate.

There are actually three different relevant use cases here, with some 
patterns available to draw from. I'm going to spell them out to just 
make sure we're on the same page.

* Library
* Application
* Suite of Coordinated Applications

A Library needs to declare the requirements it has along with any 
relevant ranges. Such as "this library requires 'foo' at at least 
version 2 but less than version 4". Since it's a library it needs to be 
able to handle being included in more than one application that may have 
different sets of requirements, so as a library it should attempt to 
have as wide a set of acceptable requirements as possible - but it 
should declare if there are versions of requirements it does not work 
with. In Pipfile world, this means "commit Pipfile but not 
Pipfile.lock". In pbr+requirements.txt it means "commit the 
requirements.txt with ranges and not == declared."

An Application isn't included in other things, it's the end point. So 
declaring a specific set of versions of things that the application is 
known to work in addition to the logical requirement range is considered 
a best practice. In Pipfile world, this is "commit both Pipefile and 
Pipfile.lock". There isn't a direct analog for pbr+requirements.txt, 
although you could simulate this by executing pip with a -c 
constraints.txt file.

A Suite of Coordinated Applications (like OpenStack) needs to 
communicate the specific versions the applications have been tested to 
work with, but they need to be the same so that all of the applications 
can be deployed side-by-side on the same machine without conflict. In 
OpenStack we do this by keeping a centrally managed constraints file [1] 
that our CI system adds to the pip install line when installing any of 
the OpenStack projects. A person who wants to install OpenStack from pip 
can also choose to do so using the upper-constraints.txt file and they 
can know they'll be getting the versions of dependencies we tested with. 
There is also no direct support for making this easier in pbr. For 
Pipfile, I believe we'd want to see is adding support for --constraints 
to pipenv install - so that we can update our Pipfile.lock file for each 
application in the context of the global constraints file. This can be 
simulated today without any support from pipenv directly like this:

   pipenv install
   $(pipenv --venv)/bin/pip install -U -c 
https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt 
-r requirements.txt
   pipenv lock

> There is also works on PEP around pyproject.toml ([4]), which looks 
> quite similar to PBR's setup.cfg. What do you think about it?

It's a bit different. There is also a philosophical disagreement about 
the use of TOML that's not worth going in to here - but from a pbr 
perspecitve I'd like to minimize use of pyproject.toml to the bare 
minimm needed to bootstrap things into pbr's control. In the first phase 
I expect to replace our current setup.py boilerplate:

setuptools.setup(
     setup_requires=['pbr'],
     pbr=True)

with:

setuptool.setup(pbr=True)

and add pyproject.toml files with:

[build-system]
requires = ["setuptools", "wheel", "pbr"]

This will allow us to reasonably have projects declare minimum ranges on 
the pbr depend - and can allow us to give pbr dependencies (which is 
impossible today due to how setup_requires works) If we made setuptools 
and wheel depends of pbr,we could redue the pyproject.toml file to:

[build-system]
requires = ["pbr"]

but we need to test to make sure that works first.

Once pep517 is implemented, we can implement a hook in pbr and add a 
line to pyproject.toml in all of the projects, something like:

build-backend = "pbr.core:build"

Come to think of it, we could go ahead and implement pep517 support in 
pbr today and go ahead and start having the pbr pyproject.toml file to be:

[build-system]
requires = ["pbr"]
build-backend = "pbr.core:build"

We'll have to keep the setup.py files until such a time as pip has full 
517 supported added.

> My opinion is this difference in behaviourbetween lib and app has 
> technical reasons, but as a community we would gain a lot of unifying 
> both workflows. I am using PBR + a few hacks [5], and I am pretty 
> satisfied with the overall result.

There are two topics your pbr patch opens up that need to be covered:

* pbr behavior
* dependencies

** pbr behavior **

I appreciate what you're saying about unifying the lib and app workflow, 
but I think the general pattern across the language communities 
(javascript and rust both have similar patterns to Pipefile) is that the 
two different options are important. We may just need to have a better 
document - rust has an excellent description:

https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html

In any case, I think what pbr should do with pipfiles is:

* If pbr discovers a Pipfile and no Pipfile.lock, it should treat the 
content in the packages section of Pipfile as it currently does with 
requirements.txt (and how you have done in the current patch)

* If pbr discoves a Pipfile.lock, it should treat the content in 
Pipfile.lock as it currently does requirements.txt. This way if someone 
commits a Pipfile.lock because they have chosen the application 
workflow, pbr will behave as they would expect.

Then, we either need to:

* Add support to pipfile install for specifying a pip-style constraints file

* Add support to pipfile install for specifying a constraints file that 
is in the format of a Pipfile.lock - but which does the same thing.

* Write a pbr utility subcommand for generating a Pipfile.lock from a 
Pipfile taking a provided constraints file into account.

We may also want to write a utility for creating a Pipefile and/or lock 
from a pbr-oriented requirements.txt/test-requirements.txt. (it should 
use pipfile on the backend of course) that can do the appropriate 
initial dance.

** dependencies **

The pep518 support in pip10 is really important here. Because ...

We should not vendor code into pbr.

While vendoring code has been found to be acceptable by other portions 
of the Python community, it is not acceptable here.

Once pip10 is released next week with pyproject.toml support, as 
mentioned earlier, we'll be able to start using (a reasonable set) of 
dependencies as is appropriate. In order to ensure backwards compat, I 
would recommend we do the following:

* Add toml and pipfile as depends to pbr
* Protect their imports with a try/except (for people using old pip 
which won't install any depends pbr has)
* Declare that pbr support for Pipfile will only work with people using 
pip>=10 and for projects that add a pyproject.toml to their project 
containing

   [build-system]
   requires = ["pbr"]

* If pbr tries to import toml/pipfile and fails, it should fall back to 
reading requirements.txt (this allows us to keep backwards compat until 
it's reasonable to expect everyone to be on pip10)

To support that last point, we should write a utility function, let's 
call it 'pbr lock', with the following behavior:

* If a Pipfile and a Pipfile.lock are present, it runs:

   pipfile lock -r

* If there is no Pipfile.lock, simply read the Pipfile and write the 
specifiers into requirements.txt in non-pinned format.

This will allow pbr users to maintain their projects in such a way as to 
be backwards compatible while they start to use Pipfile/Pipefile.lock

We MAY want to consider adding an option flag to setup.cfg, like:

   [pbr]
   type = application

or

   [pbr]
   type = library

for declaring to pbr which of Pipfile / Pipfile.lock should pbr pay 
attention to, regardless of which files might be present. I'm not sure 
whether that would be better or worse than inferring behavior from the 
presence of files. Of course, we could have the default behavior if the 
config setting isn't there to be to infer behavior from presence of 
files, but have the config setting for people who want to be explicit - 
and in the docs just don't mention omitting the setting - tell people to 
choose one or the other. What do you think?

> So, in short, I simply start a general thread here to retrieve your 
> general feedback around these points.
> 
> Thanks for your feedbacks


[1] 
http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt

> Gaetan
> 
> [1]: https://review.openstack.org/#/c/524436/
> [2]: https://github.com/pypa/pipenv/issues/1911
> [3]: https://docs.pipenv.org/advanced/#pipfile-vs-setup-py
> [4]: https://www.python.org/dev/peps/pep-0518/
> [5]: library:
>    - pipenv to maintain Pipfile and Pipfile.lock
>    - Pipfile.lock not tracked (local reproductivity),
>    - pipenv-to-requirements [6] to generate a `requirements.txt` without 
> version freeze, also tracked
> applications:
>    - pipenv to maintain Pipfile and Pipfile.lock
>    - Pipfile.lock not tracked (global reproductivity),
>    - pipenv-to-requirements [6] to generate a `requirements.txt` and 
> `requirements-dev.txt` with version freeze, both tracked
> The development done with [1] should allow to get rid of [6].
> 
> [6] https://github.com/gsemet/pipenv-to-requirements
> -----
> Gaetan
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>