[openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

Matthew Thode prometheanfire at gentoo.org
Thu Mar 15 15:24:10 UTC 2018

On 18-03-15 07:03:11, Doug Hellmann wrote:
> What I Want to Do
> -----------------
> 1. Update the requirements-check test job to change the check for
>    an exact match to be a check for compatibility with the
>    upper-constraints.txt value.
>    We would check the value for the dependency from upper-constraints.txt
>    against the range of allowed values in the project. If the
>    constraint version is compatible, the dependency range is OK.
>    This rule means that in order to change the dependency settings
>    for a project in a way that are incompatible with the constraint,
>    the constraint (and probably the global requirements list) would
>    have to be changed first in openstack/requirements. However, if
>    the change to the dependency is still compatible with the
>    constraint, no change would be needed in openstack/requirements.
>    For example, if the global list constraints a library to X.Y.Z
>    and a project lists X.Y.Z-2 as the minimum version but then needs
>    to raise that because it needs a feature in X.Y.Z-1, it can do
>    that with a single patch in-tree.

I think what may be better is for global-requirements to become a
gathering place for projects that requirements watches to have their
smallest constrainted installable set defined in.

Upper-constraints has a req of foo===2.0.3
Project A has a req of foo>=1.0.0,!=1.6.0
Project B has a req of foo>=1.4.0
Global reqs would be updated with foo>=1.4.0,!=1.6.0
Project C comes along and sets foo>=2.0.0
Global reqs would be updated with foo>=2.0.0

This would make global-reqs descriptive rather than prescriptive for
versioning and would represent the 'true' version constraints of

>    We also need to change requirements-check to look at the exclusions
>    to ensure they all appear in the global-requirements.txt list
>    (the local list needs to be a subset of the global list, but
>    does not have to match it exactly). We can't have one project
>    excluding a version that others do not, because we could then
>    end up with a conflict with the upper constraints list that could
>    wedge the gate as we had happen in the past.

How would this happen when using constraints?  A project is not allowed
to have a requirement that masks a constriant (and would be verified via
the requirements-check job).

There's a failure mode not covered, a project could add a mask (!=) to
their requirements before we update constraints.  The project that was
passing the requirements-check job would then become incompatable.  This
means that the requirements-check would need to be run for each
changeset to catch this as soon as it happens, instead of running only
on requirements changes.

>    We also need to verify that projects do not cap dependencies for
>    the same reason. Caps prevent us from advancing to versions of
>    dependencies that are "too new" and possibly incompatible. We
>    can manage caps in the global requirements list, which would
>    cause that list to calculate the constraints correctly.
>    This change would immediately allow all projects currently
>    following the global requirements lists to specify different
>    lower bounds from that global list, as long as those lower bounds
>    still allow the dependencies to be co-installable. (The upper
>    bounds, managed through the upper-constraints.txt list, would
>    still be built by selecting the newest compatible version because
>    that is how pip's dependency resolver works.)
> 2. We should stop syncing dependencies by turning off the
>    propose-update-requirements job entirely.
>    Turning off the job will stop the bot from proposing more
>    dependency updates to projects.
>    As part of deleting the job we can also remove the "requirements"
>    case from playbooks/proposal/propose_update.sh, since it won't
>    need that logic any more. We can also remove the update-requirements
>    command from the openstack/requirements repository, since that
>    is the tool that generates the updated list and it won't be
>    needed if we aren't proposing updates any more.
> 3. Remove the minimum specifications from the global requirements
>    list to make clear that the global list is no longer expressing
>    minimums.
>    This clean-up step has been a bit more controversial among the
>    requirements team, but I think it is a key piece. As the minimum
>    versions of dependencies diverge within projects, there will no
>    longer *be* a real global set of minimum values. Tracking a list of
>    "highest minimums", would either require rebuilding the list from the
>    settings in all projects, or requiring two patches to change the
>    minimum version of a dependency within a project.
>    Maintaining a global list of minimums also implies that we
>    consider it OK to run OpenStack as a whole with that list. This
>    message conflicts with the message we've been sending about the
>    upper constraints list since that was established, which is that
>    we have a known good list of versions and deploying all of
>    OpenStack with different versions of those dependencies is
>    untested.

As noted above I think that gathering the min versions/maskings from
openstack projects to be valuable (especially to packagers who already
use our likely invalid values already).

> After these 3 steps are done, the requirements team will continue
> to maintain the global-requirements.txt and upper-constraints.txt
> files, as before. Adding a new dependency to a project will still
> involve a review step to add it to the global list so we can monitor
> licensing, duplication, python 3 support, etc. But adjusting the
> version numbers once that dependency is in the global list will be
> easier.

Thanks for writing this up, I think it looks good in general, but like
you mentioned before, there is some discussion to be had about gathering
and creating a versionspec from all of openstack for requirements.

Matthew Thode (prometheanfire)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180315/21a99373/attachment.sig>

More information about the OpenStack-dev mailing list