Open Stack

Excerpts from Matthew Thode's message of 2018-03-15 18:36:50 -0500:
> On 18-03-15 19:29:37, Doug Hellmann wrote:
> > Excerpts from Matthew Thode's message of 2018-03-15 10:24:10 -0500:
> > > On 18-03-15 07:03:11, Doug Hellmann wrote:
> > > > What I Want to Do
> > > > -----------------
> > > > 
> > > > 1. Update the requirements-check test job to change the check for
> > > >    an exact match to be a check for compatibility with the
> > > >    upper-constraints.txt value.
> > > > 
> > > >    We would check the value for the dependency from upper-constraints.txt
> > > >    against the range of allowed values in the project. If the
> > > >    constraint version is compatible, the dependency range is OK.
> > > > 
> > > >    This rule means that in order to change the dependency settings
> > > >    for a project in a way that are incompatible with the constraint,
> > > >    the constraint (and probably the global requirements list) would
> > > >    have to be changed first in openstack/requirements. However, if
> > > >    the change to the dependency is still compatible with the
> > > >    constraint, no change would be needed in openstack/requirements.
> > > >    For example, if the global list constraints a library to X.Y.Z
> > > >    and a project lists X.Y.Z-2 as the minimum version but then needs
> > > >    to raise that because it needs a feature in X.Y.Z-1, it can do
> > > >    that with a single patch in-tree.
> > > > 
> > > 
> > > I think what may be better is for global-requirements to become a
> > > gathering place for projects that requirements watches to have their
> > > smallest constrainted installable set defined in.
> > > 
> > > Upper-constraints has a req of foo===2.0.3
> > > Project A has a req of foo>=1.0.0,!=1.6.0
> > > Project B has a req of foo>=1.4.0
> > > Global reqs would be updated with foo>=1.4.0,!=1.6.0
> > > Project C comes along and sets foo>=2.0.0
> > > Global reqs would be updated with foo>=2.0.0
> > > 
> > > This would make global-reqs descriptive rather than prescriptive for
> > > versioning and would represent the 'true' version constraints of
> > > openstack.
> > 
> > It sounds like you're suggesting syncing in the other direction, which
> > could be useful. I think we can proceed with what I've described and
> > consider the work to build what you describe as a separate project.
> > 
> 
> Yes, this would be a follow-on thing.
> 
> > > 
> > > >    We also need to change requirements-check to look at the exclusions
> > > >    to ensure they all appear in the global-requirements.txt list
> > > >    (the local list needs to be a subset of the global list, but
> > > >    does not have to match it exactly). We can't have one project
> > > >    excluding a version that others do not, because we could then
> > > >    end up with a conflict with the upper constraints list that could
> > > >    wedge the gate as we had happen in the past.
> > > > 
> > > 
> > > How would this happen when using constraints?  A project is not allowed
> > > to have a requirement that masks a constriant (and would be verified via
> > > the requirements-check job).
> > 
> > If project A excludes version X before the constraint list is updated to
> > use it, and then project B starts trying to depend on version X, they
> > become incompatible.
> > 
> > We need to continue to manage our declarations of incompatible versions
> > to ensure that the constraints list is a good list of versions to test
> > everything under.
> > 
> > > There's a failure mode not covered, a project could add a mask (!=) to
> > > their requirements before we update constraints.  The project that was
> > > passing the requirements-check job would then become incompatable.  This
> > > means that the requirements-check would need to be run for each
> > > changeset to catch this as soon as it happens, instead of running only
> > > on requirements changes.
> > 
> > I'm not clear on what you're describing here, but it sounds like a
> > variation of the failure modes that would be prevented if we require
> > exclusions to exist in the global list before they could be added to the
> > local list.
> > 
> 
> Yes, that'd work (require exclusions to be global before local).

OK. That's what I was trying to describe as the new rules.

> 
> > > 
> > > >    We also need to verify that projects do not cap dependencies for
> > > >    the same reason. Caps prevent us from advancing to versions of
> > > >    dependencies that are "too new" and possibly incompatible. We
> > > >    can manage caps in the global requirements list, which would
> > > >    cause that list to calculate the constraints correctly.
> > > > 
> > > >    This change would immediately allow all projects currently
> > > >    following the global requirements lists to specify different
> > > >    lower bounds from that global list, as long as those lower bounds
> > > >    still allow the dependencies to be co-installable. (The upper
> > > >    bounds, managed through the upper-constraints.txt list, would
> > > >    still be built by selecting the newest compatible version because
> > > >    that is how pip's dependency resolver works.)
> > > > 
> > > > 2. We should stop syncing dependencies by turning off the
> > > >    propose-update-requirements job entirely.
> > > > 
> > > >    Turning off the job will stop the bot from proposing more
> > > >    dependency updates to projects.
> > > > 
> > > >    As part of deleting the job we can also remove the "requirements"
> > > >    case from playbooks/proposal/propose_update.sh, since it won't
> > > >    need that logic any more. We can also remove the update-requirements
> > > >    command from the openstack/requirements repository, since that
> > > >    is the tool that generates the updated list and it won't be
> > > >    needed if we aren't proposing updates any more.
> > > > 
> > > > 3. Remove the minimum specifications from the global requirements
> > > >    list to make clear that the global list is no longer expressing
> > > >    minimums.
> > > > 
> > > >    This clean-up step has been a bit more controversial among the
> > > >    requirements team, but I think it is a key piece. As the minimum
> > > >    versions of dependencies diverge within projects, there will no
> > > >    longer *be* a real global set of minimum values. Tracking a list of
> > > >    "highest minimums", would either require rebuilding the list from the
> > > >    settings in all projects, or requiring two patches to change the
> > > >    minimum version of a dependency within a project.
> > > > 
> > > >    Maintaining a global list of minimums also implies that we
> > > >    consider it OK to run OpenStack as a whole with that list. This
> > > >    message conflicts with the message we've been sending about the
> > > >    upper constraints list since that was established, which is that
> > > >    we have a known good list of versions and deploying all of
> > > >    OpenStack with different versions of those dependencies is
> > > >    untested.
> > > > 
> > > 
> > > As noted above I think that gathering the min versions/maskings from
> > > openstack projects to be valuable (especially to packagers who already
> > > use our likely invalid values already).
> > 
> > OK. I don't feel that strongly about the cleanup work, so if we want to
> > keep the lower bounds in place I think that's OK.
> > 
> > > 
> > > > After these 3 steps are done, the requirements team will continue
> > > > to maintain the global-requirements.txt and upper-constraints.txt
> > > > files, as before. Adding a new dependency to a project will still
> > > > involve a review step to add it to the global list so we can monitor
> > > > licensing, duplication, python 3 support, etc. But adjusting the
> > > > version numbers once that dependency is in the global list will be
> > > > easier.
> > > > 
> > > 
> > > Thanks for writing this up, I think it looks good in general, but like
> > > you mentioned before, there is some discussion to be had about gathering
> > > and creating a versionspec from all of openstack for requirements.
> > > 
> > 
>