<div dir="ltr">This creates a bit of a problem for downstream (packagers and probably others) Shipping a requirements.txt with explicit pins will end up producing an egg with a requires.txt that reflects those pins, unless there is some other magic planned that I'm not aware of. I can't speak for all packaging flavors, but I know debian packaging interacts quite closely with things like requirements.txt and resulting egg's requires.txt to determine appropriate system-level package dependencies. This would require a lot of tedious work on packagers part to get something functional.<div><br></div><div>What if its flipped? How about keeping requirements.txt with the caps, and using that as input to produce something like requirements.gate that passed to 'pip install --no-deps' on our slaves? We'd end up installing and using the explicit pinned requirements while the services/libraries themselves remain flexible. This might the issue Doug pointed out, where requirements updates across projects are not synchronized.</div><div><br></div><div>Adam </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 19, 2015 at 12:59 PM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Wed, Feb 18, 2015 at 7:14 AM, Doug Hellmann <span dir="ltr"><<a href="mailto:doug@doughellmann.com" target="_blank">doug@doughellmann.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><br>
<br>
On Wed, Feb 18, 2015, at 10:07 AM, Donald Stufft wrote:<br>
><br>
> > On Feb 18, 2015, at 10:00 AM, Doug Hellmann <<a href="mailto:doug@doughellmann.com" target="_blank">doug@doughellmann.com</a>> wrote:<br>
> ><br>
> ><br>
> ><br>
> > On Tue, Feb 17, 2015, at 03:17 PM, Joe Gordon wrote:<br>
> >> On Tue, Feb 17, 2015 at 4:19 AM, Sean Dague <<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>> wrote:<br>
> >><br>
> >>> On 02/16/2015 08:50 PM, Ian Cordasco wrote:<br>
> >>>> On 2/16/15, 16:08, "Sean Dague" <<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>> wrote:<br>
> >>>><br>
> >>>>> On 02/16/2015 02:08 PM, Doug Hellmann wrote:<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> On Mon, Feb 16, 2015, at 01:01 PM, Ian Cordasco wrote:<br>
> >>>>>>> Hey everyone,<br>
> >>>>>>><br>
> >>>>>>> The os-ansible-deployment team was working on updates to add support<br>
> >>>>>>> for<br>
> >>>>>>> the latest version of juno and noticed some interesting version<br>
> >>>>>>> specifiers<br>
> >>>>>>> introduced into global-requirements.txt in January. It introduced some<br>
> >>>>>>> version specifiers that seem a bit impossible like the one for<br>
> >>> requests<br>
> >>>>>>> [1]. There are others that equate presently to pinning the versions of<br>
> >>>>>>> the<br>
> >>>>>>> packages [2, 3, 4].<br>
> >>>>>>><br>
> >>>>>>> I understand fully and support the commit because of how it improves<br>
> >>>>>>> pretty much everyone’s quality of life (no fires to put out in the<br>
> >>>>>>> middle<br>
> >>>>>>> of the night on the weekend). I’m also aware that a lot of the<br>
> >>>>>>> downstream<br>
> >>>>>>> redistributors tend to work from global-requirements.txt when<br>
> >>>>>>> determining<br>
> >>>>>>> what to package/support.<br>
> >>>>>>><br>
> >>>>>>> It seems to me like there’s room to clean up some of these<br>
> >>> requirements<br>
> >>>>>>> to<br>
> >>>>>>> make them far more explicit and less misleading to the human eye (even<br>
> >>>>>>> though tooling like pip can easily parse/understand these).<br>
> >>>>>><br>
> >>>>>> I think that's the idea. These requirements were generated<br>
> >>>>>> automatically, and fixed issues that were holding back several<br>
> >>> projects.<br>
> >>>>>> Now we can apply updates to them by hand, to either move the lower<br>
> >>>>>> bounds down (as in the case Ihar pointed out with stevedore) or clean<br>
> >>> up<br>
> >>>>>> the range definitions. We should not raise the limits of any Oslo<br>
> >>>>>> libraries, and we should consider raising the limits of third-party<br>
> >>>>>> libraries very carefully.<br>
> >>>>>><br>
> >>>>>> We should make those changes on one library at a time, so we can see<br>
> >>>>>> what effect each change has on the other requirements.<br>
> >>>>>><br>
> >>>>>>><br>
> >>>>>>> I also understand that stable-maint may want to occasionally bump the<br>
> >>>>>>> caps<br>
> >>>>>>> to see if newer versions will not break everything, so what is the<br>
> >>>>>>> right<br>
> >>>>>>> way forward? What is the best way to both maintain a stable branch<br>
> >>> with<br>
> >>>>>>> known working dependencies while helping out those who do so much work<br>
> >>>>>>> for<br>
> >>>>>>> us (downstream and stable-maint) and not permanently pinning to<br>
> >>> certain<br>
> >>>>>>> working versions?<br>
> >>>>>><br>
> >>>>>> Managing the upper bounds is still under discussion. Sean pointed out<br>
> >>>>>> that we might want hard caps so that updates to stable branch were<br>
> >>>>>> explicit. I can see either side of that argument and am still on the<br>
> >>>>>> fence about the best approach.<br>
> >>>>><br>
> >>>>> History has shown that it's too much work keeping testing functioning<br>
> >>>>> for stable branches if we leave dependencies uncapped. If particular<br>
> >>>>> people are interested in bumping versions when releases happen, it's<br>
> >>>>> easy enough to do with a requirements proposed update. It will even run<br>
> >>>>> tests that in most cases will prove that it works.<br>
> >>>>><br>
> >>>>> It might even be possible for someone to build some automation that did<br>
> >>>>> that as stuff from pypi released so we could have the best of both<br>
> >>>>> worlds. But I think capping is definitely something we want as a<br>
> >>>>> project, and it reflects the way that most deployments will consume this<br>
> >>>>> code.<br>
> >>>>><br>
> >>>>> -Sean<br>
> >>>>><br>
> >>>>> --<br>
> >>>>> Sean Dague<br>
> >>>>> <a href="http://dague.net" target="_blank">http://dague.net</a><br>
> >>>><br>
> >>>> Right. No one is arguing the very clear benefits of all of this.<br>
> >>>><br>
> >>>> I’m just wondering if for the example version identifiers that I gave in<br>
> >>>> my original message (and others that are very similar) if we want to make<br>
> >>>> the strings much simpler for people who tend to work from them (i.e.,<br>
> >>>> downstream re-distributors whose jobs are already difficult enough). I’ve<br>
> >>>> offered to help at least one of them in the past who maintains all of<br>
> >>>> their distro’s packages themselves, but they refused so I’d like to help<br>
> >>>> them anyway possible. Especially if any of them chime in as this being<br>
> >>>> something that would be helpful.<br>
> >>><br>
> >>> Ok, your links got kind of scrambled. Can you next time please inline<br>
> >>> the key relevant content in the email, because I think we all missed the<br>
> >>> original message intent as the key content was only in footnotes.<br>
> >>><br>
> >>> From my point of view, normalization patches would be fine.<br>
> >>><br>
> >>> requests>=1.2.1,!=2.4.0,<=2.2.1<br>
> >>><br>
> >>> Is actually an odd one, because that's still there because we're using<br>
> >>> Trusty level requests in the tests, and my ability to have devstack not<br>
> >>> install that has thus far failed.<br>
> >>><br>
> >>> Things like:<br>
> >>><br>
> >>> osprofiler>=0.3.0,<=0.3.0 # Apache-2.0<br>
> >>><br>
> >>> Can clearly be normalized to osprofiler==0.3.0 if you want to propose<br>
> >>> the patch manually.<br>
> >>><br>
> >><br>
> >> global-requirements for stable branches serves two uses:<br>
> >><br>
> >> 1. Specify the set of dependencies that we would like to test against<br>
> >> 2. A tool for downstream packagers to use when determining what to<br>
> >> package/support.<br>
> >><br>
> >> For #1, Ideally we would like a set of all dependencies, including<br>
> >> transitive, with explicit versions (very similar to the output of<br>
> >> pip-freeze). But for #2 the standard requirement file with a range is<br>
> >> preferred. Putting an upper bound on each dependency, instead of using a<br>
> >> '==' was a compromise between the two use cases.<br>
> >><br>
> >> Going forward I propose we have a <a href="http://requirements.in" target="_blank">requirements.in</a> and a requirements.txt<br>
> >> file. The <a href="http://requirements.in" target="_blank">requirements.in</a> file would contain the range of dependencies,<br>
> >> and<br>
> >> requirements.txt would contain the pinned set, and eventually the pinned<br>
> >> set including transitive dependencies.<br>
> >><br>
> >> Thoughts?<br>
> ><br>
> > I'm interested in seeing what that list looks like. I suspect we have<br>
> > some libraries listed in the global requirements now that aren't<br>
> > actually used, and I'm sure there is a long list of transitive<br>
> > dependencies to add to it.<br>
> ><br>
> > I'm not entirely comfortable with the idea of pinning completely, but I<br>
> > guess it's the best of two bad options. It solves the "we don't have<br>
> > enough people around to manage stable branches" problem in one way (by<br>
> > not letting releases outside our control break our test jobs), but if we<br>
> > don't have people around now to fix things who is going to keep up with<br>
> > updating that requirements list as new versions of projects come out? We<br>
> > can write a job to automatically detect new packages and test them, but<br>
> > who is going to review patches submitted by that bot? Maybe that's a<br>
> > small enough amount of work that it will be easier to find help.<br>
> ><br>
> > We've been playing whack-a-mole with issues because we made changes to<br>
> > the way we deal with versions and dependencies without fully<br>
> > understanding the consequences of some of the changes. They looked<br>
> > innocent at first, but because of assumptions in other jobs or other<br>
> > parts of the system they caused problems. So I think we should be<br>
> > careful about making this decision and think about some of the other<br>
> > things that might fall out before pushing more changes up.<br>
> ><br>
> > For example, if we're syncing requirements into stable branches of<br>
> > projects based on requirements.txt, and that becomes a set of pins<br>
> > instead of a set of ranges with caps, how do we update projects? Should<br>
> > we sync from <a href="http://requirements.in" target="_blank">requirements.in</a> instead of requirements.txt, to allow<br>
> > projects to maintain the ranges in their own requirements files? Or do<br>
> > we want those requirements files to reflect the pins from the global<br>
> > list?<br>
><br>
> I'm not sure I fully understand what folks are proposing here with two<br>
> different files, but if you’re putting ``==`` specifiers into the<br>
> install_requires of various projects, then I believe that is going to<br>
> cause a<br>
> fairly large amount of pain.<br>
<br>
</div></div>As I understand it, Joe's idea is to have an input file<br>
("<a href="http://requirements.in" target="_blank">requirements.in</a>") that uses >=, <=, and != to specify a range of valid<br>
versions. Those ranges would guide packagers about what requirements we<br>
think work. The list would also be "compiled" into a list of<br>
requirements using only == to create a requirements.txt file thatwould<br>
be used for the tests on our CI systems.<br></blockquote><div><br></div></div></div><div>That is correct. The workflow would be:</div><div><br></div><div>When preparing a stable branch:</div><div><br></div><div>In global requirements</div><div><br></div><div>* rename requirements.tx to <a href="http://requirements.in" target="_blank">requirements.in</a></div><div>* create a new requirements.txt file that consists of a full set of pinned, that is using '==', dependencies (transitive and all).</div><div><br></div><div>For each project consuming global requirements</div><div><br></div><div><div>* rename requirements.txt to <a href="http://requirements.in" target="_blank">requirements.in</a></div><div>* create a new requirements.txt file that consists of a full set of pinned, that is using '==', dependencies (transitive and all). While making sure the versions allign with global requirements</div></div><div>* Change how we install dependencies from 'pip install -r requirements' to 'pip install --no-deps -r requirements.txt</div><div> </div><div><br></div><div>So now we have two files for requirements. requirements.txt to specify the exact versions we use for testing, and <a href="http://requirements.in" target="_blank">requirements.in</a> to specify the range we think *should* work.</div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><br>
><br>
> ---<br>
> Donald Stufft<br>
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA<br>
><br>
</span><span>> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</span>> Email had 1 attachment:<br>
> + signature.asc<br>
> 1k (application/pgp-signature)<br>
<div><div><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></span></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>