<div dir="ltr">Its more than just the naming. In the original proposal, requirements.txt is the compiled list of all pinned deps (direct and transitive), while <a href="http://requirements.in">requirements.in</a> reflects what people will actually use. Whatever is in requirements.txt affects the egg's requires.txt. Instead, we can keep requirements.txt unchanged and have it still be the canonical list of dependencies, while reqiurements.out/requirements.gate/requirements.whatever is an upstream utility we produce and use to keep things sane on our slaves.<div><br></div><div>Maybe all we need is:</div><div><br></div><div>* update the existing post-merge job on the requirements repo to produce a requirements.txt (as it does now) as well the compiled version. </div><div><br></div><div>* modify devstack in some way with a toggle to have it process dependencies from the compiled version when necessary</div><div><br></div><div>I'm not sure how the second bit jives with the existing devstack installation code, specifically with the libraries from git-or-master but we can probably add something to warm the system with dependencies from the compiled version prior to calling pip/<a href="http://setup.py/etc">setup.py/etc</a></div><div><br></div><div>Adam</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 19, 2015 at 2:31 PM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Thu, Feb 19, 2015 at 1:48 PM, Adam Gandelman <span dir="ltr"><<a href="mailto:adamg@ubuntu.com" target="_blank">adamg@ubuntu.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">This creates a bit of a problem for downstream (packagers and probably others) Shipping a requirements.txt with explicit pins will end up producing an egg with a requires.txt that reflects those pins, unless there is some other magic planned that I'm not aware of. I can't speak for all packaging flavors, but I know debian packaging interacts quite closely with things like requirements.txt and resulting egg's requires.txt to determine appropriate system-level package dependencies. This would require a lot of tedious work on packagers part to get something functional.<div><br></div><div>What if its flipped? How about keeping requirements.txt with the caps, and using that as input to produce something like requirements.gate that passed to 'pip install --no-deps' on our slaves? We'd end up installing and using the explicit pinned requirements while the services/libraries themselves remain flexible. This might the issue Doug pointed out, where requirements updates across projects are not synchronized.</div><span><font color="#888888"><div><br></div></font></span></div></blockquote><div><br></div></span><div>Switching them to requirements.txt and requirements.gate works for me. If a simple renaming makes things better, then great!</div><div><br></div><div>As for Doug's comment, yes we need to work something out to overwrite requirements.gate, under your proposed naming, with global requirments</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><span><font color="#888888"><div></div><div>Adam </div><div><br></div><div><br></div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 19, 2015 at 12:59 PM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Wed, Feb 18, 2015 at 7:14 AM, Doug Hellmann <span dir="ltr"><<a href="mailto:doug@doughellmann.com" target="_blank">doug@doughellmann.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><br>
<br>
On Wed, Feb 18, 2015, at 10:07 AM, Donald Stufft wrote:<br>
><br>
> > On Feb 18, 2015, at 10:00 AM, Doug Hellmann <<a href="mailto:doug@doughellmann.com" target="_blank">doug@doughellmann.com</a>> wrote:<br>
> ><br>
> ><br>
> ><br>
> > On Tue, Feb 17, 2015, at 03:17 PM, Joe Gordon wrote:<br>
> >> On Tue, Feb 17, 2015 at 4:19 AM, Sean Dague <<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>> wrote:<br>
> >><br>
> >>> On 02/16/2015 08:50 PM, Ian Cordasco wrote:<br>
> >>>> On 2/16/15, 16:08, "Sean Dague" <<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>> wrote:<br>
> >>>><br>
> >>>>> On 02/16/2015 02:08 PM, Doug Hellmann wrote:<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> On Mon, Feb 16, 2015, at 01:01 PM, Ian Cordasco wrote:<br>
> >>>>>>> Hey everyone,<br>
> >>>>>>><br>
> >>>>>>> The os-ansible-deployment team was working on updates to add support<br>
> >>>>>>> for<br>
> >>>>>>> the latest version of juno and noticed some interesting version<br>
> >>>>>>> specifiers<br>
> >>>>>>> introduced into global-requirements.txt in January. It introduced some<br>
> >>>>>>> version specifiers that seem a bit impossible like the one for<br>
> >>> requests<br>
> >>>>>>> [1]. There are others that equate presently to pinning the versions of<br>
> >>>>>>> the<br>
> >>>>>>> packages [2, 3, 4].<br>
> >>>>>>><br>
> >>>>>>> I understand fully and support the commit because of how it improves<br>
> >>>>>>> pretty much everyone’s quality of life (no fires to put out in the<br>
> >>>>>>> middle<br>
> >>>>>>> of the night on the weekend). I’m also aware that a lot of the<br>
> >>>>>>> downstream<br>
> >>>>>>> redistributors tend to work from global-requirements.txt when<br>
> >>>>>>> determining<br>
> >>>>>>> what to package/support.<br>
> >>>>>>><br>
> >>>>>>> It seems to me like there’s room to clean up some of these<br>
> >>> requirements<br>
> >>>>>>> to<br>
> >>>>>>> make them far more explicit and less misleading to the human eye (even<br>
> >>>>>>> though tooling like pip can easily parse/understand these).<br>
> >>>>>><br>
> >>>>>> I think that's the idea. These requirements were generated<br>
> >>>>>> automatically, and fixed issues that were holding back several<br>
> >>> projects.<br>
> >>>>>> Now we can apply updates to them by hand, to either move the lower<br>
> >>>>>> bounds down (as in the case Ihar pointed out with stevedore) or clean<br>
> >>> up<br>
> >>>>>> the range definitions. We should not raise the limits of any Oslo<br>
> >>>>>> libraries, and we should consider raising the limits of third-party<br>
> >>>>>> libraries very carefully.<br>
> >>>>>><br>
> >>>>>> We should make those changes on one library at a time, so we can see<br>
> >>>>>> what effect each change has on the other requirements.<br>
> >>>>>><br>
> >>>>>>><br>
> >>>>>>> I also understand that stable-maint may want to occasionally bump the<br>
> >>>>>>> caps<br>
> >>>>>>> to see if newer versions will not break everything, so what is the<br>
> >>>>>>> right<br>
> >>>>>>> way forward? What is the best way to both maintain a stable branch<br>
> >>> with<br>
> >>>>>>> known working dependencies while helping out those who do so much work<br>
> >>>>>>> for<br>
> >>>>>>> us (downstream and stable-maint) and not permanently pinning to<br>
> >>> certain<br>
> >>>>>>> working versions?<br>
> >>>>>><br>
> >>>>>> Managing the upper bounds is still under discussion. Sean pointed out<br>
> >>>>>> that we might want hard caps so that updates to stable branch were<br>
> >>>>>> explicit. I can see either side of that argument and am still on the<br>
> >>>>>> fence about the best approach.<br>
> >>>>><br>
> >>>>> History has shown that it's too much work keeping testing functioning<br>
> >>>>> for stable branches if we leave dependencies uncapped. If particular<br>
> >>>>> people are interested in bumping versions when releases happen, it's<br>
> >>>>> easy enough to do with a requirements proposed update. It will even run<br>
> >>>>> tests that in most cases will prove that it works.<br>
> >>>>><br>
> >>>>> It might even be possible for someone to build some automation that did<br>
> >>>>> that as stuff from pypi released so we could have the best of both<br>
> >>>>> worlds. But I think capping is definitely something we want as a<br>
> >>>>> project, and it reflects the way that most deployments will consume this<br>
> >>>>> code.<br>
> >>>>><br>
> >>>>> -Sean<br>
> >>>>><br>
> >>>>> --<br>
> >>>>> Sean Dague<br>
> >>>>> <a href="http://dague.net" target="_blank">http://dague.net</a><br>
> >>>><br>
> >>>> Right. No one is arguing the very clear benefits of all of this.<br>
> >>>><br>
> >>>> I’m just wondering if for the example version identifiers that I gave in<br>
> >>>> my original message (and others that are very similar) if we want to make<br>
> >>>> the strings much simpler for people who tend to work from them (i.e.,<br>
> >>>> downstream re-distributors whose jobs are already difficult enough). I’ve<br>
> >>>> offered to help at least one of them in the past who maintains all of<br>
> >>>> their distro’s packages themselves, but they refused so I’d like to help<br>
> >>>> them anyway possible. Especially if any of them chime in as this being<br>
> >>>> something that would be helpful.<br>
> >>><br>
> >>> Ok, your links got kind of scrambled. Can you next time please inline<br>
> >>> the key relevant content in the email, because I think we all missed the<br>
> >>> original message intent as the key content was only in footnotes.<br>
> >>><br>
> >>> From my point of view, normalization patches would be fine.<br>
> >>><br>
> >>> requests>=1.2.1,!=2.4.0,<=2.2.1<br>
> >>><br>
> >>> Is actually an odd one, because that's still there because we're using<br>
> >>> Trusty level requests in the tests, and my ability to have devstack not<br>
> >>> install that has thus far failed.<br>
> >>><br>
> >>> Things like:<br>
> >>><br>
> >>> osprofiler>=0.3.0,<=0.3.0 # Apache-2.0<br>
> >>><br>
> >>> Can clearly be normalized to osprofiler==0.3.0 if you want to propose<br>
> >>> the patch manually.<br>
> >>><br>
> >><br>
> >> global-requirements for stable branches serves two uses:<br>
> >><br>
> >> 1. Specify the set of dependencies that we would like to test against<br>
> >> 2. A tool for downstream packagers to use when determining what to<br>
> >> package/support.<br>
> >><br>
> >> For #1, Ideally we would like a set of all dependencies, including<br>
> >> transitive, with explicit versions (very similar to the output of<br>
> >> pip-freeze). But for #2 the standard requirement file with a range is<br>
> >> preferred. Putting an upper bound on each dependency, instead of using a<br>
> >> '==' was a compromise between the two use cases.<br>
> >><br>
> >> Going forward I propose we have a <a href="http://requirements.in" target="_blank">requirements.in</a> and a requirements.txt<br>
> >> file. The <a href="http://requirements.in" target="_blank">requirements.in</a> file would contain the range of dependencies,<br>
> >> and<br>
> >> requirements.txt would contain the pinned set, and eventually the pinned<br>
> >> set including transitive dependencies.<br>
> >><br>
> >> Thoughts?<br>
> ><br>
> > I'm interested in seeing what that list looks like. I suspect we have<br>
> > some libraries listed in the global requirements now that aren't<br>
> > actually used, and I'm sure there is a long list of transitive<br>
> > dependencies to add to it.<br>
> ><br>
> > I'm not entirely comfortable with the idea of pinning completely, but I<br>
> > guess it's the best of two bad options. It solves the "we don't have<br>
> > enough people around to manage stable branches" problem in one way (by<br>
> > not letting releases outside our control break our test jobs), but if we<br>
> > don't have people around now to fix things who is going to keep up with<br>
> > updating that requirements list as new versions of projects come out? We<br>
> > can write a job to automatically detect new packages and test them, but<br>
> > who is going to review patches submitted by that bot? Maybe that's a<br>
> > small enough amount of work that it will be easier to find help.<br>
> ><br>
> > We've been playing whack-a-mole with issues because we made changes to<br>
> > the way we deal with versions and dependencies without fully<br>
> > understanding the consequences of some of the changes. They looked<br>
> > innocent at first, but because of assumptions in other jobs or other<br>
> > parts of the system they caused problems. So I think we should be<br>
> > careful about making this decision and think about some of the other<br>
> > things that might fall out before pushing more changes up.<br>
> ><br>
> > For example, if we're syncing requirements into stable branches of<br>
> > projects based on requirements.txt, and that becomes a set of pins<br>
> > instead of a set of ranges with caps, how do we update projects? Should<br>
> > we sync from <a href="http://requirements.in" target="_blank">requirements.in</a> instead of requirements.txt, to allow<br>
> > projects to maintain the ranges in their own requirements files? Or do<br>
> > we want those requirements files to reflect the pins from the global<br>
> > list?<br>
><br>
> I'm not sure I fully understand what folks are proposing here with two<br>
> different files, but if you’re putting ``==`` specifiers into the<br>
> install_requires of various projects, then I believe that is going to<br>
> cause a<br>
> fairly large amount of pain.<br>
<br>
</div></div>As I understand it, Joe's idea is to have an input file<br>
("<a href="http://requirements.in" target="_blank">requirements.in</a>") that uses >=, <=, and != to specify a range of valid<br>
versions. Those ranges would guide packagers about what requirements we<br>
think work. The list would also be "compiled" into a list of<br>
requirements using only == to create a requirements.txt file thatwould<br>
be used for the tests on our CI systems.<br></blockquote><div><br></div></div></div><div>That is correct. The workflow would be:</div><div><br></div><div>When preparing a stable branch:</div><div><br></div><div>In global requirements</div><div><br></div><div>* rename requirements.tx to <a href="http://requirements.in" target="_blank">requirements.in</a></div><div>* create a new requirements.txt file that consists of a full set of pinned, that is using '==', dependencies (transitive and all).</div><div><br></div><div>For each project consuming global requirements</div><div><br></div><div><div>* rename requirements.txt to <a href="http://requirements.in" target="_blank">requirements.in</a></div><div>* create a new requirements.txt file that consists of a full set of pinned, that is using '==', dependencies (transitive and all). While making sure the versions allign with global requirements</div></div><div>* Change how we install dependencies from 'pip install -r requirements' to 'pip install --no-deps -r requirements.txt</div><div> </div><div><br></div><div>So now we have two files for requirements. requirements.txt to specify the exact versions we use for testing, and <a href="http://requirements.in" target="_blank">requirements.in</a> to specify the range we think *should* work.</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><br>
><br>
> ---<br>
> Donald Stufft<br>
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA<br>
><br>
</span><span>> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</span>> Email had 1 attachment:<br>
> + signature.asc<br>
> 1k (application/pgp-signature)<br>
<div><div><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></span></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>