<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 19, 2020, at 10:37 AM, Doug Hellmann <<a href="mailto:doug@doughellmann.com" class="">doug@doughellmann.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">On Apr 18, 2020, at 1:29 PM, Jeremy Stanley <<a href="mailto:fungi@yuggoth.org" class="">fungi@yuggoth.org</a>> wrote:<br class=""><br class="">On 2020-04-18 18:11:58 +0200 (+0200), Andreas Jaeger wrote:<br class=""><blockquote type="cite" class="">More fun:<br class=""><br class="">change <a href="https://review.opendev.org/720877" class="">https://review.opendev.org/720877</a> removes:<br class=""><br class="">install_command = pip install {opts} {packages}<br class=""><br class="">And suddenly lower-constraints fails - and the failure looks correct, so<br class="">removing it is fine but will cause a surprise ;(<br class=""></blockquote><br class="">After analyzing this over IRC, it looks like<br class="">openstack/networking-hyperv probably has an incoherent<br class="">lower-constraints.txt file because it asks for (one of many<br class="">examples) neutron-lib==1.28.0 and oslo.concurrency==3.25.0 but<br class="">neutron-lib 1.28.0 declares it requires oslo.concurrency>=3.26.0 so<br class="">this won't work.<br class=""><br class="">The lower constraints jobs are built on the premise that they're<br class="">working with a satisfiable set of constraints. Any time you<br class="">alter a lower bound for any dependency or add a new dependency,<br class="">you'll need to *fully* rebuild your lower-constraints.txt because of<br class="">the possible knock-on effects such a change can have on the<br class="">transitive dependencies tracked there.<br class=""><br class=""><a href="https://review.opendev.org/675572" class="">https://review.opendev.org/675572</a> replaced neutron-lib>=1.18.0 with<br class="">neutron-lib>=1.28.0 in the requirements.txt, but only edited the<br class="">corresponding entry in lower-constraints.txt instead of taking all<br class="">of neutron-lib's dependencies (and their dependencies, and so on)<br class="">into account. The result is a lower constraints job which isn't<br class="">testing what they think it's testing.<br class=""><br class="">The good news is that's all fixable. The bad news is that this<br class="">problem is probably widespread, because the places I expected to<br class="">talk about how to properly maintain your lower-constraints.txt file<br class="">provide minimal and potentially misleading advice:<br class=""><br class=""><a href="https://docs.openstack.org/project-team-guide/dependency-management.html#adding-a-new-dependency" class="">https://docs.openstack.org/project-team-guide/dependency-management.html#adding-a-new-dependency</a><br class=""><br class="">https://docs.openstack.org/project-team-guide/dependency-management.html#updating-the-minimum-version-of-a-dependency<br class=""><br class="">Neither of those suggest rebuilding your lower-constraints.txt when<br class="">you alter a lower bound or add a new requirement. Further, I thought<br class="">I remembered there being a utility in the openstack/requirements<br class="">repository for correctly generating a lower-constraints.txt file,<br class="">but this does not actually exist that I can find.<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">IIRC, the scripting I did way back when [1] was pretty naive. It looked at requirements.txt, converted >= to == and assumed either that any second order dependencies installed as a result of that set of constraints were the lowest likely to work or that teams would take care of editing the list.</span><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">There did used to be an update-requirements tool in the requirements repo, but it was removed as part of that work and I’m not sure it ever dealt with lower bounds.</span><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">[1]<span class="Apple-converted-space"> </span></span><a href="http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html" style="font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html</a><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="">The more I think back, the more I'm starting to remember that I<br class="">suggested the only thorough way (short of reimplementing pip's<br class="">version selection logic ourselves) to generate a lower constraints<br class="">file from the lower bounds in a requirements set would be by using<br class="">an altered copy of pip which inverted its version comparisons so<br class="">that it selected the lowest available versions of packages which<br class="">would satisfy every version range rather than the highest. I don't<br class="">think anybody has made that, and so I expect instead we've<br class="">collectively hand-waved with an assumption that people would be able<br class="">to figure out a working set of lower constraints instead (which is<br class="">in my opinion rather unrealistic given the complexity of the<br class="">transitive dependency trees of even some of our medium-sized<br class="">projects).<br class=""></blockquote></div></blockquote><div><br class=""></div><div><br class=""></div><a href="https://github.com/pypa/pip/pull/8086" class="">https://github.com/pypa/pip/pull/8086</a> might be useful. It doesn’t force the use of lower bounds, but it should result in the minimum versions being selected where possible.</div><div><br class=""><blockquote type="cite" class=""><div class=""><blockquote type="cite" style="font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="">The upshot of this is that correctly calculating a lower constraint<br class="">set for any project is likely to involve a lot of trial and error,<br class="">probably far more than most projects are going to want to bear. If<br class="">we had a tool we could run to generate a coherent<br class="">lower-constraints.txt for a project, then this might be different,<br class="">but I suspect that too is going to be more hassle than anyone wants<br class="">to invest their time in creating.<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Yes, it seems that very few distributions actually want the *lowest* values, either. What they seem to want is something that matches what they are already packaging, which is likely to be less than the version in the upper constraints list.</span></div></blockquote><blockquote type="cite" class=""><div class=""><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: BitstreamVeraSansMono-Roman; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">--<span class="Apple-converted-space"> </span><br class="">Jeremy Stanley</blockquote></div></blockquote></div><br class=""></body></html>