[stable][requirements][neutron] Capping pip in stable branches or not
bcafarel at redhat.com
Mon Dec 14 14:16:43 UTC 2020
Top-posting to recap all the interesting answers and answer my initial mail.
The overall feeling I get is that even with the changes that may be needed
to satisfy the new resolver, we should be fine to apply these to stable
* lower-constraints was discussed a lot, this is where largest changes were
spotted but they are OK given the current use/effectiveness of these jobs
(or maybe even dropped soon)
* linters can be extracted from test-requirements, to limit linters version
bumps. I had quickly tried that for the neutron fix and it had failed in
some other job, but I will take another look in a separate patch. Then if
needed this change can be squashed with pip requirements fixes in stable
* For some recent branches (victoria for example), style fixes are small so
this can be just cherry-picked from master to have a working branch
* Other requirements bumps should be OK as they actually indicate the
proper needed versions now
* If we ever hit a change (old third-pary dependency) that cannot be fixed
without going over upper-constraints, then we may have to cap pip.
Hopefully, this will not be hit.
fix for bandit 1.6.3 may help to limit the impact (I did not retest yet)
If all of this sounds good, then I guess it will be time to play
On Mon, 14 Dec 2020 at 14:03, Dmitry Tantsur <dtantsur at redhat.com> wrote:
> On Sun, Dec 13, 2020 at 5:36 PM Jeremy Stanley <fungi at yuggoth.org> wrote:
>> On 2020-12-13 14:39:58 +0100 (+0100), Luigi Toscano wrote:
>> > On Saturday, 12 December 2020 00:12:36 CET Jeremy Stanley wrote:
>> > > On 2020-12-11 20:38:30 +0000 (+0000), Sorin Sbarnea wrote:
>> > > [...]
>> > > > Regarding decoupling linting from test-requirements: yes! This was
>> > > > already done by some when conflicts appeared. For old branches I
>> > > > personally do not care much even if maintainers decide to disable
>> > > > linting, their main benefit is on main branches.
>> > > [...]
>> > >
>> > > To be honest, if I had my way, test-requirements.txt files would die
>> > > in a fire. Sure it's a little more work to be specific about the
>> > > individual requirements for each of your testenvs in tox.ini, but
>> > > the payoff is that people aren't needlessly installing bandit when
>> > > they run flake8 (for example). The thing we got into the PTI about
>> > > using a separate doc/requirements.txt is a nice compromise in that
>> > > direction, at least.
>> > Wouldn't this mean tracking requirements into two different kind
>> > of places:the main requirements.txt file, which is still going to
>> > be needed even for tests, and the tox environment definitions?
>> Technically we already do. The requirements.txt file contains actual
>> runtime Python dependencies of the software (technically
>> setup_requires in Setuptools parlance). Then we have this vague
>> test-requirements.txt file which installs everything under the sun
>> a test might want, including the kitchen sink. Tox doesn't reuse one
>> virtualenv for multiple testenv definitions, it creates a separate
>> one for each, so for example...
>> In the nova repo, if you `tox -e bandit` or `tox -e pep8` it's going
>> to install coverage, psycopg2, PyMySQL, requests,
>> python-barbicanclient, python-ironicclient, and a whole host of
>> other stuff, including the entire transitive dependency set for
>> everything in there, rather than just the one tool it needs to run.
>> I can't even run the pep8 testenv locally because to do that I
>> apparently need a Python package named zVMCloudConnector which wants
>> root access to create files like
>> /lib/systemd/system/sdkserver.service and
>> /etc/sudoers.d/sudoers-zvmsdk and /var/lib/zvmsdk/* and
>> /etc/zvmsdk/* in my system. WHAT?!? Do nova's developers actually
>> ever run any of this themselves?
>> Okay, so that one's actually in requirements.txt (might be a good
>> candidate for a separate extras in the setup.cfg instead), but
>> seriously, it's trying to install 182 packages (present count on
>> master) just to do a "quick" style check, and the resulting .tox
>> created from that is 319MB in size. How is that in any way sane? If
>> I tweak the testenv:pep8 definition in tox.ini to set
>> deps=flake8,hacking,mypy and and usedevelop=False, and set
>> skipsdist=True in the general tox section, it installs a total of 9
>> packages for a 36MB .tox directory. It's an extreme example, sure,
>> but remember this is also happening in CI for each patch uploaded,
>> and this setup cost is incurred every time in that context.
> Thanks for the hint btw, I'll apply it to our repos.
I will have to check that too, making these jobs lighter for CI is always
>> This is already solved in a few places in the nova repo, in
>> different ways. One is the docs testenv, which installs
>> doc/requirements.txt (currently 10 mostly Sphinx-related entries)
>> instead of combining all that into test-requirements.txt too.
>> Another is the osprofiler extra in setup.cfg allowing you to `pip
>> install nova[osprofiler]` to get that specific dependency. Yet still
>> another is the bindep testenv, which explicitly declares deps=bindep
>> and so installs absolutely nothing else (save bindep's own
>> dependencies)... or, well, it would except skipsdist got set to
>> False by https://review.openstack.org/622972 making that testenv
>> effectively pointless because now `tox -e bindep` has to install
>> nova before it can tell you what packages you're missing to be able
>> to install nova. *sigh*
>> So anyway, there's a lot of opportunity for improvement, and that's
>> just in nova, I'm sure there are similar situations throughout many
>> of our projects. Using a test-requirements.txt file as a dumping
>> ground for every last package any tox testenv could want may be
>> convenient for tracking things, but it's far from convenient to
>> actually use. The main thing we risk losing is that the
>> requirements-check job currently reports whether entries in
>> test-requirements.txt are compatible with the global
>> upper-constraints.txt in openstack/requirements, so extending that
>> to check dependencies declared in tox.ini or in package extras or
>> additional external requirements lists would be needed if we wanted
>> to preserve that capability.
>> Jeremy Stanley
> Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the openstack-discuss