[openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

Robert Collins robertc at robertcollins.net
Mon Jun 8 17:56:55 UTC 2015

On 9 June 2015 at 04:19, Doug Wiegley <dougwig at parksidesoftware.com> wrote:
>> On Jun 8, 2015, at 9:58 AM, Jeremy Stanley <fungi at yuggoth.org> wrote:
>> On 2015-06-08 13:29:50 +1200 (+1200), Robert Collins wrote:
>> [...]
>>> However, we are going to move from test-requirements.txt to setup.cfg
>>> eventually, but thats a separate transition - and one could still use
>>> test-requirements.txt there to provide git references.
>> Except please don't. If you put a git requirement entry in that file
>> for neutron, every CI job which uses that test-requirements.txt will
>> reclone all of neutron over the network from scratch because pip
>> isn't smart enough to do otherwise unless you take additional
>> measures to preinstall that requirement in the environment where the
>> test is run. That's why we use tools like devstack-gate or
>> zuul-cloner which know how to check for cached repos and update them to
>> the ref that you've (or that zuul has) requested.
> The neutron-*aas repos were among the worst offenders in the ‘ninja cloning’ racket, and in addition to working against the CI infrastructure, it bit us by testing master instead of the review patches in some cases. Note that we were grabbing neutron at master and tempest at a pinned commit. And since we use the same tox env’s for devs and CI, there are slightly different requirements depending on which is in use.

tempest-lib or tempest? tempest-lib cuts releases and should AIUI be
consumed via those: if you need something thats not in the latest
release, ask that it be released.

> What I am attempting to do, and I’m open to feedback on ‘best practices’:
> - pep8/unit tests - override ‘install_program’ in tox, to be an in-repo script that detects if you’re inside jenkins or not, and then uses either zuul-cloner or pulls in neutron via a git url, respectively, then calls ‘pip install’ as usual for the rest.

So, we're going to be adding in this thing for constraints over this
cycle (https://review.openstack.org/#/c/186635/) and it will also be
gluing into tox in that sort of way. Since you can specify a git url
and reference there, I wonder if editing the constraints file
just-in-time to add e.g. a neutron reference, will be useful. Or yeah
zuul-cloner. We'll need to make sure these things play nice together.

> - tempest api tests - pulled in-tree the subset of tempest that we’re using and which is not yet in tempest-lib, with an eye to migrating it away. This part could’ve also been done via zuul-cloner at a specific commit.

I'd get the stuff into tempest-lib ASAP rather than re-arranging the
way this is done IMO.

> - We do *not* put ‘neutron’ into any requirements file, and I wasn’t planning to, until such time as neutron, or neutron-lib, is stable and published to pypi.

Without neutron there, there is no signal to packagers, deployers, or
pip that neutron is needed. I think thats just bugs waiting to happen.


Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

More information about the OpenStack-dev mailing list