Open Stack

Hi everybody,

We've been working on navigating through from an interesting situation 
over the past few months, but there isn't a top-level overview of what's 
going on with it. That's my bad - I've been telling AJaeger I was going 
to send an email out for a while.

projects with test requirements on git repo urls of other projects
------------------------------------------------------------------

There are a bunch of projects that need, for testing purposes, to depend 
on other projects. The majority are either neutron or horizon plugins, 
but conceptually there is nothing neutron or horizon specific about the 
issue. The problem they're trying to deal with is that they are a plugin 
to a service and they need to be able to import code from the service 
they are a plugin to in their unit tests.

To make things even more complicated, some of the plugins actually 
duepend on each other for real, not just as a "we need this for testing"

There is trouble in paradise though - which is that we don't allow git 
urls in requirements files. To work around this, the projects in 
question added additional pip install lines to a tox_install.sh script - 
essentially bypassing the global-requirements process and system 
completely.

This went unnoticed in a general sense until we started working through 
removing the use of zuul-cloner which is not needed any longer in Zuul v3.

unwinding things
----------------

There are a few different options, but it's important to keep in mind 
that we ultimately want all of the following:

* The code works
* Tests can run properly in CI
* "Depends-On" works in CI so that you can test changes cross-repo
* Tests can run properly locally for developers
* Deployment requirements are accurately communicated to deployers

The approach so far
-------------------

The approach so far has been releasing service projects to PyPI and 
reworking the projects to depend on those releases.

This approach takes advantage of the tox-siblings feature in the gate to 
ensure we're cross-testing master of projects with each other.

tox-siblings
-----------

There is a feature in the Zuul tox jobs we refer to as "tox-siblings" 
(this is because historically - wow we have historical context for zuul 
v3 now - it was implemented as a separate role) What it does is ensure 
that if you are running a tox job and you add additional projects to 
required-projects in the job config, that the git versions of those 
projects will be installed into the tox virtualenv - but only for 
projects that would have been installed by tox otherwise. This way 
required-projects is both safe to use and has the effect you'd expect.

tox-siblings is intended to enable ADDITIONALLY cross-testing projects 
that otherwise have a normal dependency relationship in the gate. People 
have been adding jobs like cross-something-something or something-tips 
in an ad-hoc manner for a while - and in many cases the git parts of 
that were actually somewhat not correct - so this is an attempt to 
provide the thing people want in those scenarios in a consistent manner. 
But it always should be helper logic for more complex gate jobs, not as 
a de-facto part of a project's basic install.

Current Approach is wrong
------------------------

Unfortunately, as part of trying to unwind the plugins situation, we've 
walked ourselves into a situation where the gate is the only thing that 
has the correct installation information for some projects, and that's 
not good.

 From a networking plugin approach the "depend on release and use 
tox-siblings" assumes that 'depend on release of neutron' is or can be 
the common case with the ability to add a second tox job to check master 
against master.

If that's not a real thing, then depending on releases + tox_siblings in 
the gate is solving the wrong problem.

Specific Suggestions
--------------------

As there are a few different scenarios, I want to suggest we do a few 
different things.

* Prefer interface libraries on PyPI that projects depend on

Like python-openstackclient and osc-lib, this is the *best* approach
for projects with plugins. Such interface libraries need to be able to 
do intermediate releases - and those intermediate releases need to not 
break the released version of the projects. This is the hardest and 
longest thing to do as well, so it's most likely to be a multi-cycle effort.

* Treat inter-plugin depends as normal library depends

If networking-bgpvpn depends on networking-bagpipe and networking-odl, 
then networking-bagpipe and networking-odl need to be released to PyPI 
just like any other library in OpenStack. These are real runtime 
dependencies.

Yes, this is more coordination work, but it's work we do everywhere 
already and it's important.

If we do that for inter-plugin depends, then the normal tox jobs should 
test against the most recent release of the other plugin, and people can 
make a -tips style job like the openstackclient-tox-py35-tips job to 
ALSO test that networking-bgpvpn works with tip of networking-odl.

* Relax our rules about git repos in test-requirements.txt

Introduce a whitelist of git repo urls, starting with:

   * https://git.openstack.org/openstack/neutron
   * https://git.openstack.org/openstack/horizon

For the service projects that have plugins that need to test against the
service they're intending to be used with in a real installation. For 
those plugin projects, actually put the git urls into 
test-requirements.txt. This will make the gate work AND local 
development work for the scenarios where the thing that is actually 
needed is always testing against tip of a corresponding service.

* In the zuul jobs, add something similar to tox-siblings but before the 
initial install that will detect a git url that matches a locally 
checked out repo and will swap the local copy instead so that we don't 
have tox cloning directly in gate jobs.

At this point, horizon and neutron plugin projects should be able to use 
normal tox jobs WITHOUT needing to list anything other than horizon and 
neutron themselves in required-projects, and they can also add 
project-specific -tips jobs that will add intra-plugin depends to their 
required-projects so that they can test both sides of the coin.

Finally, and this is a thing we need broadly for OpenStack and not just 
neutron/horizon plugins:

* Extract the tox-siblings logic into a standalone tool that can be 
installed and used from tox so that it's possible to replicate a -tips 
job locally. I've got this pretty much done and just need to get it 
finished up. As soon as it exists I'll update python-openstackclient's 
tox.ini file to use it - and people can cargo cult from there and/or we 
can work it up into a documented recipe for people.

There is one more scenario or concern, which is that for the horizon 
plugins, without horizon in the requirements.txt file, we can be 
erroneously communicating to a deployer that they can be used standlone 
without horizon. For now I think we're going to have to solve that with 
a documentation note coupled with having the horizon repo link in the 
test requirements ... but it might be worth pondering what we could do 
to make this better. Perhaps for horizon because of that use-case we 
really should be modelling horizon as cycle-plus-intermediary and should 
make horizon plugins depend on horizon releases? I'm don't know that I 
know the full ramifications of making that choice - so for now I think 
the above approach (horizon git url in test-requiremnts) plus 
documentation is safer and gives us time to consider whether all the 
horizon plugin projects listing horizon in their requirements.txt is 
better or worse?

Thoughts?

Monty