Open Stack

Excerpts from Monty Taylor's message of 2018-04-25 16:40:47 +0200:
> Hi everybody,
> 
> We've been working on navigating through from an interesting situation 
> over the past few months, but there isn't a top-level overview of what's 
> going on with it. That's my bad - I've been telling AJaeger I was going 
> to send an email out for a while.
> 
> projects with test requirements on git repo urls of other projects
> ------------------------------------------------------------------
> 
> There are a bunch of projects that need, for testing purposes, to depend 
> on other projects. The majority are either neutron or horizon plugins, 
> but conceptually there is nothing neutron or horizon specific about the 
> issue. The problem they're trying to deal with is that they are a plugin 
> to a service and they need to be able to import code from the service 
> they are a plugin to in their unit tests.
> 
> To make things even more complicated, some of the plugins actually 
> duepend on each other for real, not just as a "we need this for testing"
> 
> There is trouble in paradise though - which is that we don't allow git 
> urls in requirements files. To work around this, the projects in 
> question added additional pip install lines to a tox_install.sh script - 
> essentially bypassing the global-requirements process and system 
> completely.
> 
> This went unnoticed in a general sense until we started working through 
> removing the use of zuul-cloner which is not needed any longer in Zuul v3.
> 
> unwinding things
> ----------------
> 
> There are a few different options, but it's important to keep in mind 
> that we ultimately want all of the following:
> 
> * The code works
> * Tests can run properly in CI
> * "Depends-On" works in CI so that you can test changes cross-repo
> * Tests can run properly locally for developers
> * Deployment requirements are accurately communicated to deployers
> 
> The approach so far
> -------------------
> 
> The approach so far has been releasing service projects to PyPI and 
> reworking the projects to depend on those releases.
> 
> This approach takes advantage of the tox-siblings feature in the gate to 
> ensure we're cross-testing master of projects with each other.
> 
> tox-siblings
> -----------
> 
> There is a feature in the Zuul tox jobs we refer to as "tox-siblings" 
> (this is because historically - wow we have historical context for zuul 
> v3 now - it was implemented as a separate role) What it does is ensure 
> that if you are running a tox job and you add additional projects to 
> required-projects in the job config, that the git versions of those 
> projects will be installed into the tox virtualenv - but only for 
> projects that would have been installed by tox otherwise. This way 
> required-projects is both safe to use and has the effect you'd expect.
> 
> tox-siblings is intended to enable ADDITIONALLY cross-testing projects 
> that otherwise have a normal dependency relationship in the gate. People 
> have been adding jobs like cross-something-something or something-tips 
> in an ad-hoc manner for a while - and in many cases the git parts of 
> that were actually somewhat not correct - so this is an attempt to 
> provide the thing people want in those scenarios in a consistent manner. 
> But it always should be helper logic for more complex gate jobs, not as 
> a de-facto part of a project's basic install.
> 
> Current Approach is wrong
> ------------------------
> 
> Unfortunately, as part of trying to unwind the plugins situation, we've 
> walked ourselves into a situation where the gate is the only thing that 
> has the correct installation information for some projects, and that's 
> not good.
> 
>  From a networking plugin approach the "depend on release and use 
> tox-siblings" assumes that 'depend on release of neutron' is or can be 
> the common case with the ability to add a second tox job to check master 
> against master.
> 
> If that's not a real thing, then depending on releases + tox_siblings in 
> the gate is solving the wrong problem.
> 
> Specific Suggestions
> --------------------
> 
> As there are a few different scenarios, I want to suggest we do a few 
> different things.
> 
> * Prefer interface libraries on PyPI that projects depend on
> 
> Like python-openstackclient and osc-lib, this is the *best* approach
> for projects with plugins. Such interface libraries need to be able to 
> do intermediate releases - and those intermediate releases need to not 
> break the released version of the projects. This is the hardest and 
> longest thing to do as well, so it's most likely to be a multi-cycle effort.
> 
> * Treat inter-plugin depends as normal library depends
> 
> If networking-bgpvpn depends on networking-bagpipe and networking-odl, 
> then networking-bagpipe and networking-odl need to be released to PyPI 
> just like any other library in OpenStack. These are real runtime 
> dependencies.
> 
> Yes, this is more coordination work, but it's work we do everywhere 
> already and it's important.
> 
> If we do that for inter-plugin depends, then the normal tox jobs should 
> test against the most recent release of the other plugin, and people can 
> make a -tips style job like the openstackclient-tox-py35-tips job to 
> ALSO test that networking-bgpvpn works with tip of networking-odl.
> 
> * Relax our rules about git repos in test-requirements.txt
> 
> Introduce a whitelist of git repo urls, starting with:
> 
>    * https://git.openstack.org/openstack/neutron
>    * https://git.openstack.org/openstack/horizon
> 
> For the service projects that have plugins that need to test against the
> service they're intending to be used with in a real installation. For 
> those plugin projects, actually put the git urls into 
> test-requirements.txt. This will make the gate work AND local 
> development work for the scenarios where the thing that is actually 
> needed is always testing against tip of a corresponding service.

How will having git URLs in the test-requirements list work with
the constraints system, if we also have server projects like neutron
and horizon in the global-requirements and upper-constraints lists?

I wonder if it would be simpler to put those requirements into a
separate file, which the check-requirements job would ignore and which
could be installed without having the constraints applied.

> 
> * In the zuul jobs, add something similar to tox-siblings but before the 
> initial install that will detect a git url that matches a locally 
> checked out repo and will swap the local copy instead so that we don't 
> have tox cloning directly in gate jobs.
> 
> At this point, horizon and neutron plugin projects should be able to use 
> normal tox jobs WITHOUT needing to list anything other than horizon and 
> neutron themselves in required-projects, and they can also add 
> project-specific -tips jobs that will add intra-plugin depends to their 
> required-projects so that they can test both sides of the coin.
> 
> Finally, and this is a thing we need broadly for OpenStack and not just 
> neutron/horizon plugins:
> 
> * Extract the tox-siblings logic into a standalone tool that can be 
> installed and used from tox so that it's possible to replicate a -tips 
> job locally. I've got this pretty much done and just need to get it 
> finished up. As soon as it exists I'll update python-openstackclient's 
> tox.ini file to use it - and people can cargo cult from there and/or we 
> can work it up into a documented recipe for people.

It really feels like we're (again) doing a lot of work to get around
limitations of tox itself.  Are there any changes in tox itself
that would make this simpler?  Are we sure tox is still the right
tool for us?

> There is one more scenario or concern, which is that for the horizon 
> plugins, without horizon in the requirements.txt file, we can be 
> erroneously communicating to a deployer that they can be used standlone 
> without horizon. For now I think we're going to have to solve that with 

This seems to apply to neutron plugins and neutron, too, right?

Especially now with lower-constraints jobs in place, having plugins
rely on features only available in unreleased versions of service
projects doesn't make a lot of sense. We test that way *between*
services using integration tests that use the REST APIs, but we
also have some pretty strong stability requirements in place for
those APIs.

> a documentation note coupled with having the horizon repo link in the 
> test requirements ... but it might be worth pondering what we could do 
> to make this better. Perhaps for horizon because of that use-case we 
> really should be modelling horizon as cycle-plus-intermediary and should 
> make horizon plugins depend on horizon releases? I'm don't know that I 
> know the full ramifications of making that choice - so for now I think 
> the above approach (horizon git url in test-requiremnts) plus 
> documentation is safer and gives us time to consider whether all the 
> horizon plugin projects listing horizon in their requirements.txt is 
> better or worse?
> 
> Thoughts?

Another alternative is to put the plugins for things that don't
provide a stable API or interface library back into the repo with
the things they depend on so that all of the tests can just run
together in one job. Either way we do it the groups of people working
on the different things are going to need to figure out how to work
together to make their projects compatible, and combining the repos
means less work and complexity in the CI system.

Doug