Open Stack

Over the past week Monty and I have been running at how to solve the 
problem that because of the way we install devstack nodes, our loose 
coordination of projects causes some "interesting" issues.

===============
The basic problem
===============

We install projects in the gate in this order (based on linear order in 
devstack):

install_infra
install_oslo
install_keystoneclient
install_glanceclient
install_cinderclient
install_novaclient

(then conditionally if certain services are enabled)
     install_swiftclient
     install_neutronclient
     install_heatclient
     install_keystone
     install_swift
     install_glance
     install_cinder
     install_neutron
     install_neutron_third_party
     install_nova
     install_horizon
     install_ceilometerclient
     install_ceilometer
     install_heat
     install_tempest

At every point we "python setup.py develop" to pull in requirements. 
That means, that if in install_glance we require "jsonschema" we'll 
install the latest, i.e. 2.0. And if at install_tempest we require 
"jsonschema>=1.3.0,!=1.4.0,<2" (which is the global requirements line), 
we will uninstall jsonschema 2.0. And if something, like cinder, dragged 
in something that required jsonschema, and uses entry points, which 
really care about python package versions, it will be built against 2.0, 
which will have been removed, and we'll have 1.3 instead.

Note: that's not a theoretical scenario, that's why no one could merge 
code last weekend.

We had an equivalent wedge with horizon a month ago, which was the 
python-*client uncap push.

Basically.... we're spinning 20 plates all at the same time, which all 
have different review times, +2 teams, and entropy means they tend to 
drift away from each other over time. It took us a month to get 
python-*clients uncapped across all the projects because of the 
intricate ways in which our projects require our other projects in 
interesting ways.

================
Proposed Solution - part #1 - use global requirements in devstack gate
================

The basic idea is simple, in the devstack gate, we're going to forcibly 
update all the project's requirements.txt files to global requirements 
versions, before running setup.py. That means that any project which 
requires a thing will require it at exactly the same version range as 
everyone else. Which ensures that we don't have this uninstall / 
reinstall issue.

We are really close to having this working. Monty and I have been 
running at it all week, and I actually think this requirements review - 
https://review.openstack.org/#/c/39461/ might be the last bit we need 
before we can land the devstack changes for this.

Things this solves:
  * makes global requirements actually mean something, right now the 
sqlalchemy line is known wrong, which prevents nova from working. There 
are probably other issues as well.
  * ensures we really have a set of requirements that work for all projects.

Things this will allow:
  * devstack gating on proposed changes to global-requirements, so 
you'll know if all the projects can work with global-requirements before 
you land them.

Things that we had to change along the way:
  * oslo.config / oslo.messaging now in devstack - in order to let 
projects use prerelease features of those projects, in this model, we 
needed their git trees to be in our environment, like the python* clients.
  * oslo.config in the devstack-gate (landing soon) - we'll now know on 
every oslo.config proposed commit that it works with all the projects in 
the devstack gate. Once oslo.messaging is proposed for use, we'll 
devstack-gate that as well.

Assuming sqlalchemy is the only requirements issue (it's likely that is 
the case), the rest of the gating around this should land today. This 
should just slide in quietly, and not cause any problems, however, there 
is an interesting caveat....

=================
Interesting Caveat
=================

Global requirements will be used by everyone in the devstack-gate and 
the grenade-gate, *however* in unit tests, projects will still be using 
their local requirements. These might not be the same (actually, for 
almost all projects I can tell you these aren't the same today).

We need to think about the resolution here. There are a number of ones I 
can think of, but this is going to clearly require more thought to get 
right. We need to be able to support things like prerelease Oslo, for 
instance. So lets just say this is tabled until phase #1 is fixed.

	-Sean

-- 
Sean Dague
http://dague.net