Open Stack

Some of the production Oslo libraries are currently being installed
into the "oslo" namespace package (oslo.config, oslo.messaging,
oslo.vmware, oslo.rootwrap, and oslo.version). Over the course of the
last 2 release cycles, we have seen an increase in the number of
developers who end up with broken systems, where an oslo library (most
often oslo.config) cannot be imported. This is usually caused by
having one copy of a library installed normally (via a system package
or via pip) and another version in "development" (a.k.a., "editable")
mode as installed by devstack. The symptom is most often an error
about importing oslo.config, although that is almost never the library
causing the problem.

We have already worked around this issue with the non-production
libraries by installing them into their own packages, without using
the namespace (oslotest, oslosphinx, etc.). We have also changed the
way packages are installed in nova's tox.ini, to force installation of
packages into the virtualenv (since exposing the global site-packages
was a common source of the problem). And very recently, Sean Dague
changed devstack to install the oslo libraries not in editable mode,
so that installing from source should replace any existing installed
version of the same library.

However, the problems seem to persist, and so I think it's time to
revisit our decision to use a namespace package.

After experimenting with non-namespace packages, I wasn't able to
reproduce the same import issues. I did find one case that may cause
us some trouble, though. Installing a package and then installing an
editable version from source leaves both installed and the editable
version appears first in the import path. That might cause surprising
issues if the source is older than the package, which happens when a
devstack system isn't updated regularly and a new library is released.
However, surprise due to having an old version of code should occur
less frequently than, and have less of an impact than, having a
completely broken set of oslo libraries.

We can avoid adding to the problem by putting each new library in its
own package. We still want the Oslo name attached for libraries that
are really only meant to be used by OpenStack projects, and so we need
a naming convention. I'm not entirely happy with the "crammed
together" approach for oslotest and oslosphinx. At one point Dims and
I talked about using a prefix "oslo_" instead of just "oslo", so we
would have "oslo_db", "oslo_i18n", etc. That's also a bit ugly,
though. Opinions?

Given the number of problems we have now (I help about 1 dev per week
unbreak their system), I think we should also consider renaming the
existing libraries to not use the namespace package. That isn't a
trivial change, since it will mean updating every consumer as well as
the packaging done by distros. If we do decide to move them, I will
need someone to help put together a migration plan. Does anyone want
to volunteer to work on that?

Before we make any changes, it would be good to know how bad this
problem still is. Do developers still see issues on clean systems, or
are all of the problems related to updating devstack boxes? Are people
figuring out how to fix or work around the situation on their own? Can
we make devstack more aggressive about deleting oslo libraries before
re-installing them? Are there other changes we can make that would be
less invasive?

Doug