Open Stack

On Aug 27, 2014, at 10:31 AM, Sean Dague <sean at dague.net> wrote:

> So this change came in with adding glance.store -
> https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a
> bad direction to be headed.
> 
> Here is the problem when it comes to working with code from git, in
> python, that uses namespaces, it's kind of a hack that violates the
> principle of least surprise.
> 
> For instance:
> 
> cd /opt/stack/oslo.vmware
> pip install .
> cd /opt/stack/olso.config
> pip install -e .
> python -m olso.vmware
> /usr/bin/python: No module named olso.vmware
> 
> In python 2.7 (using pip) namespaces are a bolt on because of the way
> importing modules works. And depending on how you install things in a
> namespace will overwrite the base __init__.py for the top level part of
> the namespace in such a way that you can't get access to the submodules.
> 
> It's well known, and every conversation with dstuft that I've had in the
> past was "don't use namespaces”.

I’ve been using namespace packages on and off for 10+ years, and OpenStack is the first project where I’ve encountered any issues. That doesn’t necessarily mean we shouldn’t change, but it’s also not fair to paint them as completely broken. Many projects continue to use them successfully.

> 
> A big reason we see this a lot is due to the fact that devstack does
> 'editable' pip installs for most things, because the point is it's a
> development environment, and you should be able to change code, and see
> if live without having to go through the install step again.
> 
> If people remember the constant issues with oslo.config in unit tests 9
> months ago, this was because of mismatch of editable vs. non editable
> libraries in the system and virtualenvs. This took months to get to a
> consistent workaround.
> 
> The *workaround* that was done is we just gave up on installing olso
> libs in a development friendly way. I don't consider that a solution,
> it's a work around. But it has some big implications for the usefulness
> of the development environment. It also definitely violates the
> principle of least surprise, as changes to oslo.messaging in a devstack
> env don't immediately apply, you have to reinstall olso.messaging to get
> them to take.

We did make that change, but IIRC we also found and fixed some dependency management issues that were causing non-editable versions of libraries to be installed even though there was already another version present on the system. I thought at the time that not installing the oslo libs editable was a “belt and suspenders” change, rather than the final fix.

> 
> If this is just oslo, that's one thing (and still something I think
> should be revisted, because when the maintainer of pip says "don't do
> this" I'm inclined to go by that). But this change aims to start brining
> this pattern into other projects. Realistically I'm quite concerned that
> this will trigger more work arounds and confusion.

The reason we decided to leave oslo packages in the oslo namespace was due to the pain of renaming the existing libs and repackaging them in the distros. It might be possible to build a package that contains both oslo.config and oslo_config, though, so I’ll spend some time investigating that. If we can make that work, the Oslo team can look into applying that change to all of our libs in the next cycle. If not, we’ll need to get some feedback from our packagers about how much pain renaming things is going to cause for upgrades before we make any changes.

> 
> It also means, for instance, that once we are in a namespace we can
> never decide to install some of the namespace from pypi and some of it
> from git editable (because it's a part that's under more interesting
> rapid development).
> 
> So I'd like us to revisit using a namespace for glance, and honestly,
> for other places in OpenStack, because these kinds of violations of the
> principle of least surprise is something that I'd like us to be actively
> minimizing.
> 
> 	-Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev