Open Stack

On 04/15/2015 09:50 PM, Robert Collins wrote:
> On 16 April 2015 at 11:59, Sean Dague <sean at dague.net> wrote:
> 
>> I think the completeness statement here is as follows:
>>
>> 1. For OpenStack to scale to the small end, we need to be able to
>> overlap services, otherwise you are telling people they basically have
>> to start with a full rack of hardware to get 1 worker.
> 
> I don't see how that follows: you can run N venvs on one machine. Its
> basically a lighter-weight container than containers, just without all
> the process and resource isolation.
> 
>> 2. The Linux Distributors (who are a big part of our community) install
>> everything at a system level. Python has no facility for having multiple
>> versions of libraries installed at the system level, largely because
>> virtualenvs were created (which solve the non system application level
>> problem really well).
> 
> Actually, the statement about Python having no facility isn't true -
> there are eggs and a mechanism to get a specific version of a
> dependency into a process. It's not all that widely used, largely
> because of the actual things thats missing: Python *doesn't* have the
> ability to load multiple versions of a package into one process. So
> once you ask for testtools==1.0.0, that process has only
> testtools-1.0.0 in the singleton sys.modules['testtools'], and the
> import machinery is defined as having the side effect of changing
> global state, so this is sufficiently tricky to tackle noone has, even
> with importlib etc being around now. NB: 'vendoring' does something in
> this space by shifting an import to a different location, but its
> fragile.
> 
>> 3. The alternative of putting a venv inside of every service violates
>> the principle of single location security update.
> 
> So does copying code around, but we've officially adopted that as our
> approach-until-things-are-mature... and in fact we're talking cluster
> software, so there is (except the small scale) absolutely no
> expectation of single-location security updates: we know and expect to
> have to update N==machines locations, making that M>N is a small
> matter of automation, be that docker/lxc/venvs.
> 
> I think the arguments about devstack and ease of hacking, + our
> deployer community specifically requesting it are sufficient. Whether
> they need to request it or not, they have :).
> 
>> Note: there is an aside about why we might *not* want to do this....
>>
>> That being said, if you deploy at a system level your upgrade unit is
>> now 1 full box, instead of 1 service, because you can't do library
>> isolation between old and new. A worker might have neutron, cinder, and
>> nova agents. Only the Nova agents support rolling upgrade (cinder is
>> working hard on this, I don't think neutron has visited this yet). So
>> real rolling upgrade is sacrificed on this alter of install everything
>> at a system level.
> 
> Yup. And most deployment frameworks want to scale by service, not by
> box, which makes genuine containers super interesting....
> 
>>> Concretely, devstack should be doing one pip install run, and in
>>> stable branches that needs to look something like:
>>>
>>> $ pip install -r known-good-list $path_to_nova $path_to_neutron ....
>>
>> Also remember we need to -e install a bunch of directories, not sure if
>> that makes things easier or harder.
> 
> There's a particular bit of ugly in pip where directories have to be
> resolved to packages, but if we use egg fragment names we can tell pip
> the name and avoid that. The thing that that lookup does is cause
> somewhat later binding of some requirements calls - I'm not sure it
> would be a problem, but if it is its fairly straight forward to
> address.
> 
>> So, one of the problems with that is often what we need is buried pretty
>> deep like -
>> https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/functions-libvirt#L39
> 
> Both apt and rpm will perform much faster given a single invocation
> than 10 or 20 little ones. There's a chunk of redundant work in dpkg
> itself for instance that can be avoided by a single call. So we might
> want to do that for that reason alone. pip doesn't currently do much
> global big-O work, so it shouldn't be affected, but once we do start
> considering already-installed-requirements, then it will start to have
> the same issue.
> 
>> If devstack needs to uplift everything into an install the world phase,
>> I think we're basically on the verge of creating OpenStack Package
>> Manager so that we can specify this all declaratively (after processing
>> a bunch of conditional feature logic). Which, is a thing someone could
>> do (not it), but if we really get there we're probably just better off
>> thinking about reviving building debs so that we can have the package
>> manager globally keep track of all the requirements across multiple
>> invocations.
> 
> mmm, I don't see that - to me a package manager has a lot more to do
> with dependencies and distribution- I'd expect an opm to know about
> things like 'nova requires the optional feature X to work with dvr, or
> Y to work with ovn'. And where to get the tarballs or git repos from
> given just 'openstack/nova'. Refactoring devstack into a bunch of pure
> code - calculate the needed binaries and python bits, and a bunch of
> impure code - do the installs, write the configs, start the services -
> seems straight forward, good for maintenance, and a pretty low bar.
> Maybe even migrate some of it to Python at the same time :).

Possibly, the devil is in the details. Also it means it won't play nice
with manual pip installs before / after, which are sometimes needed.

Mostly I find it annoying that pip has no global view, so all tools that
call it have to construct their own global view and only ever call pip
once to get a right answer. It feels extremely fragile. It's also not
clear to me how easy it's going to be to debug.

pip install .... <150 packages> <10 directories> ...

Version conflict A>=1.4, A<1.3.

(For reference in
http://logs.openstack.org/42/173842/1/check/check-tempest-dsvm-full/8b0b195/logs/devstacklog.txt.gz


we install 192 pip packages & 10 directories from pip install -e & 190
deb packages from apt. The pip packages will increase once we remove
some additional python deb packages.

	-Sean

-- 
Sean Dague
http://dague.net