Open Stack

On 09/05/2014 04:21 PM, Monty Taylor wrote:
> On 09/05/2014 06:32 AM, Sean Dague wrote:
>> While reviewing this zookeeper service group fix in Nova -
>> https://review.openstack.org/#/c/102639/ it was exposed that the
>> zookeeper tests aren't running in infra.
>>
>> The crux of the issue is that zookeeper python modules are C extensions.
>> So you have to either install from packages (which we don't do in unit
>> tests) or install from pip, which means forcing zookeeper dev packages
>> locally. Realistically this is the same issue we end up with for mysql
>> and pg, but given their wider usage we just forced that pain on
>> developers.
>>
>> But it seems like a bad stand off between testing upstream and testing
>> normal path locally.
>>
>> Big picture it would be nice to not require a ton of dev libraries
>> locally for optional components, but still test them upstream. So that
>> in the base case I'm not running zookeeper locally, but if it fails
>> upstream because I broke something in zookeeper, it's easy enough to
>> spin up that dev env that has it.
>>
>> Which feels like we need some decoupling on our requirements vs. tox
>> targets to get there. CC to Monty and Clark as our super awesome tox
>> hackers to help figure out if there is a path forward here that makes
>> sense.
> 
> Funny story - I've come to dislike what we're doing here, so I've been
> slowly working on an idea in this area:
> 
> https://github.com/emonty/dox
> 
> The tl;dr is "it's like tox, except it uses docker instead of
> virtualenv" - which means we can express all of our requirements, not
> just pip ones.
> 
> It's not quite ready yet - although I'd be happy to move it in to
> stackforge or even openstack-dev and get other people hacking on it with
> me until it is. The main problem that needs solving, I think, is how to
> sanely express multiple target environments (like py26,py27) without
> making a stupidly baroque config file. OTOH, tox's insistence of making
> a new virtualenv for each "environment" is heavyweight and has led to
> some pretty crazy hacks across the project. Luckily, docker itself does
> an EXCELLENT job at handling caching and reuse - so I think we can have
> a set of containers that something in infra (waves hands) publishes to
> dockerhub, like:
> 
>   infra/py27
>   infra/py26
> 
> And then have things like nova build on those, like:
> 
>   infra/nova/py27
> 
> Which would have zookeeper as well
> 
> The _really_ fun part, again, if we can figure out how to express it in
> config without reimplementing make accidentally, is that we could start
> to have things like:
> 
>   infra/mysql
>   infra/postgres
>   infra/mongodb
> 
> And have dox files say things like:
> 
>   Nova unittests want a python27 environment, this means we want an
> infra/mysql container, an infra/postgres container and for those to be
> linked to the infra/nova/py27 container where the tests will run.
> 
> Since those are all reusable, the speed should be _Excellent_ and we
> should be able to more easily get more things runnable locally without
> full devstack.
> 
> Thoughts? Anybody wanna hack on it with me? I think it could wind up
> being a pretty useful tool for folks outside of OpenStack too if we get
> it right.
> 

I think it is sexy - I don't describe ideas/software as sexy that often
but this one deserves it. I'm interested in helping out.

I'll clone it and give it a try - or at least take a look at it.

Flavio

> Monty
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
@flaper87
Flavio Percoco