Open Stack

Robert Collins wrote:
> Most production systems I know don't run with open ended dependencies.
> One of our contributing issues IMO is that we have the requirements
> duplicated everywhere - and then ignore them for many of our test runs
> (we deliberately override the in-tree ones with global requirements).
> Particularly, since the only reason unified requirements matter is for
> distro packages, and they ignore our requirements files *anyway*, I'm
> not sure our current aggregate system is needed in that light.
>
> That said, making requirements be capped and auto adjust upwards would
> be extremely useful IMO, but its a chunk of work;
>   - we need the transitive dependencies listed, not just direct dependencies

Wouldn't a pip install of the requirements.txt from the requirements 
repo itself get this? That would tell pip to download all the things and 
there transitive dependencies (aka step #1).

>   - we need a thing to find possible upgrades and propose bumps

This is an analysis of the $ pip freeze after installing into that 
virtualenv (aka step #2)?

>   - we would need to very very actively propogate those out from global
> requirements

Sounds like an enhanced updater.py that uses the output from step #2?

>
> For now I think making 'react to the situation faster and easier' is a
> good thing to push on.

One question I have is that not all things specify all there 
dependencies, since some of them are pluggable (for example kombu can 
use couchdb, or a transport exists that seems like it could, yet kombu 
doesn't list that dependency in its requirements (it gets listed in 
https://github.com/celery/kombu/blob/master/setup.py#L122 under 
'extra_requires' though); I'm sure other pluggable libraries 
(sqlalchemy, taskflow, tooz...) are similar in this regard so I wonder 
how those kind of libraries would work with this kind of proposal.

>
> -Rob
>
>
>
> On 18 November 2014 12:02, Sean Dague<sean at dague.net>  wrote:
>> As we're dealing with the fact that testtools 1.4.0 apparently broke
>> something with attribute additions to tests (needed by tempest for
>> filtering), it raises an interesting problem.
>>
>> Our current policy on requirements is to leave them open ended, this
>> lets us take upstream fixes. It also breaks us a lot. But our max
>> version of dependencies happens with 0 code review or testing.
>>
>> However, fixing these things takes a bunch of debug, code review, and
>> test time. Seen by the fact that the testtools 1.2.0 block didn't even
>> manage to fully merge this weekend.
>>
>> This is an asymetric break/fix path, which I think we need a better plan
>> for. If fixing is more expensive than breaking, then you'll tend to be
>> in a broken state quite a bit. We really actually want the other
>> asymetry if we can get it.
>>
>> There are a couple of things we could try here:
>>
>> == Cap all requirements, require code reviews to bump maximums ==
>>
>> Benefits, protected from upstream breaks.
>>
>> Down sides, requires active energy to move forward. The SQLA 0.8
>> transition took forever.
>>
>> == Provide Requirements core push authority ==
>>
>> For blocks on bad versions, if we had a fast path to just merge know
>> breaks, we could right ourselves quicker. It would have reasonably
>> strict rules, like could only be used to block individual versions.
>> Probably that should also come with sending email to the dev list any
>> time such a thing happened.
>>
>> Benefits, fast to fix
>>
>> Down sides, bypasses our testing infrastructure. Though realistically
>> the break bypassed it as well.
>>
>> ...
>>
>> There are probably other ways to make this more symetric. I had a grand
>> vision one time of building a system that kind of automated the
>> requirements bump, but have other problems I think need to be addressed
>> in OpenStack.
>>
>>
>> The reason I think it's important to come up with a better way here is
>> that making our whole code gating system lock up for 12+ hrs because of
>> an external dependency that we are pretty sure is the crux of our break
>> becomes very discouraging for developers. They can't get their code
>> merged. They can't get accurate test results. It means that once we get
>> the fix done, everyone is rechecking their code, so now everyone is
>> waiting extra long for valid test results. People don't realize their
>> code can't pass and just keep pushing patches up consuming resources
>> which means that parts of the project that could pass tests, is backed
>> up behind 100% guarunteed failing parts. All in all, not a great system.
>>
>>          -Sean
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>