Open Stack

So another idea that was talked about on IRC.

Taskflow exposes entrypoints for these storage backends (like your storage
callback/interface idea).

It currently provides 3 such 'default' backends [sqlalchemy, file/dir
based, in-memory <--> mainly for testing].

A 4th one is in progress for icehouse (zookeeper based).

These backends are used to store intermediary 'flow/task' state (allowing
the taskflow engine to resume if an app crashes/stopped while doing stuff).

A couple ideas about this, since its already pluggable.

Split the sqlalchemy backend -> 'taskflow-sa' repo/new package. For those
projects that want to use this backend, they include it (still means
'taskflow-sa' package has a dependency on sqlalchemy of some version).
Another idea is to move the sqlalchemy dependency currently in taskflow
requirements.txt -> taskflow test-requirements.txt and for those projects
that want to use the sqlalchemy backend, they include the sqlalchemy
version themselves in there requirements.txt (taskflow keeps it in
test-requirements.txt so that it can run its unit/integration tests,
making sure the backend still works).

I'm not really sure which is the best, none seem super-great, but
sometimes u just work with what u got :-P

On 1/3/14, 3:32 PM, "Sean Dague" <sean at dague.net> wrote:

>On 01/03/2014 06:14 PM, Joshua Harlow wrote:
>> Since the library model is what most everyone else uses outside of
>> openstack (I assume?) what can we do to get there so that this model
>>works
>> as well?
>>
>> Expanding dependencies recursively seems like it could help? This could
>> then detect transitive dependency issues (and doesn't seem so hard to
>>do).
>
>We actually talked about having pip be able to help us here with a
>--requirements-override piece of function. dstufft seemed positive on
>the concept.
>
>> I like the idea of 'gate on all the things' (that I've heard come up
>> before) but I don't know if its possible? If we hooked into the pypi
>> upload 'stream' it would seem like we could automatically trigger
>> openstack builds when a known dependency (or dependency of a
>> dependency...) is uploaded (maybe?). That would be pretty neat.
> >
>> In general it really seems like having more libraries, not less is ideal
>> (making us solve this issue whether we want to or not really). As
>> libraries can then be used outside of openstack (taskflow is already
>>being
>> used elsewhere also), libraries create well-defined apis and boundaries
>> between programs (...). I know they also create this dependency hell
>> problem (anvil has been hitting these same issues for a while to). But I
>> think we can figure out a solution that fits both worlds, the world of
>> things we can gate on and the world of things we can't (3rd party
>> libraries that aren't under openstack control). Taskflow is in-between
>> those worlds, so it allows us to explore how to make both of those
>>worlds
>> work.
>
>In general I agree.... however, if you get between OpenStack and SQLA
>you've now touched the 3rd rail. Because we have deep experience about
>how bad the compatibility between versions can be, and we can't be
>beholden to another project about our SQLA upgrade timeframe.
>
>So I think that generalities aside, if are a library, and use SQLA, we
>probably need to really think about putting it in the integrated gate.
>
>Because otherwise what we are saying is taskflow is completely dictating
>the SQLA version in the Icehouse release of OpenStack. And that's the
>wrong place for that decision to be.
>
>If taskflow worked with a storage callback mechanism, and got a storage
>interface from the program that was calling it, then things would be
>much different. But because it's going straight to the database and
>managing tables directly, through a known unstable library, that
>OpenStack itself needs some control over, it's definitely a different
>case.
>
>	-Sean
>
>-- 
>Sean Dague
>Samsung Research America
>sean at dague.net / sean.dague at samsung.com
>http://dague.net
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev