Open Stack

On 05/02/2013 03:38 PM, Mike Wilson wrote:
> I agree, many people aren't going to want to run ZK just to run
> Openstack. Also, I'm not real familiar with ZK, but besides distributed
> locking, what other features are vital workflow management? I heard
> leader election mentioned earlier, but we don't necessarily have to run
> with a leader, 

The problem with queue-based systems is that when you have more than one
consumer you lose ordering. And, because consumer_1 doesn't know about
consumer_N you run into race conditions when there's shared state (as
there is with workflow task locking).

So you have a few choices, you can make all the workers chatty, or you
can mess with locking, or you can run single threaded so you avoid the
race conditions ... like the scheduler does today.

ZK, and others like it, give you a way to have a leader to do the work
and, if it fails, elect a new leader. It's like single threaded with
failover. And it gives you a means to use a locking approach if you
prefer that (better scalability since you have more than one worker at a
time).

It's certainly possible to take the scheduler approach for small
deployments and just run one orchestrator, but as you grow you'll
definitely need a coordinator between the consumers/workers.

There are already some very large OpenStack deployments out there and
imposing a new service that can't scale out of the box would be bad. And
thinking about this now will help you get your abstractions right.


I feel like a simpler design doesn't have a leader but
> rather just generic conductors that process workflows. Even with my
> large scale deployment, I would prefer to not have to incur the cost of
> operating ZK when I have other technologies I am already familiar with
> (DB, memcache, etc). I'll research ZK more so that I'm not so much in
> the dark, but can anyone explain why a leaderless group of conductors is
> bad? I think what I envision is more along the lines of what John
> Garbutt was explaining. Can we poke holes in that before requiring ZK?
> 
> -Mike Wilson
> 
> 
> On Thu, May 2, 2013 at 12:27 PM, Alex Glikson <GLIKSON at il.ibm.com
> <mailto:GLIKSON at il.ibm.com>> wrote:
> 
>     Right.. Maybe this was not a good suggestion. The thought was that
>     in small-scale deployments, requiring ZK might be a significant
>     management overhead. While for large-scale ones it would be more
>     acceptable. So, the question is how to make this work reasonably
>     well on small scale without ZK, and enable flexible scale-up/out by
>     adding ZK.
>     Maybe assume one conductor that would serialize (some of the) tasks
>     and keep enough state in DB for failure recovery if there is no ZK,
>     and do it in a more scalable & resilient manner if ZK is present.
> 
>     Alex
> 
> 
>     Joshua Harlow <harlowja at yahoo-inc.com
>     <mailto:harlowja at yahoo-inc.com>> wrote on 02/05/2013 08:50:29 PM:
> 
>     > From: Joshua Harlow <harlowja at yahoo-inc.com
>     <mailto:harlowja at yahoo-inc.com>>
>     > To: OpenStack Development Mailing List <openstack-
>     > dev at lists.openstack.org <mailto:dev at lists.openstack.org>>, Alex
>     Glikson/Haifa/IBM at IBMIL,
>     > Date: 02/05/2013 08:50 PM
>     > Subject: Re: [openstack-dev] Nova workflow management update
>     >
>     > So this brings up an interesting issue, the reason ZK exists or
>     > partially exists is that something like ZK api's over DB weren't
>     > really possible. Or that’s what I thought :-P
> 
>     >
>     > Or they weren't possible in a accurate and provable accurate manner.
>     > So spending time creating apis that 'sorta' work with DB's but
>     > really don't (since ZK wouldn't exist if it was possible) does seem
>     > sorta awkward. I thought most cloud providers are already using
>     > zookeeper, for these exact same reasons, and deploying it now-adays
>     > is pretty simple…
>     >
>     > I just worry about providing API's that really don't work correctly
>     > with DB's that will cause more bugs (since certain problems just
>     > can't be done with a DB, or at least any of the DBs that I have
>     > used, maybe db2 can, idk) that we will have to say 'oh ya we know
>     > that doesn't work with a DB'. But maybe that is a compromise that we
>     > have to make and is a evolutionary process where the amount of bugs
>     > that will be caused by DB impls will eventually just cause people to
>     > move to the more attractive ZK backend… Its also sorta concerning
>     > that those types of DB like bugs will be harder than heck to trace
>     > down, but that might be a different issue that we can resolve.
>     >
>     > From: Alex Glikson <GLIKSON at il.ibm.com <mailto:GLIKSON at il.ibm.com>>
>     > Reply-To: OpenStack Development Mailing List <openstack-
>     > dev at lists.openstack.org <mailto:dev at lists.openstack.org>>
>     > Date: Thursday, May 2, 2013 7:53 AM
>     > To: OpenStack Development Mailing List
>     <openstack-dev at lists.openstack.org
>     <mailto:openstack-dev at lists.openstack.org>>
>     > Subject: Re: [openstack-dev] Nova workflow management update
>     >
>     > Changbin Liu <changbin.liu at gmail.com
>     <mailto:changbin.liu at gmail.com>> wrote on 02/05/2013 05:32:05 PM:
>     >
>     > > Subject: Re: [openstack-dev] Nova workflow management update
>     > >
>     > > Hi Joshua,  
>     > >
>     > > Just to share some thoughts:
>     > [...]
>     >
>     > Using ZK makes a lot of sense.. The problem is that making ZooKeeper
>     > a mandatory component to support even basic workflow management
>     > might be an issue. So, the approach which seems to make most sense
>     > is to define abstract internal interfaces for the various
>     > capabilities that ZK can provide (distributed locking, leader
>     > election, etc), and then have one or more implementations (one of
>     > which might be based on ZK). This is the approach that has been
>     > taken for the membership service (service group monitoring APIs) --
>     > introducing the flexibility to use ZK backend, but keeping the
>     > default to be DB-backed.
>     >
>     > Regards,
>     > Alex
>     >
>     > P.S. thinking about this.. would it be possible to implement ZK APIs
>     > over a DB? with some limitations, perhaps..
> 
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev at lists.openstack.org
>     <mailto:OpenStack-dev at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>