Open Stack

On Fri, Jul 31, 2015 at 03:18:39PM -0700, Clint Byrum wrote:
> Excerpts from Mike Perez's message of 2015-07-31 10:40:04 -0700:
> > On Fri, Jul 31, 2015 at 8:56 AM, Joshua Harlow <harlowja at outlook.com> wrote:
> > > ...random thought here, skip as needed... in all honesty orchestration
> > > solutions like mesos
> > > (http://mesos.apache.org/assets/img/documentation/architecture3.jpg),
> > > map-reduce solutions like hadoop, stream processing systems like apache
> > > storm (...), are already using zookeeper and I'm not saying we should just
> > > use it cause they are, but the likelihood that they just picked it for no
> > > reason are imho slim.
> > 
> > I'd really like to see focus cross project. I don't want Ceilometer to
> > depend on Zoo Keeper, Cinder to depend on etcd, etc. This is not ideal
> > for an operator to have to deploy, learn and maintain each of these
> > solutions.

If we finally implement the DLM with Tooz then we wouldn't really need
to use different DLM solution in each project, they could all be using
the same. It's just a matter of choice for the admin. If admin wants to
use the same tool he will set the same configuration in Ceilometer and
Cinder, and he wants more work and use different tools then he will set
different configurations.

The discussion is not about choosing ZooKeeper over Redis or anything
like that, the discussion is more in the lines of we need/don't need a
DLM for HA A-A.


> > 
> > I think this is difficult when you consider everyone wants options of
> > their preferred DLM. If we went this route, we should pick one.
> > 
> > Regardless, I want to know if we really need a DLM. Does Ceilometer
> > really need a DLM? Does Cinder really need a DLM? Can we just use a
> > hash ring solution where operators don't even have to know or care
> > about deploying a DLM and running multiple instances of Cinder manager
> > just works?
> > 
> 
> So in the Ironic case, if two conductors decide they both own one IPMI
> controller, _chaos_ can ensue. They may, at different times, read that
> the power is up, or down, and issue power control commands that may take
> many seconds, and thus on the next status run of the other command may
> cause the conductor to react by reversing, and they'll just fight over
> the node in a tug-o-war fashion.
> 
> Oh wait, except, thats not true. Instead, they use the database as a
> locking mechanism, and AFAIK, no nodes have been torn limb from limb by
> two conductors thus far.

One thing we must not forget is that we are not talking about using
locks just for mutual exclusion or for the sake of it, we are doing it
to maintain current Cinder behavior intact. Something people keep
forgetting.

Right now if you are using volume A for reading, lets say for cloning
into a new volume, and a request that changes the resource status comes
in for that same volume A, like backup or delete, the API will accept
that request and when the message arrives at the Volume Node it will be
"queued" on the lock, and once that lock is released operation will be
performed.

When locking only using the DB, which I believe is mostly possible with
possible exception of some driver sections of software based storage
systems, you will be probably changing this behavior and no longer
allowing this.  Depending on who you ask this is a big deal (breaking an
implicit contract based on current behavior) or not.

Keeping things backward compatible is a pain.  And I think that's one of
the differences between Ironic and Cinder, at this point we cannot just
decide to change Cinder however we like, we have to take into account
Nova interactions as well as external client interactions.

> 
> But, a DLM would be more efficient, and actually simplify failure
> recovery for Ironic's operators. The database locks suffer from being a
> little too conservative, and sometimes you just have to go into the DB
> and delete a lock after something explodes (this was true 6 months ago,
> it may have better automation sometimes now, I don't know).
> 
> Anyway, I'm all for the simplest possible solution. But, don't make it
> _too_ simple.
> 

I agree, like Einstein said, "everything should be made as simple as
possible, but no simpler".

Cheers,
Gorka.