Open Stack

Hi guys,

A recent question came up about how do we test better with redis for 
tooz. I think this question is also relevant for ceilometer (and other 
users of redis) and in general applies to the whole of openstack as the 
larger system is what people run (I hope not everyone just runs devstack 
on a single-node and that's where they stop, ha).

The basic question is that redis (or zookeeper) have (and typically are) 
ways to be setup with multi-node instances (for example with redis + 
sentinel or zookeeper in multi-node configurations, or the newly 
released redis clustering...). It seems though that our testing 
infrastructure is setup to do the basics of tests (which isn't bad, but 
does have its limits), and this got me thinking on what would be needed 
to actually test these multi-node configurations of things like redis 
(configured in sentinel mode, or redis in clustering mode) in a 
realistic manner that tests 'common' failure patterns (net splits for 
example).

I guess we can split it up into 3 or 4 (or more questions).

1. How do we get a multi-node configuration (of say redis) setup in the 
first place, configured so that all nodes are running and sentinel (for 
example) is running as expected?
2. How do we then inject failures into this setup to ensure that the 
applications and clients built ontop of those systems reliably handle 
these type of injected failures (something like 
https://github.com/aphyr/jepsen or similar?).
3. How do we analyze those results (for when #2 doesn't turn out to work 
as expected) in a meaningful manner, so that we can then turn those 
experiments into more reliable software?

Anyone else have any interesting ideas for this?

-Josh