[openstack-dev] Multi-node testing (for redis and others...)
harlowja at outlook.com
Mon Nov 17 21:06:26 UTC 2014
A recent question came up about how do we test better with redis for
tooz. I think this question is also relevant for ceilometer (and other
users of redis) and in general applies to the whole of openstack as the
larger system is what people run (I hope not everyone just runs devstack
on a single-node and that's where they stop, ha).
The basic question is that redis (or zookeeper) have (and typically are)
ways to be setup with multi-node instances (for example with redis +
sentinel or zookeeper in multi-node configurations, or the newly
released redis clustering...). It seems though that our testing
infrastructure is setup to do the basics of tests (which isn't bad, but
does have its limits), and this got me thinking on what would be needed
to actually test these multi-node configurations of things like redis
(configured in sentinel mode, or redis in clustering mode) in a
realistic manner that tests 'common' failure patterns (net splits for
I guess we can split it up into 3 or 4 (or more questions).
1. How do we get a multi-node configuration (of say redis) setup in the
first place, configured so that all nodes are running and sentinel (for
example) is running as expected?
2. How do we then inject failures into this setup to ensure that the
applications and clients built ontop of those systems reliably handle
these type of injected failures (something like
https://github.com/aphyr/jepsen or similar?).
3. How do we analyze those results (for when #2 doesn't turn out to work
as expected) in a meaningful manner, so that we can then turn those
experiments into more reliable software?
Anyone else have any interesting ideas for this?
More information about the OpenStack-dev