Open Stack

On Mon, Nov 17, 2014 at 1:06 PM, Joshua Harlow <harlowja at outlook.com> wrote:

> Hi guys,
>
> A recent question came up about how do we test better with redis for tooz.
> I think this question is also relevant for ceilometer (and other users of
> redis) and in general applies to the whole of openstack as the larger
> system is what people run (I hope not everyone just runs devstack on a
> single-node and that's where they stop, ha).
>

https://review.openstack.org/#/c/106043/23


>
> The basic question is that redis (or zookeeper) have (and typically are)
> ways to be setup with multi-node instances (for example with redis +
> sentinel or zookeeper in multi-node configurations, or the newly released
> redis clustering...). It seems though that our testing infrastructure is
> setup to do the basics of tests (which isn't bad, but does have its
> limits), and this got me thinking on what would be needed to actually test
> these multi-node configurations of things like redis (configured in
> sentinel mode, or redis in clustering mode) in a realistic manner that
> tests 'common' failure patterns (net splits for example).
>
> I guess we can split it up into 3 or 4 (or more questions).
>
> 1. How do we get a multi-node configuration (of say redis) setup in the
> first place, configured so that all nodes are running and sentinel (for
> example) is running as expected?
> 2. How do we then inject failures into this setup to ensure that the
> applications and clients built ontop of those systems reliably handle these
> type of injected failures (something like https://github.com/aphyr/jepsen
> or similar?).
> 3. How do we analyze those results (for when #2 doesn't turn out to work
> as expected) in a meaningful manner, so that we can then turn those
> experiments into more reliable software?
>
> Anyone else have any interesting ideas for this?
>
> -Josh
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141117/e118f19d/attachment.html>