<div dir="ltr">Hi Jorge, <div><br></div><div>Reliability testing automation is the thing that OpenStack Performance team [1] is working right now. The whole solution will consist of:</div><div> * Fault injection library os-failures [2] that provides an API to do different kind of faults on different OpenStack clouds. The lib is currently under active development, our primary goal is to support Fuel-based clouds and then DevStack-based.</div><div> * Rally as an engine to run different types of test scenarios. As Boris mentioned the main feature is called "hooks" - it allows to execute arbitrary code at predefined points of scenario. In our case we will have plugin that uses os-failures library.</div><div> * Set of scenarios and reports to them - this will go into OpenStack Performance Docs [3]. </div><div> * Rally plugin for results processing. Basically we are interested in calculating the following metrics:</div><div> * Count errors appeared during scenario execution (e.g. number of failed requests)</div><div> * Performance degradation - compare performance (e.g. operation duration) after the failure against sample data collected before</div><div> * MTTR - how long does it takes for all errors to disappear and how long does it takes for performance to become normal</div><div><br></div><div>If you are interesting in contribution, we have meetings by Tuesday at 15:30 UTC at #openstack-performance IRC channel. </div><div><br></div><div>Thanks,</div><div>Ilya</div><div><br></div><div>[1] <a href="https://wiki.openstack.org/wiki/Performance_Team">https://wiki.openstack.org/wiki/Performance_Team</a></div><div>[2] <a href="https://github.com/openstack/os-failures">https://github.com/openstack/os-failures</a></div><div>[3] <a href="http://docs.openstack.org/developer/performance-docs/">http://docs.openstack.org/developer/performance-docs/</a><br><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">2016-09-01 10:33 GMT+03:00 Boris Pavlovic <span dir="ltr"><<a href="mailto:bpavlovic@mirantis.com" target="_blank">bpavlovic@mirantis.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Jorge, <div><br></div><div>Rally team is working on feature called "Hooks". </div><div>"Hooks" are going to allow to use Rally to run workloads and inject any actions (including using existing Chaos frameworks) </div><div><br></div><div>Here is the patch: <a href="https://review.openstack.org/#/c/352276/" target="_blank">https://review.<wbr>openstack.org/#/c/352276/</a></div><div>Here is merged spec: <a href="https://github.com/openstack/rally/blob/master/doc/specs/in-progress/hook_section.rst" target="_blank">https://github.com/<wbr>openstack/rally/blob/master/<wbr>doc/specs/in-progress/hook_<wbr>section.rst</a></div><div><br></div><div><br></div><div>You are very welcome to join this effort and help Rally team deliver it faster. </div><div><br></div><div>Thanks!</div><div><br></div><div>Best regards, </div><div>Boris Pavlovic </div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Wed, Aug 31, 2016 at 11:55 PM, Jorge Cardoso (Cloud Operations and Analytics, IT R&D Division) <span dir="ltr"><<a href="mailto:Jorge.Cardoso@huawei.com" target="_blank">Jorge.Cardoso@huawei.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="h5">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">Hi all,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">Is there any work being done on Reliability for OpenStack using e.g. fault-injection, Chaos Engineering from Netflix, and Site Reliability Engineering principles from Google?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">I only found this page in the documentation
<a href="http://docs.openstack.org/developer/performance-docs/test_results/reliability/index.html#openstack-reliability-testing" target="_blank">
http://docs.openstack.org/deve<wbr>loper/performance-docs/test_<wbr>results/reliability/index.<wbr>html#openstack-reliability-<wbr>testing</a>.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">I am working on Cloud Reliability and Resilience and I would like to explore this area for OpenStack.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">You can check some of my interests and work at:
<a href="http://jorge-cardoso.github.io/research/" target="_blank">http://jorge-cardoso.github.io<wbr>/research/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">Any interest from you guys?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">Any suggestions on how to proceed?
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">Best Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt">Jorge Cardoso<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:9pt;color:gray"><u></u> <u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br></div></div>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div></div></div>