<div dir="ltr">Davanum, <div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">1. Suppose a project say Nova wants to enable this Rally Integration<br></span><span style="font-size:12.8000001907349px">for its functional tests, what does that project have to do? (other<br></span><span style="font-size:12.8000001907349px">than the existing well defined tox targets)</span></blockquote><div><br></div><div>Actually project by it self shouldn't do anything. </div><div><br></div><div>The whole work regarding to integration belongs to Infra and Rally code. </div><div>In Infra we should add job that runs few commands like: </div><div><br></div><div> rally deployment use devstack # this is already predefined in rally devstack plugin</div><div> rally verify nova start # ...</div><div> rally verify nova results --html /files-that-will-be-plubished/result.html</div><div> rally verify nova results --json /files-that-will-be-plubished/result.json<br></div><div> rally verify nova check_status # return 1 if some of tests failed</div><div><br></div><div>In Rally we should add auto generation on nova .conf file, based on OpenStack credentials.</div><div><br></div><div>The whole idea is to make common, short and simple set of command that is easy to integrate in gates and use locally.<br></div><div><br></div><div>2 more things:</div><div> 1) We are going to support running specific releases of Rally in gates (so projects won't depend on Rally master) </div><div> 2) Rally has own DB where it stores full results of rally verify runs, if we share this DB between different runs, we can collect all results of all functional test runs in one place, and in future analyze (like generating graphs failures_of_test_x_per/week) </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">2. Is there a "test" project with Gabbi based tests that you know of?</span></blockquote><div><br></div><div>As far as I know Ceilometer is first path-finder here: </div></div><div><a href="https://github.com/openstack/ceilometer/tree/master/ceilometer/tests/gabbi">https://github.com/openstack/ceilometer/tree/master/ceilometer/tests/gabbi</a><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">3. What changes if any are needed in Gabbi to make this happen?</span></blockquote><div><br></div><div>More or less no changes. It can be run via testr/nose that is enough. </div><div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 9, 2015 at 5:05 PM, Davanum Srinivas <span dir="ltr"><<a href="mailto:davanum@gmail.com" target="_blank">davanum@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Boris,<br>
<br>
1. Suppose a project say Nova wants to enable this Rally Integration<br>
for its functional tests, what does that project have to do? (other<br>
than the existing well defined tox targets)<br>
2. Is there a "test" project with Gabbi based tests that you know of?<br>
3. What changes if any are needed in Gabbi to make this happen?<br>
<br>
Guessing going forward, we can setup weekly Rally jobs against<br>
different projects so we can compare performance over time etc?<br>
<br>
thanks,<br>
dims<br>
<div><div class="h5"><br>
<br>
On Fri, Mar 6, 2015 at 6:47 PM, Boris Pavlovic <<a href="mailto:boris@pavlovic.me">boris@pavlovic.me</a>> wrote:<br>
> Hi stackers,<br>
><br>
> Intro (Gabbi)<br>
> -------------------------<br>
><br>
> Gabbi is amazing tool that allows you to describe in human readable way what<br>
> API requests to execute and what you are expecting as a result. It<br>
> Simplifies a lot API testing.<br>
><br>
> It's based on unittest so it can be easily run using tox/tester/nose and so<br>
> on..<br>
><br>
><br>
> Intro (Functional in-tree tests)<br>
> ---------------------------------------<br>
><br>
> Keeping all tests in one project like Tempest, that is maintained by one<br>
> team, was not enough scalable approach. To scale things, projects started<br>
> maintaining their own functional tests in their own tree. This resolves<br>
> scale issues and now new features can be merged with functional tests.<br>
><br>
><br>
> The Problem<br>
> -----------------<br>
><br>
> As far as you know there are a lot of OpenStack projects with their own<br>
> functional tests / gabbi tests in tree. It becomes hard for developers,<br>
> devops and operators to work with them. (Like it's hard to install OpenStack<br>
> by hands without DevStack. )<br>
><br>
> Usually, end users are choosing 2 approach:<br>
> 1) Make own tests<br>
> 2) Make scripts that runs somehow all these tests<br>
><br>
><br>
> Small Intro (Rally)<br>
> ------------------------<br>
><br>
> Rally idea is to make tool that simplifies all kinds of testing of multiple<br>
> OpenStack clouds.<br>
> It should be for human and as well simple to integrated in CI/CD process.<br>
><br>
> Rally automates all testing process (managing testing systems / running<br>
> tests / storing results / working with results)<br>
><br>
> At this moment there are 3 major parts:<br>
> *) deployment - manages OpenStack deployments (creates or uses existing)<br>
> *) verify - manages fully tempest (installs/configurtion/running/parsing<br>
> output/storing results/working with results)<br>
> *) task - own rally testing framework that allows you to do all kinds of<br>
> testing functional/load/performance/scale/load/volume and others.<br>
><br>
> I can say that "rally verify" command that automates work with Tempest is<br>
> very popular. More details here:<br>
> <a href="https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/" target="_blank">https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/</a><br>
><br>
><br>
> Proposal to make the life better<br>
> ------------------------------------------<br>
><br>
> Recently Yair Fried and Prasanth Anbalagan proposed a great idea to extend<br>
> "rally verify" command to add ability to run in-tree functional tests in the<br>
> same way as tempest.<br>
><br>
> In other words to have next syntax: "rally verify <project> <command>"<br>
><br>
> Something like this:<br>
><br>
> rally verify swift start # 1. Check is installed swift for active rally<br>
> deployment.<br>
> # IF NO:<br>
> # Downloads from default (our<br>
> specified place) swift<br>
> # Switch to master or specified tag<br>
> # Installs in venv swift<br>
> # Configure swift functional test<br>
> config for active deployment<br>
> # 2. Run swift functional test<br>
> # 3. Parse subunit output and store to<br>
> Rally DB (for future work)<br>
><br>
> rally verify swift list # List all swift<br>
> verification runs<br>
> rally verify swift show UUID # Shows results<br>
> rally verify swift compare UUID1 UUID2 # Compare results of two runs<br>
><br>
><br>
> Why it makes sense?<br>
> ----------------------------<br>
><br>
> 1) Unification of testing process.<br>
><br>
> There is a simple to learn set of command "rally verify <project> <cmd>"<br>
> that works for all projects in the same way. End users like such things=)<br>
><br>
> 2) Simplification of testing process.<br>
><br>
> "rally verify <project> start" - will automate all steps, so you won't need<br>
> to install project manually, and configure functional test, collect and<br>
> somewhere store results.<br>
><br>
> 3) Avoiding duplication of effort<br>
><br>
> We don't need to implement part of "rally verify" functionality in every<br>
> project.<br>
> It is better to implement it in one place with plugin support. Adding new<br>
> project means implementing new plugin (in most case it will be just<br>
> functional test conf generation)<br>
><br>
> 4) Reusing already existing code<br>
><br>
> Most of the code that we need is already implemented in Rally,<br>
> it requires just small refactoring and generalization.<br>
><br>
><br>
> Thoughts?<br>
><br>
><br>
> Best regards,<br>
> Boris Pavlovic<br>
><br>
><br>
</div></div>> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
><br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
--<br>
Davanum Srinivas :: <a href="https://twitter.com/dims" target="_blank">https://twitter.com/dims</a><br>
</font></span></blockquote></div><br></div>