<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<base href="x-msg://1365/">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div>Hi Rohit,</div>
<div><br>
</div>
<div>I'm glad to see so much interest in getting testing done right. So here's my thoughts. As far as the nova client/euca-tools portion, I think we absolutely need a series of tests that validate that these bindings work correctly. As a nice side effect they
do test their respective APIs, which is good. I think duplication of testing between these two bindings and even what I'm envisioning as the "main" test suite is necessary, as we have to verify at least at a high level that they work correctly.</div>
<div><br>
</div>
<div>My thoughts for our core testing is that those would the ones that do not use language bindings. I think this is where the interesting architectural work can be done. Test framework is a very loose term that gets used a lot, but to me a framework includes:</div>
<div><br>
</div>
<div>
<ul class="MailOutline">
<li>The test runner and it's capabilities</li><li>How the test code is structured to assure maintainability/flexibility/ease of code re-use</li><li>Any utilities provided to extend or ease the ability to test</li></ul>
<div><br>
</div>
</div>
<div>I think we all have a lot of good ideas about this, it's just a matter of consolidating that and choosing one direction to go forward with.</div>
<div><br>
</div>
<div>Daryl</div>
<div><br>
</div>
<div>
<div>On Oct 19, 2011, at 9:58 AM, Rohit Karajgi wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; ">
<div lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1" style="page: WordSection1; ">
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hello Stackers,<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I was at the design summit and the sessions that were ‘all about QA’ and had shown my interest in supporting this effort. Sorry I could not be present at the first QA
IRC meeting due to a vacation.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I had a chance to eavesdrop at the meeting log and Nachi-san also shared his account of the outcome with me. Thanks Nachi!<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Just a heads up to put some of my thoughts on ML before today’s meeting.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I had a look at the various (7 and counting??) test frameworks out there to test OpenStack API.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Jay, Gabe and Tim put up a neat wiki (</span><a href="http://wiki.openstack.org/openstack-integration-test-suites" style="color: blue; text-decoration: underline; ">http://wiki.openstack.org/openstack-integration-test-suites</a>)<span class="Apple-converted-space"> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">to
compare many of these.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I looked at<a href="https://github.com/gabrielfalcao/lettuce" style="color: blue; text-decoration: underline; "><span class="Apple-converted-space"> </span>Lettuce</a><span class="Apple-converted-space"> </span>and
felt it was quite effective. It’s incredibly easy to write tests once the wrappers over the application are setup. Easy as in “Given a ttylinux image create a Server” would be how a test scenario would be written in a typical .feature file, (which is basically
a list of test scenarios for a particular feature) in a natural language. It has nose support, and there’s some neat<span class="Apple-converted-space"> </span><a href="http://lettuce.it/index.html" style="color: blue; text-decoration: underline; ">documentation</a><span class="Apple-converted-space"> </span>too.
I was just curious if anyone has already tried out Lettuce with OpenStack? From the ODS, I think the Grid Dynamics guys already have their own implementation. It would be great if one of you guys join the meeting and throw some light on how you’ve got it to
work.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Just for those who may be unaware, Soren’s branch<span class="Apple-converted-space"> </span><a href="https://github.com/openstack/openstack-integration-tests" style="color: blue; text-decoration: underline; ">openstack-integration-tests</a><span class="Apple-converted-space"> </span>is
actually a merge of Kong and Stacktester.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The other point I wanted to have more clarity on was on using both novaclient AND httplib2 to make the API requests. Though <wwkeyboard> did mention issues regarding
spec bug proliferation into the client, how can we best utilize this dual approach and avoid another round of duplicate test cases. Maybe we target novaclient first and then the use httplib2 to fill in gaps? After-all novaclient does call httplib2 internally.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I would like to team up with Gabe and others for the unified test runner task. Please chip me in if you’re doing some division of labor there.<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks!<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Rohit<o:p></o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">(NTT)<o:p></o:p></span></div>
<div>
<div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; ">
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:openstack-bounces+rohit.karajgi=vertex.co.in@lists.launchpad.net" style="color: blue; text-decoration: underline; ">openstack-bounces+rohit.karajgi=vertex.co.in@lists.launchpad.net</a><span class="Apple-converted-space"> </span>[mailto:openstack-bounces+rohit.karajgi=<a href="mailto:vertex.co.in@lists.launchpad.net" style="color: blue; text-decoration: underline; ">vertex.co.in@lists.launchpad.net</a>]<span class="Apple-converted-space"> </span><b>On
Behalf Of<span class="Apple-converted-space"> </span></b>Gabe Westmaas<br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Monday, October 10, 2011 9:22 PM<br>
<b>To:</b><span class="Apple-converted-space"> </span><a href="mailto:openstack@lists.launchpad.net" style="color: blue; text-decoration: underline; ">openstack@lists.launchpad.net</a><br>
<b>Subject:</b><span class="Apple-converted-space"> </span>[Openstack] [QA] openstack-integration-tests<o:p></o:p></span></div>
</div>
</div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<o:p> </o:p></div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">I'd like to try to summarize and propose at least one next step for the content of the openstack-integration-tests git repository. Note that this is only about the actual tests
themselves, and says absolutely nothing about any gating decisions made in other sessions. <o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">First, there was widespread agreement that in order for an integration suite to be run in the openstack jenkins, it should be included in the community github repository.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Second, it was agreed that there is value in having tests in multiple languages, especially in the case where those tests add value beyond the base language. Examples of this
may include testing using another set of bindings, and therefore testing the API. Using a testing framework that just takes a different approach to testing. Invalid examples include implanting the exact same test in another language simply because you don't
like python.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Third, it was agreed that there is value in testing using novaclient as well as httplib2. Similarly that there is value in testing both XML and JSON.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Fourth, for black box tests, any fixture setup that a suite of tests requires should be done via script that is close to but not within that suite – we want tests to be as agnostic
to an implementation of openstack as possible, and anything you cannot do from the the API should not be inside the tests.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Fifth, there are suites of white box tests – we understand there can be value here, but we aren't sure how to approach that in this project, definitely more discussion needed
here. Maybe we have a separate directory for holding white and black box tests?<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Sixth, no matter what else changes, we must maintain the ability to run a subset of tests through a common runner. This can be done via command line or configuration, whichever
makes the most sense. I'd personally lean towards configuration with the ability to override on the command line.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">If you feel I mischaracterized any of the agreements, please feel free to say so.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Next, we want to start moving away from the multiple entry points to write additional tests. That means taking inventory of the tests that are there now, and figuring out what
they are testing, and how we run them, and then working to combine what makes sense, into a directory structure that makes sense. As often as possible, we should make sure the tests can be run in the same way. I started a little wiki to start collecting
information. I think a short description of the general strategy of each suite and then details about the specific tests in that suite would be useful.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><a href="http://wiki.openstack.org/openstack-integration-test-suites" style="color: blue; text-decoration: underline; ">http://wiki.openstack.org/openstack-integration-test-suites</a><o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Hopefully this can make things a little easier to start contributing.<o:p></o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></div>
</div>
<div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Gabe<o:p></o:p></span></div>
</div>
<div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">
<span style="font-size: 10.5pt; font-family: 'Courier New'; color: black; ">This email may include confidential information. If you received it in error, please delete it.</span><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div>
</div>
_______________________________________________<br>
Mailing list:<span class="Apple-converted-space"> </span><a href="https://launchpad.net/~openstack" style="color: blue; text-decoration: underline; ">https://launchpad.net/~openstack</a><br>
Post to :<span class="Apple-converted-space"> </span><a href="mailto:openstack@lists.launchpad.net" style="color: blue; text-decoration: underline; ">openstack@lists.launchpad.net</a><br>
Unsubscribe :<span class="Apple-converted-space"> </span><a href="https://launchpad.net/~openstack" style="color: blue; text-decoration: underline; ">https://launchpad.net/~openstack</a><br>
More help :<span class="Apple-converted-space"> </span><a href="https://help.launchpad.net/ListHelp" style="color: blue; text-decoration: underline; ">https://help.launchpad.net/ListHelp</a></div>
</span></blockquote>
</div>
<br>
<font face="monospace">This email may include confidential information. If you received it in error, please delete it.</font></body>
</html>