<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Thanks, Terry. Tempest would be a great
place for stress tests. There has already been some work on
Tempest stress tests and there is currently a blueprint
<a class="moz-txt-link-freetext" href="https://blueprints.launchpad.net/tempest/+spec/stress-tests">https://blueprints.launchpad.net/tempest/+spec/stress-tests</a> and
some code <br>
<a class="moz-txt-link-freetext" href="https://github.com/openstack/tempest/tree/master/tempest/stress">https://github.com/openstack/tempest/tree/master/tempest/stress</a>.
This code is a more general driver that does things similar to
what you have and includes things like checking logs for errors.
There is a case for volumes but more simple than your scenario.
The idea was that more stress scenarios could be provided in that
framework. I suggest you look at that code and the blueprint and
let us know what you think. <br>
<br>
This discussion would probably be better held just on the qa list.
Looking forward to your input.<br>
<br>
-David<br>
<br>
On 07/01/2013 02:06 PM, Gong, Terry H wrote:<br>
</div>
<blockquote
cite="mid:FD3892F73AE1D64781509046036DB3242A28B871@G1W3640.americas.hpqcorp.net"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
h1
{mso-style-priority:9;
mso-style-link:"Heading 1 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:24.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-link:"Heading 1";
font-family:"Times New Roman","serif";
font-weight:bold;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.bugnumber
{mso-style-name:bugnumber;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.yui3-editabletext-text
{mso-style-name:yui3-editable_text-text;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle29
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle30
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1686514559;
mso-list-type:hybrid;
mso-list-template-ids:-1056914040 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:1818110147;
mso-list-type:hybrid;
mso-list-template-ids:-474592432 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal">We have a stress tool (called qaStressTool)
that we want to contribute to the OpenStack community to help
improve the quality of the OpenStack. Initially it was written
to stress the block storage driver, but it appear to touch
several pieces of the OpenStack projects. We would like your
feedback on which project to submit the tool. One suggestion
is to submit qaStressTool into the Tempest project. Is this
the appropriate location? A preliminary copy of qaStressTool
is available from github at <a moz-do-not-send="true"
href="https://github.com/terry7/openstack-stress.git"><b>https://github.com/terry7/openstack-stress.git</b></a><b>
</b>for you to review to help make this determination.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:black">Here is a brief
summary what </span>
qaStressTool <span style="color:black">can do and what issues
were found while running
</span>qaStressTool<span style="color:black">.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">qaStressTool is used to generate <span
style="color:black">a
</span>load<span style="color:#1F497D"> </span><span
style="color:black">that </span>
test<span style="color:#1F497D">s</span> the block storage
driver<span style="color:#1F497D">s</span> and the related
OpenStack software. qaStressTool is written in Python and uses
the Python Cinder Client and Nova Client API’s.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To run qaStressTool, the user specify the
number of threads, the number of servers, and the number of
volumes on the command line along with the IP address of the
controller node. The user name, the tenant name, and the user
password must be configured in the environment variables
before the start of the test.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">qaStressTool will perform the following
during the run:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Create the specified
number of virtual machines (or instances) that will be used by
all the threads.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Create the specified
number of threads that will be generating the load.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Each thread will create
the specified number of volumes.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->After creating the
specified number of volumes, create a snapshot for each of the
volume.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->After creating the
snapshot, the thread will attach each volume to a randomly
selected virtual machine until all volumes are used.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->After attaching all the
volumes, start detaching the volumes from the instances.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->After detaching all the
volumes, delete the snapshot from each volume.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->After deleting all the
snapshots, delete the volumes.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Finally, delete all the
virtual machines.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Display the results of
the test after performing any necessary cleanup.<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Note that each thread
run asynchronously performing the creating volumes, creating
snapshots, attaching volumes to instances, detaching volumes
from instances, deleting snapshots, and deleting volumes.
<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l1 level1 lfo2"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Initially, the test ran
with no confirmation for each action. This has proven to be
too stressful. However, there is an command line option to
turn off confirmation if one want to run in this mode.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have found the following bugs from
running qaStressTool:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo4"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Cinder Bug# 1157506
Snapshot in-use count not decrement after snapshot is deleted<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo4"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Cinder Bug# 1172503
3par driver not synchronize causing duplicate hostname error<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo4"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Nova Bug# 1175366 Fibre
Channel Multipath attach race condition<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo4"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Nova Bug# 1180497 FC
attach code doesn’t discover multipath device<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo4"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Neutron Bug# 1182662
Cannot turn off quota checking<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo4"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Nova Bug# 1192287
Creating server did not fail when exceeded Quantum quota limit<o:p></o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo4"><!--[if !supportLists]--><span
style="font-family:Symbol"><span style="mso-list:Ignore">·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><!--[endif]-->Nova Bug# 1192763
Removing FC device causes exception preventing detachment
completion<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal">We have found other issues like libvirt
errors or dangling LUNs that we need to investigate further to
determine whether we have a problem or not.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Terry<o:p></o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>