<div dir="ltr">Hi Alexander, Vladimir,<div><br></div><div>First of all, thanks for working on testing Ironic! Functional testing is critical to our project, and I'm very happy you're looking into it.</div><div><br>
</div><div><div class="gmail_extra"><div class="gmail_quote">On Tue, Jan 14, 2014 at 6:28 AM, Alexander Gordeev <span dir="ltr"><<a href="mailto:agordeev@mirantis.com" target="_blank">agordeev@mirantis.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>We (agordeev and vkozhukalov) had a discussion about functional testing flow described here <a href="https://etherpad.openstack.org/p/IronicDevstackTesting" target="_blank">https://etherpad.openstack.org/p/IronicDevstackTesting</a>. From our perspective it has some non-critical, but still notable flaws. If
you ask me, it looks a bit strange to create testing networks and VMs
during devstack run using virsh and shell. Maybe more suitable to use
libvirt python API for this purpose and create test environment right
before launching testing scenario (aka setUp stage). We know that
according to tempest development requirements it is not allowed to hit
hypervisors directly ( <a href="http://docs.openstack.org/developer/tempest/overview.html" target="_blank">http://docs.openstack.org/developer/tempest/overview.html</a> ), but using libvirt is not hitting hypervisor directly. We described alternative testing flow at the same document <a href="https://etherpad.openstack.org/p/IronicDevstackTesting" target="_blank">https://etherpad.openstack.org/p/IronicDevstackTesting</a>.<br>
</div></div></blockquote><div><br></div><div>I've read your proposed approach and I think you raise some very valid points, particularly about separating the "start devstack" from "prepare a tempest test run". I agree that the enrollment of VMs (simulated bare metal nodes) is clearly a property of the latter case, and a failure during enrollment is a failure of that test, not of devstack. However, I think the creation of VMs is a bit of a grey area... </div>
<div><br></div><div>Also, you noted that "having <span style="background-color:rgb(227,255,234);color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11.818181991577148px;line-height:15.994318008422852px">baremetal poseurs in devstack is convenient for development." </span>While it may be useful to add some automation to devstack allowing for the creation and enrollment of VMs for Ironic development, I'm much more eager to see good functional testing. Chris and I are working to simplify the use of tripleo-incubator for easily creating an ironic development environment, so let's focus this work on adding functional testing for the gate :)</div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div>
<div><br></div><div><span>There
are some reasons why it is more correct to create VMs inside tempest
itself, not inside devstack. <br></span><ul><li><span>Firstly, we possibly want to have several
functional testing scenarios and every scenario needs to have clean
testing environment, so we can just add setUp and tearDown stages for
creating and destroying testing envs.</span></li></ul></div></div></blockquote><div>Makes a lot of sense to me.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><ul><li><span> </span></li><li><span>Secondly, virsh has some
performance issues if you deal with >30 VMs (it is not our case for
now but who knows). </span></li></ul></div></div></blockquote><div>This is a reason why you want to use python libvirt api instead of virsh CLI, correct? I don't see a problem, but I will defer to the tempest devs on whether that's OK.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><ul><li><span>Besides, as we found out ssh power driver also uses
virsh for VM power management and does it ineffectively looking over all
VMs (virsh dumpxml for every VM and grep MAC). It is possible to
significantly improve testing power management performance if we
substitute ssh power driver with driver which uses libvirt python API
and lookups VM by UUID, not by MAC.</span></li></ul></div></div></blockquote><div>So, this is a slightly separate discussion about the Ironic SSH power driver. I'm aware of the inefficient approach, but it was necessary to provide support for systems that are not supported (well) by libvirt. If the performance becomes a serious problem for testing, let's look at how we can improve the SSH power driver rather than adding another one.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><ul><li><span> </span></li>
<li><span>Third point here is that adding
nodes into <span>ironic</span> is also a part of <span>ironic</span> functionality and it is
supposed to be tested as well as others. Besides, if something goes
wrong during adding nodes into <span>ironic</span>, we'll get testing scenario
failed, not devstack failed. It is what we usually expect from testing
procedure.</span></li></ul></div></div></blockquote><div>Agreed. Another valid point. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><ul><li><span> </span></li><li><span>And finally, using tempest with python libvirt API we can
create widely customizable testing envs, it is not possible if we use
devstack for that purpose.</span></li></ul></div></div></blockquote><div>I'm not sure I agree with this reasoning. Devstack is very customizable...</div><div><br></div><div>As I said, you've raised some good points, and I think putting the setUp/tearDown to create+enroll and delete+undefine VMs in tempest makes sense. I think the network bridge creation still belongs in devstack -- that's part of the basic environment, not variable between tests.</div>
<div><br></div><div>Of course, I'd like to see what the Tempest/QA/Infra folks think. I've added the [QA] tag to this email to get their attention.</div><div><br></div><div>Cheers,</div><div>Devananda</div></div>
</div>
</div></div>