Open Stack

Hi Alexander, Vladimir,

First of all, thanks for working on testing Ironic! Functional testing is
critical to our project, and I'm very happy you're looking into it.

On Tue, Jan 14, 2014 at 6:28 AM, Alexander Gordeev <agordeev at mirantis.com>wrote:

> We (agordeev and vkozhukalov) had a discussion about functional testing
> flow described here https://etherpad.openstack.org/p/IronicDevstackTesting.
> From our perspective it has some non-critical, but still notable flaws. If
> you ask me, it looks a bit strange to create testing networks and VMs
> during devstack run using virsh and shell. Maybe more suitable to use
> libvirt python API for this purpose and create test environment right
> before launching testing scenario (aka setUp stage). We know that according
> to tempest development requirements it is not allowed to hit hypervisors
> directly ( http://docs.openstack.org/developer/tempest/overview.html ),
> but using libvirt is not hitting hypervisor directly. We described
> alternative testing flow at the same document
> https://etherpad.openstack.org/p/IronicDevstackTesting.
>

I've read your proposed approach and I think you raise some very valid
points, particularly about separating the "start devstack" from "prepare a
tempest test run". I agree that the enrollment of VMs (simulated bare metal
nodes) is clearly a property of the latter case, and a failure during
enrollment is a failure of that test, not of devstack. However, I think the
creation of VMs is a bit of a grey area...

Also, you noted that "having baremetal poseurs in devstack is convenient
for development." While it may be useful to add some automation to devstack
allowing for the creation and enrollment of VMs for Ironic development, I'm
much more eager to see good functional testing. Chris and I are working to
simplify the use of tripleo-incubator for easily creating an ironic
development environment, so let's focus this work on adding functional
testing for the gate :)


>
> There are some reasons why it is more correct to create VMs inside tempest
> itself, not inside devstack.
>
>    - Firstly, we possibly want to have several functional testing
>    scenarios and every scenario needs to have clean testing environment, so we
>    can just add setUp and tearDown stages for creating and destroying testing
>    envs.
>
> Makes a lot of sense to me.

>
>    -
>    - Secondly, virsh has some performance issues if you deal with >30 VMs
>    (it is not our case for now but who knows).
>
> This is a reason why you want to use python libvirt api instead of virsh
CLI, correct? I don't see a problem, but I will defer to the tempest devs
on whether that's OK.


>    - Besides, as we found out ssh power driver also uses virsh for VM
>    power management and does it ineffectively looking over all VMs (virsh
>    dumpxml for every VM and grep MAC). It is possible to significantly improve
>    testing power management performance if we substitute ssh power driver with
>    driver which uses libvirt python API and lookups VM by UUID, not by MAC.
>
> So, this is a slightly separate discussion about the Ironic SSH power
driver. I'm aware of the inefficient approach, but it was necessary to
provide support for systems that are not supported (well) by libvirt. If
the performance becomes a serious problem for testing, let's look at how we
can improve the SSH power driver rather than adding another one.


>
>    -
>    - Third point here is that adding nodes into ironic is also a part of
>    ironic functionality and it is supposed to be tested as well as
>    others. Besides, if something goes wrong during adding nodes into
>    ironic, we'll get testing scenario failed, not devstack failed. It is
>    what we usually expect from testing procedure.
>
> Agreed. Another valid point.

>
>    -
>    - And finally, using tempest with python libvirt API we can create
>    widely customizable testing envs, it is not possible if we use devstack for
>    that purpose.
>
> I'm not sure I agree with this reasoning. Devstack is very customizable...

As I said, you've raised some good points, and I think putting the
setUp/tearDown to create+enroll and delete+undefine VMs in tempest makes
sense. I think the network bridge creation still belongs in devstack --
that's part of the basic environment, not variable between tests.

Of course, I'd like to see what the Tempest/QA/Infra folks think. I've
added the [QA] tag to this email to get their attention.

Cheers,
Devananda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140114/7b188cd9/attachment.html>