[openstack-qa] Use Mock OpenStack instead of real one on the tempest gate

Daryl Walleck daryl.walleck at RACKSPACE.COM
Tue Jun 11 16:47:50 UTC 2013

Tempest is a functional/system level test suite, which is meant to run end to end. If you wanted to run against mocks, why not just improve the existing unit/mocked integration tests per project? It's not that you can't mock these APIs (we've been experimenting with HTTPretty for testing of CloudCafe itself this way with some success), but it goes against the charter of what Tempest itself should be doing.

From: Attila Fazekas [afazekas at redhat.com]
Sent: Tuesday, June 11, 2013 11:40 AM
To: All Things QA.
Subject: [openstack-qa] Use Mock OpenStack instead of real one on the   tempest gate

The usual unit test rules consequences in real life environment is

The easiest way to avoid this, Is not having real environment on the tempest-devstack gate,
just using mock objects.

Benefits of this:
 - We do not have non-deterministic test responses
 - Tests will be able to run in parallel easily
 - ;)

Using computer emulator/simulator:
 - The starting parameters should be the same all time (CMOS data..)
 - The simulated HW interrupts will always happen at the "same time". (Preventing all true random functionality)
 - The host system load will not have effect to the system
 - The simplest model is ~10 times slower than a real VM,
   the more precise models can be 10^20 times slower or more.
   The simplest model is not enough for low level system code testing in many cases.

The Vertically scaled test cases are the most riskiest way of extending tempest performance,
on devstack-tempest gate.

 - The admins can observer and manipulate more than just basic user.
 - The race conditions issues will be triggered more frequently.
 - Smarter withebox test May not be able to operate properly
 - Figuring out what was the real issue, will be more difficult.

horizontal scalability does not have the above issues.

The additional consequences of multiprocessing:
 - We (will) have test cases which are not able run in parallel by nature.
 - We need to create dedicated gate jobs or periodic jobs for them.
  (It is an another test case selection topic)

BTW: The stress tests should get a periodic job, as soon as the nova is logging properly.

openstack-qa mailing list
openstack-qa at lists.openstack.org

More information about the openstack-qa mailing list