Open Stack

The usual unit test rules consequences in real life environment is
http://en.wikipedia.org/wiki/Isolated_system.

The easiest way to avoid this, Is not having real environment on the tempest-devstack gate,
just using mock objects.

Benefits of this:
 - We do not have non-deterministic test responses
 - Tests will be able to run in parallel easily
 - ;)


Using computer emulator/simulator:
 - The starting parameters should be the same all time (CMOS data..)
 - The simulated HW interrupts will always happen at the "same time". (Preventing all true random functionality)
 - The host system load will not have effect to the system
 - The simplest model is ~10 times slower than a real VM, 
   the more precise models can be 10^20 times slower or more.
   The simplest model is not enough for low level system code testing in many cases.


The Vertically scaled test cases are the most riskiest way of extending tempest performance,
on devstack-tempest gate.

 - The admins can observer and manipulate more than just basic user.
 - The race conditions issues will be triggered more frequently.
 - Smarter withebox test May not be able to operate properly
 - Figuring out what was the real issue, will be more difficult.

horizontal scalability does not have the above issues.

The additional consequences of multiprocessing:
 - We (will) have test cases which are not able run in parallel by nature.
 - We need to create dedicated gate jobs or periodic jobs for them. 
  (It is an another test case selection topic)


BTW: The stress tests should get a periodic job, as soon as the nova is logging properly.