Open Stack

I had a debate with my team about how can I be sure the strategy I want to use
to optimize my cluster with Watcher works well. Also, how a developer can guarantee its 
strategy works well?

After a lot of discussions, we found a consensus:

- For the unit tests, a developer should be able to demonstrate that its strategy algorithm is correct,
  by mocking all data required by this algorithm (Cluster data model, metrics, scoring data, ...). A good 
  test coverage is required to be confident about a given strategy. If you implement a deterministic algorithm,
  unit tests should be able to thoroughly check the algorithm outputs.

- For the integration tests (Tempest), a developer/tester should be able to check that the strategy has been
correctly implemented, according to the plugin implementation documentation available within the Watcher
developer guide[0]. One should at least implement a Tempest scenario test that creates a executes a new audit
with this strategy (and related goal). Note that with tempest gate job, there is not a single measurement being
collected, so one should take into account such a case.

Finally, we added a new section in the Watcher documentation to promote each strategy. This piece of documentation
is very important, hence strategy developers should make sure they describe both how the strategy works and
the goal it achieves. A complete documentation, with references, will make end users more confident about
using it.

Feel free to comment my post :)

[0]: http://docs.openstack.org/developer/watcher/


BR,
David