Open Stack

We talked about this last week at the Oslo meeting [1], but I also promised to send an email for a broader audience.

We have recently had a couple of issues when we released a library where we broke unit tests running in other projects. We test the source tree of the libraries against the applications using the integration test suite, but we do not run the unit tests. This isn’t a new situation — we had similar problems in icehouse, and juno. We discussed setting up gate jobs to run the consuming project's unit tests during Juno, but eventually dropped that idea because of the server requirements needed to actually run all of the required jobs. That means we still have a small risk of breaking things with a release if we don’t have an API test in place for something we change, or if a test suite mocks out an implementation detail of a library instead of mocking the public API.

As part of releasing each library, we have tried to create test APIs and test fixtures that can be used to control the library’s behavior within a unit test suite in a well-known, testable, and supportable way. We need the liaisons to help identify missing fixtures from existing and not-yet-graduated libraries.

There are two main ways application test suites interact with Oslo libraries that we want to address: Using configuration options directly to control library behavior and mocking. Learning more about both will help us understand how the library interacts with the test, and we can then either design a fixture or test API for the library or modify the test (in cases where it is mocking implementation details).

A change of this scale is a long term project, but we need to start gathering data now if we want to start writing new fixtures in the next cycle. Please review your project’s test suite and make notes about how it uses mocks and configuration options, then add the information to the etherpad [2]. We will talk about it again at the Oslo meeting in a few weeks.

Thanks,
Doug

[1] http://eavesdrop.openstack.org/meetings/oslo/2014/oslo.2014-12-08-16.00.html
[2] https://etherpad.openstack.org/p/oslo-mocks-in-project-unit-tests