Open Stack

On 2014-09-12 12:41:42 +0200 (+0200), Ihar Hrachyshka wrote:
[...]
> That said, those failures were solved by pinning the version of the
> library in openstack/requirements and those projects. This is in major
> contrast to how we handled the new testtools release just several
> weeks ago, when the problem was solved by fixing three affected
> projects because of their incorrect usage of tearDown/setUp methods.
[...]

This was of course different because it came during a period where
integrated projects are supposed to be focusing on stabilizing what
they have toward release, but also our behavior was somewhat altered
because we needed to perform some immediate damage control.

One of the side-effects of the failure mode this sqlalchemy-migrate
release induced was that each nova unit tests was generating ~0.5GiB
of log data, instantly overwhelming our test log analysis systems
and flooding our artifact archive (both in terms of bandwidth and
disk). The fastest way to stop this was to "roll back" what changed,
for which the options were either to introduce an exclusionary
version pin or convince the library authors to release an even newer
version tagged to the old one. We chose the first solution as it was
more directly under the control of the infrastructure and nova core
teams involved at that moment.
-- 
Jeremy Stanley