So we are close to being able to start doing this. The current whitelist is here https://github.com/openstack/tempest/blob/master/etc/whitelist.yaml. I have a "find-errors" script that watches for successful builds and pulls out the non-whitelisted errors. For the past few weeks I have been doing the following: 1. Run find-errors 2. File bugs on any new errors 3. Add to whitelist 4. Repeat There are still some very flaky cases. I will do one more iteration of this. Right now this script https://github.com/openstack/tempest/blob/master/tools/check_logs.py dumps non-whitelisted errors to the console log but always returns success. The question now is how long should all jobs run with no new errors showing, before changing check_logs.py to fail if there are any new errors? -David