Open Stack

tl;dr Soon, perhaps next week, tempest gate jobs will start failing if 
there are any ERROR lines in the logs that are not matched by an entry 
in https://github.com/openstack/tempest/blob/master/etc/whitelist.yaml. 
There is an exception for neutron because
more work needs to be done there for this to be feasible.

The whitelist file contains a lot of entries that look more like 
substantial bugs than incorrectly logging an ERROR due to bad
client data. I have been tracking this for a while and new things show 
up frequently. But since the tests pass no one looks at the logs and 
notices the bug indicators. We need to stop these bugs from getting 
merged. I have filed individual bugs for many of the items in the 
whitelist, but by no means all of them. The neutron team is taking on 
the task of getting rid of their errors and there are so many it is too 
much work to keep an up-to-date whitelist for neutron. So for the time 
being, neutron runs will dump all errors to the console but not fail.

In addition to the fact that these log errors indicate bugs, they make 
it more difficult to diagnose a problem when builds actually fail in the 
tempest tests because it can be hard to tell which log errors are 
"known" and which might be causing the failure. Hopefully some priority 
will be given to fixing these bugs and removing entries from the 
whitelist until it is driven to zero.

If any one has any comments or suggestions to improve this process, 
please speak up.

  -David