So I've spent a day or so debugging an occasionally flakey unit test that I see in the py3 test suite for openstacksdk. I'm at the point that I want to check that I actually understand what it's doing before proposing a fix.

Specifically, openstack.tests.unit.test_stats.TestStats.test_servers_no_detail periodically fails to find a statsd value:

self.assert_reported_stat(
'openstack.api.compute.GET.servers.200', value='0', kind='ms'
)

This assertion is effectively saying that a GET to the servers compute API should return a HTTP 200, and take 0 milliseconds to execute.

The unit test is failing because sometimes it doesn't take 0 milliseconds. In my debugging for example, it takes 4 milliseconds in my environment every now and then:

Considering key openstack.api.compute.GET.servers.200, value 4.000000|ms
Value 4.000000, kind ms
Value 0 != 4.000000

Am I understanding this correctly? It seems to me that perhaps we should be asserting the call took less than a threshold (5 milliseconds) instead of just insisting it always be instantaneous?

Michael