Hi All,
From 7th December 2018, all of sudden functional-py35 check job is failing on current master. Earlier this job was failing intermittently (2-3 times from 100). I have tried to dig into the issue and find out that it is failing obly for ubuntu and working fine with Fedora, centos etc. Below are some of the findings:
1. Even if I add failure tests [1] in black-list file every time other tests reported as failure. 2. When I compared failed tests results [1] with passed tests results [2] I have found that passing tests shows 'Pass 741 Skip 7' whereas failing tests shows 'Pass 627 Failure 9 Skip 5' (it does not execute all the tests in case of failure) 3. I have compared version of subunit, oslo.service, oslo.log, tox with failed [3] and passed [4] logs and they are same. 4. I came across a bug in cinder [5] which talks about excessive logs might cause this error and I have tried to ignore the deprecation warning in tox.ini by adding 'PYTHONWARNINGS=ignore::DeprecationWarning' but it still shows DeprecationWarning in the logs. 5. I have tried executing tests by setting '--concurrency' to 1 but still the error persists. 6. Whenver subunit.parser error occurs the tests remains 'inprogress' state and sometimes time-outs. I have no clues so far to tackle this issue, kindly provide your inputs for the same. [1] http://logs.openstack.org/89/617889/4/check/openstack-tox-functional-py35/4c... [2] http://logs.openstack.org/89/617889/4/check/openstack-tox-functional-py35/39... [3] http://logs.openstack.org/89/617889/4/check/openstack-tox-functional-py35/4c... [4] http://logs.openstack.org/89/617889/4/check/openstack-tox-functional-py35/39... [5] https://bugs.launchpad.net/cinder/+bug/1728640 Thanks & Best Regards, Abhishek Kekane