Open Stack

On Mon, Dec 5, 2016, at 04:15 PM, Eric K wrote:
> Hi all,
> 
> Does anyone know what could be causing these subunit.parser failures in
> some pythonXX check jobs (they do not show up on my local setup). I
> vaguely remember seeing them before, but can't remember what they were.
> And haven't been able to find relevant info online or in lists. Thanks so
> much!
> 
> Example:
> 
> FAIL: subunit.parser
> tags: worker-0
> Binary content:
> Packet data (application/octet-stream)
> Parser Error: {{{Bad checksum - calculated (0xc62b176a), stored
> (0x494e464f)}}}
> 
> http://logs.openstack.org/29/254429/8/check/gate-congress-python27-ubuntu-x
> enial/b3dacaf/testr_results.html.gz

If you grab the subunit file (its in the logs dir for that job run) you
can inspect it directly. I find version 2 of subunit to be a bit harder
for humans to read so you can also convert it to v1 using subunit-2to1
(included with a python-subunit installation). Digging around that a bit
it appears that the logs are getting truncated:

Datasource driver raised exception
Traceback (most recent call last):
  File "congress/datasources/datasource_driver.py", line 1384, in poll
    self.update_from_datasource()  # sets self.state
  File "congress/datasources/datasource_driver.py", line 1371, in
  update_from_datasource
    self.update_methods[registered_table]()
  File "congress/datasources/nova_driver.py", line 215, in <lambda>
    self.nova_client.floating_ips.list())
  File
  "/home/jenkins/workspace/gate-congress-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/novaclient/api_versions.py",
  line 443, in wrapped
    re2016-12-05 20:49:40.921 2394 INFO
    congress.datasources.datasource_driver [-] nova:: polling
2016-12-05 20:49:40.922 2394 ERRO0^M
Content-Type: text/plain;charset=utf8
Parser Error 
3B^M
Bad checksum - calculated (0x8ecaea4f), stored (0x4552524f)0^M
]

Notice that ERRO0 ends that traceback. Of course now we must wonder if
that is truncated due to the checksum error or if the checksum error is
due to truncation. Either way I would look to see if something is
closing stdout inappropriately. ps on my local machine says you have a
subprocess congress_server running during at least some of these test
cases, perhaps it is interfering?

As for reproducing locally are you attempting to do so on Ubuntu Xenial
(so that python and other libs are the same version)? Also be sure to
use the same version of the constraints file to make sure all the python
deps are installed at the same version.

Other debugging things, you could run the specific failing test cases
with subunit directly and avoid the testr indirection layer:
`.tox/py27/bin/python -m subunit.run module.path.to.test` and
s/subunit/testtools/ if you just want to see it without the subunit data
stream.

Hope this helps,
Clark