Hi Eric, This seems the same issue happening on congress-tempest-plugin gate where 'congress-devstack-py35-api-mysql-queens' is failing [1]. python-congressclient was not able to install and openstack client trow error for congress command. The issue is stable branch jobs on congress-tempest-plugin does checkout the master version for all repo instead of what mentioned in override-checkout var. If you see congress's rocky patch, congress is checkout out with rocky version[2] but congress-tempest-plugin patch's rocky job checkout the master version of congress instead of rocky version [3]. That is why your test expectedly fail on congress patch but pass on congress-tempest-plugin. Root cause is that override-checkout var does not work on the legacy job (it is only zuulv3 job var, if I am not wrong), you need to use BRANCH_OVERRIDE for legacy jobs. Myself, amotoki and akhil was trying lot other workarounds to debug the root cause but at the end we just notice that congress jobs are legacy jobs and using override-checkout :). I have submitted the testing patch with BRANCH_OVERRIDE for congress-tempest-plugin queens job[4]. Which seems working fine, I can make those patches more formal for merge. Another thing I was discussing with Akhil that new tests of builins feature need another feature flag (different than congressz3.enabled) as that feature of z3 is in stein onwards only. [1] https://review.openstack.org/#/c/618951/ [2] http://logs.openstack.org/70/629070/4/check/congress-devstack-api-mysql/87474d7/logs/pip2-freeze.txt.gz [3] http://logs.openstack.org/51/618951/3/check/congress-devstack-api-mysql-rocky/23c0214/logs/pip2-freeze.txt.gz [4] https://review.openstack.org/#/q/topic:fix-stable-branch-testing+(status:open+OR+status:merged) -gmann ---- On Fri, 11 Jan 2019 10:40:39 +0900 Eric K <ekcs.openstack at gmail.com> wrote ---- > The congress-tempest-plugin zuul jobs against stable branches appear > to be working incorrectly. Tests that should fail on stable/rocky (and > indeed fails when triggered by congress patch [1]) are passing when > triggered by congress-tempest-plugin patch [2]. > > I'd assume it's some kind of zuul misconfiguration in > congress-tempest-plugin [3], but I've so far failed to figure out > what's wrong. Particularly strange is that the job-output appears to > show it checking out the right thing [4]. > > Any thoughts or suggestions? Thanks so much! > > [1] > https://review.openstack.org/#/c/629070/ > http://logs.openstack.org/70/629070/4/check/congress-devstack-api-mysql/87474d7/logs/testr_results.html.gz > The two failing z3 tests should indeed fail because the feature was > not available in rocky. The tests were introduced because for some > reason they pass in the job triggered by a patch in > congress-tempest-plugin. > > [2] > https://review.openstack.org/#/c/618951/ > http://logs.openstack.org/51/618951/3/check/congress-devstack-api-mysql-rocky/23c0214/logs/testr_results.html.gz > > [3] https://github.com/openstack/congress-tempest-plugin/blob/master/.zuul.yaml#L4 > > [4] http://logs.openstack.org/51/618951/3/check/congress-devstack-api-mysql-rocky/23c0214/job-output.txt.gz#_2019-01-09_05_18_08_183562 > shows congress is checked out to the correct commit at the top of the > stable/rocky branch. > >