On Tue, Aug 14, 2018 at 9:34 PM, Ghanshyam Mann <gmann at ghanshyammann.com> wrote: > ---- On Wed, 15 Aug 2018 09:37:18 +0900 Eric K <ekcs.openstack at gmail.com> wrote ---- > > I'm adding jobs [1] to the tempest plugin to run tests against > > congress stable/queens. The job output seems to show stable/queens > > getting checked out [2], but I know the test is *not* run against > > queens because it's using features not available in queens. The > > expected result is for several tests to fail as seen here [3]. All > > hints and tips much appreciated! > > You are doing it in right way by 'override-checkout: stable/queens'. And as log also show, congress is checkout from stable/queens. I tried to check the results but could not get what tests should fail and why. > > If you can give me more idea, i can debug that. > > -gmann Thanks so much gmann! For example, looking at 'congress_tempest_plugin.tests.scenario.congress_datasources.test_vitrage.TestVitrageDriver' here: http://logs.openstack.org/61/591861/1/check/congress-devstack-api-mysql/36bacbe/logs/testr_results.html.gz It shows passing 1 of 1, but that feature is not in the queens branch at all. The expected result can be seen here: http://logs.openstack.org/05/591805/2/check/congress-devstack-api-mysql/7d7b28e/logs/testr_results.html.gz > > > > > [1] https://review.openstack.org/#/c/591861/1 > > [2] http://logs.openstack.org/61/591861/1/check/congress-devstack-api-mysql-queens/f7b5752/job-output.txt.gz#_2018-08-14_22_30_36_899501 > > [3] https://review.openstack.org/#/c/591805/ (the depends-on is > > irrelevant because that patch has been merged) > >