From doug at doughellmann.com Thu Nov 1 12:52:05 2018 From: doug at doughellmann.com (Doug Hellmann) Date: Thu, 01 Nov 2018 08:52:05 -0400 Subject: [Release-job-failures] [release][infra] Tag of openstack/keystone failed In-Reply-To: References: Message-ID: zuul at openstack.org writes: > Build failed. > > - publish-openstack-releasenotes-python3 http://logs.openstack.org/86/86022835fb39cb318e28994ed0290caddfb451be/tag/publish-openstack-releasenotes-python3/7347218/ : POST_FAILURE in 13m 53s > - publish-openstack-releasenotes http://logs.openstack.org/86/86022835fb39cb318e28994ed0290caddfb451be/tag/publish-openstack-releasenotes/a4c2e21/ : SUCCESS in 13m 54s > > _______________________________________________ > Release-job-failures mailing list > Release-job-failures at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures Keystone is configured in stable/queens with the release-notes-jobs project template and in master with the release-notes-jobs-python3 template. This is, AFAICT, exactly what we said should be done. However, both of the templates include jobs in the "tag" pipeline, and tags are not branch-aware. That means we end up running both versions of the release notes publishing jobs when we tag a release, and the results may be unpredictable. This problem will affect other projects as well. I think we have a few options. 1. Move the job settings out of the source repository into project-config, like we do for the release jobs, so that we always run the same version in all cases. This has two downsides. First, we would have to support the python3 variant even on very old stable branches. That shouldn't be a very big concern, though, because that job does not install the source for the project or its dependencies. Second, we would have to modify all of the zuul configurations for all of our old branches, again. As much as I'm enjoying the jokes about how I'm padding my contribution stats this cycle, I don't really want to do that. 2. Make the job itself smart enough to figure out whether to run with python 2 or 3. We could update both jobs to look at some setting in the repo to decide which version to use. This feels excessively complicated, might still result in having to modify some settings in the stable branches, and would mean we would have to customize the shared versions of the jobs defined in the zuul-jobs repo. 3. Modify the python 2 version of the project template ("release-notes-jobs") to remove the "tag" pipeline settings. This would let us continue to use the python 2 variant for tests and when patches merge, and only use the python 3 job for tags. As with option 1, we would have to be sure that the release notes job works under python 3 for all repos, but as described above that isn't a big concern. I think we should take option 3, and will be preparing a patch to do that shortly. Did I miss any options or issues with this approach? Doug From doug at doughellmann.com Thu Nov 1 12:52:54 2018 From: doug at doughellmann.com (Doug Hellmann) Date: Thu, 01 Nov 2018 08:52:54 -0400 Subject: [Release-job-failures] [release][infra] Tag of openstack/keystone failed In-Reply-To: References: Message-ID: Doug Hellmann writes: > zuul at openstack.org writes: > >> Build failed. >> >> - publish-openstack-releasenotes-python3 http://logs.openstack.org/86/86022835fb39cb318e28994ed0290caddfb451be/tag/publish-openstack-releasenotes-python3/7347218/ : POST_FAILURE in 13m 53s >> - publish-openstack-releasenotes http://logs.openstack.org/86/86022835fb39cb318e28994ed0290caddfb451be/tag/publish-openstack-releasenotes/a4c2e21/ : SUCCESS in 13m 54s >> >> _______________________________________________ >> Release-job-failures mailing list >> Release-job-failures at lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures > > Keystone is configured in stable/queens with the release-notes-jobs > project template and in master with the release-notes-jobs-python3 > template. This is, AFAICT, exactly what we said should be done. However, > both of the templates include jobs in the "tag" pipeline, and tags are > not branch-aware. That means we end up running both versions of the > release notes publishing jobs when we tag a release, and the results may > be unpredictable. This problem will affect other projects as well. > > I think we have a few options. > > 1. Move the job settings out of the source repository into > project-config, like we do for the release jobs, so that we always > run the same version in all cases. > > This has two downsides. > > First, we would have to support the python3 variant even on very old > stable branches. That shouldn't be a very big concern, though, > because that job does not install the source for the project or its > dependencies. > > Second, we would have to modify all of the zuul configurations for > all of our old branches, again. As much as I'm enjoying the jokes > about how I'm padding my contribution stats this cycle, I don't > really want to do that. > > 2. Make the job itself smart enough to figure out whether to run with > python 2 or 3. > > We could update both jobs to look at some setting in the repo to > decide which version to use. This feels excessively complicated, > might still result in having to modify some settings in the stable > branches, and would mean we would have to customize the shared > versions of the jobs defined in the zuul-jobs repo. > > 3. Modify the python 2 version of the project template > ("release-notes-jobs") to remove the "tag" pipeline settings. > > This would let us continue to use the python 2 variant for tests and > when patches merge, and only use the python 3 job for tags. As with > option 1, we would have to be sure that the release notes job works > under python 3 for all repos, but as described above that isn't a big > concern. > > I think we should take option 3, and will be preparing a patch to do > that shortly. > > Did I miss any options or issues with this approach? > > Doug See https://review.openstack.org/614758 Doug From zuul at openstack.org Thu Nov 1 13:36:04 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Thu, 01 Nov 2018 13:36:04 +0000 Subject: [Release-job-failures] Release of openstack/python-glanceclient failed Message-ID: Build failed. - release-openstack-python3 http://logs.openstack.org/7f/7ff373d1b8f38666fd61adac798318877e14477d/release/release-openstack-python3/ae53d6b/ : FAILURE in 3m 58s - announce-release announce-release : SKIPPED - propose-update-constraints propose-update-constraints : SKIPPED From zuul at openstack.org Tue Nov 6 22:18:17 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Tue, 06 Nov 2018 22:18:17 +0000 Subject: [Release-job-failures] release-post job for openstack/releases failed Message-ID: Build failed. - tag-releases http://logs.openstack.org/9d/9d2593cee917d4956ffd5b6ea809e0bd872465f6/release-post/tag-releases/3f0d010/ : SUCCESS in 4m 33s - publish-tox-docs-static http://logs.openstack.org/9d/9d2593cee917d4956ffd5b6ea809e0bd872465f6/release-post/publish-tox-docs-static/4b9be16/ : POST_FAILURE in 6m 54s From zuul at openstack.org Mon Nov 19 02:00:00 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 02:00:00 +0000 Subject: [Release-job-failures] Release of openstack/python-jenkins failed Message-ID: Build failed. - release-openstack-python http://logs.openstack.org/9e/9eb41759ee3bbdecc648f3fb5e781b790cf9e634/release/release-openstack-python/4f22beb/ : SUCCESS in 2m 23s - announce-release http://logs.openstack.org/9e/9eb41759ee3bbdecc648f3fb5e781b790cf9e634/release/announce-release/26524ed/ : SUCCESS in 2m 37s - propose-update-constraints http://logs.openstack.org/9e/9eb41759ee3bbdecc648f3fb5e781b790cf9e634/release/propose-update-constraints/6fd3b9b/ : SUCCESS in 1m 32s - trigger-readthedocs-webhook http://logs.openstack.org/9e/9eb41759ee3bbdecc648f3fb5e781b790cf9e634/release/trigger-readthedocs-webhook/72ca0a2/ : FAILURE in 1m 24s From zuul at openstack.org Mon Nov 19 20:52:41 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 20:52:41 +0000 Subject: [Release-job-failures] Tag of openstack/puppet-cloudkitty failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/49/49e9fcb42ac53a29ee2bdf301bc004de19aa99f2/tag/publish-openstack-releasenotes-python3/21fb215/ : RETRY_LIMIT in 2m 35s - publish-openstack-releasenotes http://logs.openstack.org/49/49e9fcb42ac53a29ee2bdf301bc004de19aa99f2/tag/publish-openstack-releasenotes/b71faaf/ : SUCCESS in 3m 13s From zuul at openstack.org Mon Nov 19 20:52:51 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 20:52:51 +0000 Subject: [Release-job-failures] Tag of openstack/puppet-gnocchi failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/00/001dbb012a3063795342c641f346c2af5e7db542/tag/publish-openstack-releasenotes-python3/44559b5/ : RETRY_LIMIT in 3m 02s - publish-openstack-releasenotes http://logs.openstack.org/00/001dbb012a3063795342c641f346c2af5e7db542/tag/publish-openstack-releasenotes/4f3bef4/ : SUCCESS in 3m 11s From zuul at openstack.org Mon Nov 19 20:53:57 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 20:53:57 +0000 Subject: [Release-job-failures] Tag of openstack/puppet-ec2api failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/c5/c557e31356b3391c8879df17909cfaf189d919ad/tag/publish-openstack-releasenotes-python3/5ff334a/ : SUCCESS in 3m 05s - publish-openstack-releasenotes http://logs.openstack.org/c5/c557e31356b3391c8879df17909cfaf189d919ad/tag/publish-openstack-releasenotes/60ef93d/ : RETRY_LIMIT in 2m 37s From zuul at openstack.org Mon Nov 19 20:54:08 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 20:54:08 +0000 Subject: [Release-job-failures] Tag of openstack/puppet-manila failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/a1/a19058f00f7a89c344f4eb218b445c92581bf300/tag/publish-openstack-releasenotes-python3/755e458/ : SUCCESS in 4m 03s - publish-openstack-releasenotes http://logs.openstack.org/a1/a19058f00f7a89c344f4eb218b445c92581bf300/tag/publish-openstack-releasenotes/8df9900/ : RETRY_LIMIT in 2m 34s From zuul at openstack.org Mon Nov 19 20:57:10 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 20:57:10 +0000 Subject: [Release-job-failures] Tag of openstack/puppet-sahara failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/bf/bf9230ba9f684e871220fa1e57832d49cd446b57/tag/publish-openstack-releasenotes-python3/ecaf4b5/ : SUCCESS in 3m 10s - publish-openstack-releasenotes http://logs.openstack.org/bf/bf9230ba9f684e871220fa1e57832d49cd446b57/tag/publish-openstack-releasenotes/be2342e/ : RETRY_LIMIT in 3m 06s From zuul at openstack.org Mon Nov 19 20:57:31 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 20:57:31 +0000 Subject: [Release-job-failures] Tag of openstack/puppet-tacker failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/23/238b49b074f22a145e0ad5db98c64fd584eb49eb/tag/publish-openstack-releasenotes-python3/e384f82/ : SUCCESS in 3m 55s - publish-openstack-releasenotes http://logs.openstack.org/23/238b49b074f22a145e0ad5db98c64fd584eb49eb/tag/publish-openstack-releasenotes/23da69a/ : RETRY_LIMIT in 3m 31s From zuul at openstack.org Mon Nov 19 21:01:29 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 21:01:29 +0000 Subject: [Release-job-failures] Tag of openstack/puppet-zaqar failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/dc/dc9fe05ddb4b0b5b6fcabfc3f47fc3c8ca0f66e3/tag/publish-openstack-releasenotes-python3/524ed89/ : RETRY_LIMIT in 3m 31s - publish-openstack-releasenotes http://logs.openstack.org/dc/dc9fe05ddb4b0b5b6fcabfc3f47fc3c8ca0f66e3/tag/publish-openstack-releasenotes/dc9fb54/ : SUCCESS in 3m 51s From zuul at openstack.org Mon Nov 19 21:04:01 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Mon, 19 Nov 2018 21:04:01 +0000 Subject: [Release-job-failures] Tag of openstack/python-openstackclient failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/da/daa46c1396af8780185ed7e92da8c28c98ae52af/tag/publish-openstack-releasenotes-python3/3d4fc19/ : SUCCESS in 5m 09s - publish-openstack-releasenotes http://logs.openstack.org/da/daa46c1396af8780185ed7e92da8c28c98ae52af/tag/publish-openstack-releasenotes/2709f8e/ : RETRY_LIMIT in 1m 51s From zuul at openstack.org Wed Nov 28 11:48:15 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Wed, 28 Nov 2018 11:48:15 +0000 Subject: [Release-job-failures] Release of openstack/os-brick failed Message-ID: Build failed. - release-openstack-python http://logs.openstack.org/a4/a4f415071660a1377b112936d6356bfe855bb0f9/release/release-openstack-python/159b465/ : FAILURE in 15m 36s - announce-release announce-release : SKIPPED - propose-update-constraints propose-update-constraints : SKIPPED From zuul at openstack.org Wed Nov 28 12:34:47 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Wed, 28 Nov 2018 12:34:47 +0000 Subject: [Release-job-failures] Release of openstack/tripleo-image-elements failed Message-ID: Build failed. - release-openstack-python http://logs.openstack.org/6d/6d55000bca42bb0ffa3eb841fc8022af33bbe3d2/release/release-openstack-python/fdba1db/ : FAILURE in 6m 16s - announce-release announce-release : SKIPPED - propose-update-constraints propose-update-constraints : SKIPPED From zuul at openstack.org Wed Nov 28 12:37:29 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Wed, 28 Nov 2018 12:37:29 +0000 Subject: [Release-job-failures] Release of openstack/tripleo-common failed Message-ID: Build failed. - release-openstack-python http://logs.openstack.org/ff/ff34e6541baa539de36c80c6f0ce8d687129644d/release/release-openstack-python/0658165/ : POST_FAILURE in 13m 01s - announce-release announce-release : SKIPPED - propose-update-constraints propose-update-constraints : SKIPPED From zuul at openstack.org Thu Nov 29 22:17:43 2018 From: zuul at openstack.org (zuul at openstack.org) Date: Thu, 29 Nov 2018 22:17:43 +0000 Subject: [Release-job-failures] Tag of openstack/openstacksdk failed Message-ID: Build failed. - publish-openstack-releasenotes-python3 http://logs.openstack.org/8f/8f15441e72b3aae31013f564ae6e4d55ba217e42/tag/publish-openstack-releasenotes-python3/12bb0c9/ : POST_FAILURE in 4m 32s - publish-openstack-releasenotes http://logs.openstack.org/8f/8f15441e72b3aae31013f564ae6e4d55ba217e42/tag/publish-openstack-releasenotes/ad96f51/ : SUCCESS in 4m 44s