Zuul log location changing
Hi, We've been working for some time[1] to retire our current static log server in favor of storing Zuul job logs in Swift. We're just about ready to do that. This means that not only do we get to use another great OpenStack project, but we also can stop worrying about fscking our 14TB log partition whenever it gets hit with a comsic ray. The change will happen in two phases, only the first of which should be readily apparent. Phase 1: On Monday August 12, 2019 we will change the URL that Zuul reports back to Gerrit. Instead of being a direct link to the log server, it will be a link to the Zuul Build page for the job. This is part of Zuul's web interface which has been around for a while, but isn't well known since we haven't linked to it yet. The build page shows a summary of information about the build, including output snippets for failed tasks. Next to the "Summary" tab, you'll find the "Logs" tab. This contains an expandable index of all the log files uploaded for the build. If they are text files, they can be rendered in-app with line-number hyperlinks and severity filtering. There are also direct links to the files on the log server. Links to preview sites (e.g., for docs builds) will show up in the "Artifacts" section. We also plan to further enhance the build page with additional features. Here are some links to sample build pages so you can see what it's like: https://zuul.opendev.org/t/openstack/build/a6e13a8098fc4a1fbff43d8f2c27ad29 https://zuul.opendev.org/t/openstack/build/75d1e8d4ffaf477db00520d7bfd77246 This step is necessary because our static log server implements a number of features as WSGI middleware in Apache. We have re-implemented the these on the Zuul build page, so there should be no loss in functionality (in fact, we think this is an improvement). Once in place, we can change the backend storage options without impacting the user interface. Phase 2: Shortly afterwards (depending on how phase 1 goes), we will configure jobs to upload logs to Swift instead of the static log server. At this point, there should be no user-visible change, since the main interface for interacting with logs is now the Zuul build page. However, you may notice that log urls have changed from our static log server to one of six different Swift instances. The Swift instance used to store the logs for any given build is chosen at random from among our providers, and is yet another really cool multi-cloud feature we get by using OpenStack. Thanks to our amazing providers and all of the folks who have helped with this effort over the years[1]. Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC. -Jim [1] Years. So. Many. Years.
Congratulations to everyone involved! I really liked the new build page, pretty cool! Em qua, 7 de ago de 2019 às 02:05, James E. Blair <corvus@inaugust.com> escreveu:
Hi,
We've been working for some time[1] to retire our current static log server in favor of storing Zuul job logs in Swift. We're just about ready to do that.
This means that not only do we get to use another great OpenStack project, but we also can stop worrying about fscking our 14TB log partition whenever it gets hit with a comsic ray.
The change will happen in two phases, only the first of which should be readily apparent.
Phase 1:
On Monday August 12, 2019 we will change the URL that Zuul reports back to Gerrit. Instead of being a direct link to the log server, it will be a link to the Zuul Build page for the job. This is part of Zuul's web interface which has been around for a while, but isn't well known since we haven't linked to it yet.
The build page shows a summary of information about the build, including output snippets for failed tasks. Next to the "Summary" tab, you'll find the "Logs" tab. This contains an expandable index of all the log files uploaded for the build. If they are text files, they can be rendered in-app with line-number hyperlinks and severity filtering. There are also direct links to the files on the log server.
Links to preview sites (e.g., for docs builds) will show up in the "Artifacts" section.
We also plan to further enhance the build page with additional features.
Here are some links to sample build pages so you can see what it's like:
https://zuul.opendev.org/t/openstack/build/a6e13a8098fc4a1fbff43d8f2c27ad29
https://zuul.opendev.org/t/openstack/build/75d1e8d4ffaf477db00520d7bfd77246
This step is necessary because our static log server implements a number of features as WSGI middleware in Apache. We have re-implemented the these on the Zuul build page, so there should be no loss in functionality (in fact, we think this is an improvement). Once in place, we can change the backend storage options without impacting the user interface.
Phase 2:
Shortly afterwards (depending on how phase 1 goes), we will configure jobs to upload logs to Swift instead of the static log server. At this point, there should be no user-visible change, since the main interface for interacting with logs is now the Zuul build page. However, you may notice that log urls have changed from our static log server to one of six different Swift instances.
The Swift instance used to store the logs for any given build is chosen at random from among our providers, and is yet another really cool multi-cloud feature we get by using OpenStack.
Thanks to our amazing providers and all of the folks who have helped with this effort over the years[1].
Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC.
-Jim
[1] Years. So. Many. Years.
-- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory@gmail.com <iurygregory@gmail.com>*
Hi, We have made the switch to begin storing all of the build logs from Zuul in Swift. Each build's logs will be stored in one of 7 randomly chosen Swift regions in Fort Nebula, OVH, Rackspace, and Vexxhost. Thanks to those providers! You'll note that the links in Gerrit to the Zuul jobs now go to a page on the Zuul web app. A lot of the features previously available on the log server are now available there, plus some new ones. If you're looking for a link to a docs preview build, you'll find that on the build page under the "Artifacts" section now. If you're curious about where your logs ended up, you can see the Swift hostname under the "logs_url" row in the summary table. Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC. -Jim
On Thu, 15 Aug 2019 at 18:05, James E. Blair <corvus@inaugust.com> wrote:
Hi,
We have made the switch to begin storing all of the build logs from Zuul in Swift.
Each build's logs will be stored in one of 7 randomly chosen Swift regions in Fort Nebula, OVH, Rackspace, and Vexxhost. Thanks to those providers!
You'll note that the links in Gerrit to the Zuul jobs now go to a page on the Zuul web app. A lot of the features previously available on the log server are now available there, plus some new ones.
If you're looking for a link to a docs preview build, you'll find that on the build page under the "Artifacts" section now.
If you're curious about where your logs ended up, you can see the Swift hostname under the "logs_url" row in the summary table.
Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC.
One minor thing I noticed is that the emails to openstack-stable-maint list no longer reference the branch. It was previously visible in the URL, e.g. - openstack-tox-py27 https://logs.opendev.org/periodic-stable/opendev.org/openstack/networking-mi... : RETRY_LIMIT in 3m 08s However now it is not: openstack-tox-py27 https://zuul.opendev.org/t/openstack/build/464ae8b594cf4dc5b6da532c4ea179a7 : RETRY_LIMIT in 3m 31s I can see the branch if I click through to the linked Zuul build page.
-Jim
On Thu, Aug 15, 2019 at 10:04:53AM -0700, James E. Blair wrote:
Hi,
We have made the switch to begin storing all of the build logs from Zuul in Swift.
Each build's logs will be stored in one of 7 randomly chosen Swift regions in Fort Nebula, OVH, Rackspace, and Vexxhost. Thanks to those providers!
You'll note that the links in Gerrit to the Zuul jobs now go to a page on the Zuul web app. A lot of the features previously available on the log server are now available there, plus some new ones.
If you're looking for a link to a docs preview build, you'll find that on the build page under the "Artifacts" section now.
If you're curious about where your logs ended up, you can see the Swift hostname under the "logs_url" row in the summary table.
Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC.
Where should I go to see the logs for periodic jobs? I assume these have been transferred over, since (for example) the neutron periodic jobs stopped logging their daily runs after 8/15, except for one time on 8/24. Thanks, Nate
On Tue, Aug 27, 2019, at 9:19 AM, Nate Johnston wrote:
On Thu, Aug 15, 2019 at 10:04:53AM -0700, James E. Blair wrote:
Hi,
We have made the switch to begin storing all of the build logs from Zuul in Swift.
Each build's logs will be stored in one of 7 randomly chosen Swift regions in Fort Nebula, OVH, Rackspace, and Vexxhost. Thanks to those providers!
You'll note that the links in Gerrit to the Zuul jobs now go to a page on the Zuul web app. A lot of the features previously available on the log server are now available there, plus some new ones.
If you're looking for a link to a docs preview build, you'll find that on the build page under the "Artifacts" section now.
If you're curious about where your logs ended up, you can see the Swift hostname under the "logs_url" row in the summary table.
Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC.
Where should I go to see the logs for periodic jobs? I assume these have been transferred over, since (for example) the neutron periodic jobs stopped logging their daily runs after 8/15, except for one time on 8/24.
You can use the builds tab in the dashboard to query for previous builds including those of periodic jobs. For example http://zuul.openstack.org/builds?pipeline=periodic-stable&project=openstack%2Fneutron will get you the periodic stable jobs for neutron. Clark
On Tue, Aug 27, 2019 at 09:23:56AM -0700, Clark Boylan wrote:
On Tue, Aug 27, 2019, at 9:19 AM, Nate Johnston wrote:
On Thu, Aug 15, 2019 at 10:04:53AM -0700, James E. Blair wrote:
Hi,
We have made the switch to begin storing all of the build logs from Zuul in Swift.
Each build's logs will be stored in one of 7 randomly chosen Swift regions in Fort Nebula, OVH, Rackspace, and Vexxhost. Thanks to those providers!
You'll note that the links in Gerrit to the Zuul jobs now go to a page on the Zuul web app. A lot of the features previously available on the log server are now available there, plus some new ones.
If you're looking for a link to a docs preview build, you'll find that on the build page under the "Artifacts" section now.
If you're curious about where your logs ended up, you can see the Swift hostname under the "logs_url" row in the summary table.
Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC.
Where should I go to see the logs for periodic jobs? I assume these have been transferred over, since (for example) the neutron periodic jobs stopped logging their daily runs after 8/15, except for one time on 8/24.
You can use the builds tab in the dashboard to query for previous builds including those of periodic jobs. For example http://zuul.openstack.org/builds?pipeline=periodic-stable&project=openstack%2Fneutron will get you the periodic stable jobs for neutron.
Perfect, thanks! Nate
Nate Johnston <nate.johnston@redhat.com> writes:
On Tue, Aug 27, 2019 at 09:23:56AM -0700, Clark Boylan wrote:
On Tue, Aug 27, 2019, at 9:19 AM, Nate Johnston wrote:
On Thu, Aug 15, 2019 at 10:04:53AM -0700, James E. Blair wrote:
Hi,
We have made the switch to begin storing all of the build logs from Zuul in Swift.
Each build's logs will be stored in one of 7 randomly chosen Swift regions in Fort Nebula, OVH, Rackspace, and Vexxhost. Thanks to those providers!
You'll note that the links in Gerrit to the Zuul jobs now go to a page on the Zuul web app. A lot of the features previously available on the log server are now available there, plus some new ones.
If you're looking for a link to a docs preview build, you'll find that on the build page under the "Artifacts" section now.
If you're curious about where your logs ended up, you can see the Swift hostname under the "logs_url" row in the summary table.
Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC.
Where should I go to see the logs for periodic jobs? I assume these have been transferred over, since (for example) the neutron periodic jobs stopped logging their daily runs after 8/15, except for one time on 8/24.
You can use the builds tab in the dashboard to query for previous builds including those of periodic jobs. For example http://zuul.openstack.org/builds?pipeline=periodic-stable&project=openstack%2Fneutron will get you the periodic stable jobs for neutron.
We also just added the buildsets search page, where you can perform a similar search and see entire buildsets -- click the result link to go to a page that shows you all the builds in the buildset: http://zuul.openstack.org/buildsets?project=openstack%2Fneutron&pipeline=periodic -Jim
participants (5)
-
Clark Boylan
-
corvus＠inaugust.com
-
Iury Gregory
-
Mark Goddard
-
Nate Johnston