Open Stack

Hi,

We've been working for some time[1] to retire our current static log
server in favor of storing Zuul job logs in Swift.  We're just about
ready to do that.

This means that not only do we get to use another great OpenStack
project, but we also can stop worrying about fscking our 14TB log
partition whenever it gets hit with a comsic ray.

The change will happen in two phases, only the first of which should be
readily apparent.

Phase 1:

On Monday August 12, 2019 we will change the URL that Zuul reports back
to Gerrit.  Instead of being a direct link to the log server, it will be
a link to the Zuul Build page for the job.  This is part of Zuul's web
interface which has been around for a while, but isn't well known since
we haven't linked to it yet.

The build page shows a summary of information about the build, including
output snippets for failed tasks.  Next to the "Summary" tab, you'll
find the "Logs" tab.  This contains an expandable index of all the log
files uploaded for the build.  If they are text files, they can be
rendered in-app with line-number hyperlinks and severity filtering.
There are also direct links to the files on the log server.

Links to preview sites (e.g., for docs builds) will show up in the
"Artifacts" section.

We also plan to further enhance the build page with additional features.

Here are some links to sample build pages so you can see what it's like:

  https://zuul.opendev.org/t/openstack/build/a6e13a8098fc4a1fbff43d8f2c27ad29
  https://zuul.opendev.org/t/openstack/build/75d1e8d4ffaf477db00520d7bfd77246

This step is necessary because our static log server implements a number
of features as WSGI middleware in Apache.  We have re-implemented the
these on the Zuul build page, so there should be no loss in
functionality (in fact, we think this is an improvement).  Once in
place, we can change the backend storage options without impacting the
user interface.

Phase 2:

Shortly afterwards (depending on how phase 1 goes), we will configure
jobs to upload logs to Swift instead of the static log server.  At this
point, there should be no user-visible change, since the main interface
for interacting with logs is now the Zuul build page.  However, you may
notice that log urls have changed from our static log server to one of
six different Swift instances.

The Swift instance used to store the logs for any given build is chosen
at random from among our providers, and is yet another really cool
multi-cloud feature we get by using OpenStack.

Thanks to our amazing providers and all of the folks who have helped
with this effort over the years[1].

Please let us know if you have any questions or encounter any issues,
either here, or in #openstack-infra on IRC.

-Jim

[1] Years.  So.  Many.  Years.