Open Stack

Hey all,

I like this plan as a kind of next steps for OpenStack-Infra. I have some
thoughts on how zuul might better improve it's logging story but will post
those on the other thread.

I do, however, share both of Clark's concerns.

At the moment zuul_swift_uploads makes a POST request for each individual
file. I do believe we can group them up to a limit, but that limit is still
small and complicated by things such as the total size of the data (which
is probably why the script does them individually but I don't recall). This
is just to say that we need to test how it will go uploading a lot of files
and the time that it may take.

I know the CDN was complicated with the cloud provider we were using at the
time. However, I'm unsure what the CDN options are these days. Will there
be an API we can use to turn the CDN on per container and get the public
URL for example?

If the above two items turn out sub-optimal, I'm personally not opposed to
continuing to run our own middleware. We don't necessarily need that to be
in os_loganalyze as the returned URL could be a new middleware. The
middleware can then handle the ARA and possibly even work as our own CDN
choosing the correct container as needed (if we can't get CDN details
otherwise).

Cheers,
Josh

On Tue, Jul 17, 2018 at 8:27 AM, James E. Blair <corvus at inaugust.com> wrote:

> Hi,
>
> As you may know, all of the logs from Zuul builds are currently uploaded
> to a single static fileserver with about 14TB of storage available in
> one large filesystem.  This was easy to set up, but scales poorly, and
> we live in constant fear of filesystem corruption necessitating a
> lengthy outage for repair or loss of data (an event which happens, on
> average, once or twice a year and takes several days to resolve).
>
> Our most promising approaches to solving this involve moving log storage
> to swift.  We (mostly Joshua) have done considerable work in the past
> but kept hitting blockers.  I think the situation has changed enough
> that the issues we hit before won't be a problem now.  I believe we can
> use this work as a foundation to, relatively quickly, move our log
> storage into swift.  Once there, there's a number of possibilities to
> improve the experience around logs and artifacts in Zuul and general.
>
> This email is going to focus mostly on how OpenStack Infra can move our
> current log storage and hosting to swift.  I will follow it up with an
> email to the zuul-discuss list about further work that we can do that's
> more generally applicable to all Zuul users.
>
> This email is the result of a number of previous discussions, especially
> with Monty, and many of the ideas here are his.  It also draws very
> heavily on Joshua's previous work.  Here's the general idea:
>
> Pre-generate any content for which we currently rely on middleware
> running on logs.openstack.org.  Then upload all of that to swift.
> Return a direct link to swift for serving the content.
>
> In more detail:
>
> In addition to using swift as the storage backend, we would also like to
> avoid running a server as an intermediary.  This is one of the obstacles
> we hit last time.  We started to make os-loganalyze (OSLA) a smart proxy
> which could serve files from disk and swift.  It threatened to become
> very complicated and tax the patience of OSLA's reviewers.  OSLA's
> primary author and reviewer isn't really around anymore, so I suspect
> the appetite for major changes to OSLA is even less than it may have
> been in the past (we have merged 2 changes this year so far).
>
> There are three kinds of automatically generated content on logs.o.o:
>
> * Directory indexes
> * OSLA HTMLification of logs
> * ARA
>
> If we pre-generate all of those, we don't need any of the live services
> on logs.o.o.  Joshua's zuul_swift_upload script already generates
> indexes for us.  OSLA can already be used to HTMLify files statically.
> And ARA has a mode to pre-generate its output as well (which we used
> previously until we ran out of inodes.  So today, we basically have what
> we need to pre-generate this data and store it in swift.
>
> Another issue we ran into previously was the transition from filesystem
> storage to swift.  This was because in Zuul v2, we could not dynamically
> change the log reporting URL.  However, in Zuul v3, since the job itself
> reports the final log URL, we can handle the transition by creating new
> roles to perform the swift upload and return the swift URL.  We can
> begin by using these roles in a new base job so that we can verify
> correct operation.  Then, when we're ready, we can switch the default
> base job.  All jobs which upload logs to swift will report the new swift
> URL; the existing logs.o.o URLs will continue to function until they age
> out.
>
> The Zuul dashboard makes finding the location of logs for jobs
> (especially post jobs) simpler.  So we no longer need logs.o.o to find
> the storage location (files or swift) for post jobs -- a user can just
> follow the link from the build history in the dashboard.
>
> Finally, the apache config (and to some degree, OSLA middleware) handles
> compression.  Ultimately, we don't actually care if the files are
> compressed in storage.  That's an implementation detail (which we care
> about now because we operate the storage).  But it's not a user
> requirement.  In fact, what we want is for jobs to produce logs in
> whatever format they want (plain text, journal, etc).  We want to store
> those.  And we want to present them to the user in the original format.
> Right now we compress many of them before we upload them to the log
> server because, lacking a dedicated upload handler on the log server,
> there's no other way to cause them to be stored compressed.
>
> If we're relieved of that burden, then the only thing we really care
> about is transfer efficiency.  We should be able to upload files to
> swift with Content-Encoding: gzip, and, likewise, users should be able
> to download files with Accept-Encoding: gzip.  We should be able to have
> efficient transfer without having to explicitly compress and rename
> files.  Our first usability win.
>
> The latest version of the zuul_swift_upload script uses the swift
> tempurl functionality to upload logs.  This is because it was designed
> to run on untrusted nodes.  A closer analog to our current Zuul v3 log
> upload system would be to run the uploader on the executor, giving it a
> real swift credential.  It can then upload logs to swift in the normal
> manner, rather than via tempurl.  It can also create containers as
> needed -- another consideration from our earlier work.  By default, it
> could avoid creating containers, but we could configure it to create,
> say, containers for each first-level of our sharding scheme.  This could
> be a general feature of the role that allows for per-site customization.
>
> I think that's the approach we should start with, because it will be the
> easiest transition from our current scheme.  However, in the future, we
> can move to having the uploads occur from the test nodes themselves
> (rather than, or in addition to, the executor), by having a two-part
> system.  The first part runs on the executor in a trusted context and
> creates any containers needed, then generates a tempurl, and uses that
> to have the worker nodes upload to the container directly.  I only
> mention this to show that we're not backing ourselves permanently into
> executor-only uploads.  But we shouldn't consider this part of the first
> phase.
>
> We have also discussed using multiple swifts.  It may be easiest to
> start with one, but in a future where we have executor affinity in Zuul,
> we may want to upload to the nearest swift.  In that case, we can modify
> the role to, rather than be configured with a single swift, support
> multiple swifts, and use the executor affinity information to determine
> if there is a swift colocated in the executor's cloud, and if not, use a
> fallback.  This way we can use multiple swifts as they are available,
> but not require them.
>
> To summarize: static generation combined with a new role to upload to
> swift using openstacksdk should allow us to migrate to swift fairly
> quickly.  Once there, we can work on a number of enhancements which I
> will describe in a followup post to zuul-discuss.
>
> -Jim
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20180717/329db6a9/attachment.html>