Open Stack

Hey everybody!

I'd like to make a proposal for changing how we do logs/artifacts/docs 
collection based on the last few weeks/months of writing things- and of 
having to explain to people how to structure build and publish jobs over 
the last couple of weeks.

tl;dr - I'd like to change the publication interface to be directories 
on the remote node, not directories on the executor

Rationale
=========

If jobs have to copy files back to the executor as part of the 
publication interface, then the zuul admins can't change the mechanism 
of how artifacts, logs or docs are published without touching a ton of 
potentially in-tree job content.

Doing so should also allow us to stop having a second copy of build 
logic in the artifact release jobs.

Implementation
==============

Define a root 'output' dir on the remote nodes. Different types of 
output can be collected by putting them into subdirectories of that dir 
on the remote nodes and expecting that base jobs will take care of them.

People using jobs defined in zuul-jobs should define a variable 
"zuul_output_dir", either in site-variables or in their base job. Jobs 
in zuul-jobs can and will depend on that variable existing - it will be 
considered part of the base job interface zuul-jobs expects.

Jobs in zuul-jobs will recognize three specific types of job output:
* logs
* artifacts
* docs

Jobs in zuul-jobs will put job outputs into "{{ zuul_output_dir 
}}/logs", "{{ zuul_ouptut_dir }}/artifacts" and "{{ zuul_output_dir 
}}/docs" as appropriate.

A role will be added to zuul-jobs that can be used in base jobs that 
will ensure those directories all exist.

Compression
-----------

Deployers may choose to have their base job compress items in {{ 
zuul_output_dir }} as part of processing them, or may prefer not to 
depending on whether CPU or network is more precious. Jobs in zuul-jobs 
should just move/copy things into {{ zuul_output_dir }} on the node and 
leave compression, transfer and publication as a base-job operation.

Easy Output Copying
-------------------

A role will also be added to zuul-jobs to facilitate easy/declarative 
output copying.

It will take as input a dictionary of files/folders named 
'zuul_copy_output'. The role will copy contents into {{ zuul_output_dir 
}} on the remote node and is intended to be used before output fetching 
in a base job's post-playook.

The input will be a dictionary so that zuul variable merging can be used 
for accumulating.

Keys of the dictionary will be things to copy. Valid values will be the 
type of output to copy:

* logs
* artifacts
* docs
* null   # ansible null, not the string null

null will be used for 'a parent job said to copy this, but this job 
wants to not do so'

The simple content copying role will not be flexible or powerful. People 
wanting more expressive output copying have all of the normal tools for 
moving files around at their disposal. It will obey the following rules:

* All files/folders will be copied if they exist, or ignored if they don't
* Items will be copied as-if 'cp -a' had been used.
* Order of files is undefined
* Conflicts between declared files are an error.

Jobs defined in zuul-jobs should not depend on the {{ zuul_copy_output 
}} variable for content they need copied in place on a remote node. Jobs 
defined in zuul-jobs should instead copy their output to {{ 
zuul_output_dir }} This prevents zuul deployers from being required to 
put the easy output copying role into their base jobs. Jobs defined in 
zuul-jobs may use the role behind the scenes.

Filter Plugin
-------------

Since the pattern of using a dictionary in job variables to take 
advantage of variable merging is bound to come up more than once, we'll 
define a filter plugin in zuul called 'zuul_list_from_value' (or some 
better name) that will return the list of keys that match a given value. 
So that given the following in a job defintion:

vars:
   zuul_copy_output:
     foo/bar.html: logs
     other/logs/here.html: logs
     foo/bar.tgz: artifacts

Corresponding Ansible could do:

- copy:
     src: {{ item }}
     dest: {{ zuul_log_dir }}
   with_items: {{ zuul_copy_output | zuul_list_from_value(logs) }}

For OpenStack Today
===================

We will define zuul_output_dir to be "{{ ansible_user_dir 
}}/zuul-output" in our site-variables.yaml file.

Implement the following in OpenStack's base job:

We will have the base job will include the simple copying role.

Logs
----

Base job post playbook will always grab {{ zuul_output_dir }}/logs from 
nodes, same as today:
* If there are more than one node, grab it into {{ zuul.executor.log_dir 
}}/{{ inventory_hostname }}.
* If only one node, grab into  {{ zuul.executor.log_dir }} directly

This will mean moving things like 'fetch-tox-output' logic into the 
'tox' role itself, so after it's run the appropriate tox logs will be 
copied to {{ zuul_output_dir }}/logs

Artifacts and docs
------------------

Base job will always ...

* Grab {{ zuul_output_dir }}/artifacts and {{ zuul_output_dir }}/docs to 
{{ zuul.executor.work_dir }}/artifacts and {{ zuul.executor.work_dir }}/docs
* Publish docs content to logs.openstack.org/{{ zuul_log_path }}/docs - 
allows for easy setting of success-url to 'docs/html" in base jobs 
regardless of the subdir the job puts docs in to (see docs vs. releasenotes)
* rsync docs content to logs.o.o/{{ zuul_log_path }}/docs after contents 
of {{ zuul_output_dir }}/logs so that explicit docs win if there is overlap
* Sign each artifact
* Copy each artifact to {{ zuul.project.short_name }}.{{ zuul.tag }}.{{ 
suffix }} OR {{ zuul.project.short_name }}.{{ zuul.branch }}.{{ suffix 
}} depending on if {{ zuul.tag exists }} or not
* If {{ zuul.change }} is set, upload artifacts to {{ zuul_log_path 
}}/artifacts - to allow people to iterate on artifact upload jobs in 
check and verify that they work
* If {{ zuul.change }} is not set, upload artifacts to 
tarballs.openstack.org/{{ zuul.project.short_name }, same as today.
Communicate artifact and docs upload paths to child jobs using: 
https://review.openstack.org/#/c/504808/

Artifact Release jobs
=====================

The release jobs become similar to v2 release jobs, but can be much more 
generalized and reusable. They can also mostly be nodeless jobs 
(javascript npm publication is a notable exception - it will need a node)

Artifacts
---------

* Fetch artifacts and signatures from tarballs.openstack.org/{{ 
zuul.project.short_name }} - based on values set by parent job
Verify fetched signature is correct
* Upload artifact to appropriate destination(s) (pypi, npm, maven, etc)

Docs
----

* Rsync docs from logs.openstack.org/{{ zuul_log_path }}/docs - based on 
values set by parent job
* Publish to release-job-specific destination as the current jobs do

This should allow us to make job/project/content specific publication 
jobs without needing those jobs to duplicate content in the build jobs.

It means we can ALSO put jobs like 'publish-to-pypi' in zuul-jobs 
without needing a second copy in openstack-zuul-jobs because openstack 
happens to builds tarballs differently. The pypi publication step is the 
same no matter how the tarball is produced.

     - jobs:
         - build-openstack-sphinx-docs
         - publish-sphinx-docs-to-afs:
             dependencies:
                 - build-openstack-sphinx-docs

Rollout
=======

The steps should be fairly straightforward (and are fine to do post rollout)

* Add variable to site-variables
* Add support to base job for creating directories and then copying 
content back
* Add easy copy helper role
* Start transitioning jobs in zuul-jobs and openstack-zuul-jobs to using 
the new interface - can be done one at a time
* Update usage

If people are ok with the direction generally, we should be able to get 
the enablement pieces in pretty quickly.

Monty