Open Stack

On Tue, Oct 10, 2017 at 05:42:12PM -0500, Monty Taylor wrote:
> Hey everybody!
> 
> I'd like to make a proposal for changing how we do logs/artifacts/docs
> collection based on the last few weeks/months of writing things- and of
> having to explain to people how to structure build and publish jobs over the
> last couple of weeks.
> 
> tl;dr - I'd like to change the publication interface to be directories on
> the remote node, not directories on the executor
> 
> Rationale
> =========
> 
> If jobs have to copy files back to the executor as part of the publication
> interface, then the zuul admins can't change the mechanism of how artifacts,
> logs or docs are published without touching a ton of potentially in-tree job
> content.
> 
> Doing so should also allow us to stop having a second copy of build logic in
> the artifact release jobs.
> 
> Implementation
> ==============
> 
> Define a root 'output' dir on the remote nodes. Different types of output
> can be collected by putting them into subdirectories of that dir on the
> remote nodes and expecting that base jobs will take care of them.
> 
> People using jobs defined in zuul-jobs should define a variable
> "zuul_output_dir", either in site-variables or in their base job. Jobs in
> zuul-jobs can and will depend on that variable existing - it will be
> considered part of the base job interface zuul-jobs expects.
> 
> Jobs in zuul-jobs will recognize three specific types of job output:
> * logs
> * artifacts
> * docs
> 
> Jobs in zuul-jobs will put job outputs into "{{ zuul_output_dir }}/logs",
> "{{ zuul_ouptut_dir }}/artifacts" and "{{ zuul_output_dir }}/docs" as
> appropriate.
> 
> A role will be added to zuul-jobs that can be used in base jobs that will
> ensure those directories all exist.
> 
> Compression
> -----------
> 
> Deployers may choose to have their base job compress items in {{
> zuul_output_dir }} as part of processing them, or may prefer not to
> depending on whether CPU or network is more precious. Jobs in zuul-jobs
> should just move/copy things into {{ zuul_output_dir }} on the node and
> leave compression, transfer and publication as a base-job operation.
> 
> Easy Output Copying
> -------------------
> 
> A role will also be added to zuul-jobs to facilitate easy/declarative output
> copying.
> 
> It will take as input a dictionary of files/folders named
> 'zuul_copy_output'. The role will copy contents into {{ zuul_output_dir }}
> on the remote node and is intended to be used before output fetching in a
> base job's post-playook.
> 
> The input will be a dictionary so that zuul variable merging can be used for
> accumulating.
> 
> Keys of the dictionary will be things to copy. Valid values will be the type
> of output to copy:
> 
> * logs
> * artifacts
> * docs
> * null   # ansible null, not the string null
> 
> null will be used for 'a parent job said to copy this, but this job wants to
> not do so'
> 
> The simple content copying role will not be flexible or powerful. People
> wanting more expressive output copying have all of the normal tools for
> moving files around at their disposal. It will obey the following rules:
> 
> * All files/folders will be copied if they exist, or ignored if they don't
> * Items will be copied as-if 'cp -a' had been used.
> * Order of files is undefined
> * Conflicts between declared files are an error.
> 
> Jobs defined in zuul-jobs should not depend on the {{ zuul_copy_output }}
> variable for content they need copied in place on a remote node. Jobs
> defined in zuul-jobs should instead copy their output to {{ zuul_output_dir
> }} This prevents zuul deployers from being required to put the easy output
> copying role into their base jobs. Jobs defined in zuul-jobs may use the
> role behind the scenes.
> 
> Filter Plugin
> -------------
> 
> Since the pattern of using a dictionary in job variables to take advantage
> of variable merging is bound to come up more than once, we'll define a
> filter plugin in zuul called 'zuul_list_from_value' (or some better name)
> that will return the list of keys that match a given value. So that given
> the following in a job defintion:
> 
> vars:
>   zuul_copy_output:
>     foo/bar.html: logs
>     other/logs/here.html: logs
>     foo/bar.tgz: artifacts
> 
> Corresponding Ansible could do:
> 
> - copy:
>     src: {{ item }}
>     dest: {{ zuul_log_dir }}
>   with_items: {{ zuul_copy_output | zuul_list_from_value(logs) }}
> 
> For OpenStack Today
> ===================
> 
> We will define zuul_output_dir to be "{{ ansible_user_dir }}/zuul-output" in
> our site-variables.yaml file.
> 
> Implement the following in OpenStack's base job:
> 
> We will have the base job will include the simple copying role.
> 
> Logs
> ----
> 
> Base job post playbook will always grab {{ zuul_output_dir }}/logs from
> nodes, same as today:
> * If there are more than one node, grab it into {{ zuul.executor.log_dir
> }}/{{ inventory_hostname }}.
> * If only one node, grab into  {{ zuul.executor.log_dir }} directly
> 
> This will mean moving things like 'fetch-tox-output' logic into the 'tox'
> role itself, so after it's run the appropriate tox logs will be copied to {{
> zuul_output_dir }}/logs
> 
> Artifacts and docs
> ------------------
> 
> Base job will always ...
> 
> * Grab {{ zuul_output_dir }}/artifacts and {{ zuul_output_dir }}/docs to {{
> zuul.executor.work_dir }}/artifacts and {{ zuul.executor.work_dir }}/docs
> * Publish docs content to logs.openstack.org/{{ zuul_log_path }}/docs -
> allows for easy setting of success-url to 'docs/html" in base jobs
> regardless of the subdir the job puts docs in to (see docs vs. releasenotes)
> * rsync docs content to logs.o.o/{{ zuul_log_path }}/docs after contents of
> {{ zuul_output_dir }}/logs so that explicit docs win if there is overlap
> * Sign each artifact
> * Copy each artifact to {{ zuul.project.short_name }}.{{ zuul.tag }}.{{
> suffix }} OR {{ zuul.project.short_name }}.{{ zuul.branch }}.{{ suffix }}
> depending on if {{ zuul.tag exists }} or not
> * If {{ zuul.change }} is set, upload artifacts to {{ zuul_log_path
> }}/artifacts - to allow people to iterate on artifact upload jobs in check
> and verify that they work
> * If {{ zuul.change }} is not set, upload artifacts to
> tarballs.openstack.org/{{ zuul.project.short_name }, same as today.
> Communicate artifact and docs upload paths to child jobs using:
> https://review.openstack.org/#/c/504808/
> 
++

I've made one small change your current logic, by creating new
fetch-zuul-log[2] and fetch-zuul-docs[3] roles. Only because I think it is much
easier to read then adding everything into a single fetch-output role.

We'd do the same for artifacts, and any new folder a zuul user would create.
This also allows an operator the ability to choose which role to also include
for their base jobs.

[2] https://review.openstack.org/583346/
[3] https://review.openstack.org/583350/

> Artifact Release jobs
> =====================
> 
> The release jobs become similar to v2 release jobs, but can be much more
> generalized and reusable. They can also mostly be nodeless jobs (javascript
> npm publication is a notable exception - it will need a node)
> 
> Artifacts
> ---------
> 
> * Fetch artifacts and signatures from tarballs.openstack.org/{{
> zuul.project.short_name }} - based on values set by parent job
> Verify fetched signature is correct
> * Upload artifact to appropriate destination(s) (pypi, npm, maven, etc)
> 
> Docs
> ----
> 
> * Rsync docs from logs.openstack.org/{{ zuul_log_path }}/docs - based on
> values set by parent job
> * Publish to release-job-specific destination as the current jobs do
> 
> This should allow us to make job/project/content specific publication jobs
> without needing those jobs to duplicate content in the build jobs.
> 
> It means we can ALSO put jobs like 'publish-to-pypi' in zuul-jobs without
> needing a second copy in openstack-zuul-jobs because openstack happens to
> builds tarballs differently. The pypi publication step is the same no matter
> how the tarball is produced.
> 
>     - jobs:
>         - build-openstack-sphinx-docs
>         - publish-sphinx-docs-to-afs:
>             dependencies:
>                 - build-openstack-sphinx-docs
> 
> Rollout
> =======
> 
> The steps should be fairly straightforward (and are fine to do post rollout)
> 
> * Add variable to site-variables
> * Add support to base job for creating directories and then copying content
> back
> * Add easy copy helper role
> * Start transitioning jobs in zuul-jobs and openstack-zuul-jobs to using the
> new interface - can be done one at a time
> * Update usage
> 
> If people are ok with the direction generally, we should be able to get the
> enablement pieces in pretty quickly.
> 
> Monty
> 
I'd like to first land our base-minimal[3] (trusted) / base (untrusted) changes
then this. But I've already done some testing of new log publishing[4] and works
as expected.

- Paul

[3] https://review.openstack.org/#/q/topic:base-minimal-jobs
[4] https://review.openstack.org/584614/