Open Stack

In general I agree on this proposal (as it's similar to what I and others
have talked about implementing in another Zuul installation). I can't speak
too much on the OpenStack use side as I don't have the domain knowledge,
but I agree it's far easier to train people to "plunk files here that you
want archived".


- jlk

On Tue, Oct 10, 2017 at 3:42 PM, Monty Taylor <mordred at inaugust.com> wrote:

> Hey everybody!
>
> I'd like to make a proposal for changing how we do logs/artifacts/docs
> collection based on the last few weeks/months of writing things- and of
> having to explain to people how to structure build and publish jobs over
> the last couple of weeks.
>
> tl;dr - I'd like to change the publication interface to be directories on
> the remote node, not directories on the executor
>
> Rationale
> =========
>
> If jobs have to copy files back to the executor as part of the publication
> interface, then the zuul admins can't change the mechanism of how
> artifacts, logs or docs are published without touching a ton of potentially
> in-tree job content.
>
> Doing so should also allow us to stop having a second copy of build logic
> in the artifact release jobs.
>
> Implementation
> ==============
>
> Define a root 'output' dir on the remote nodes. Different types of output
> can be collected by putting them into subdirectories of that dir on the
> remote nodes and expecting that base jobs will take care of them.
>
> People using jobs defined in zuul-jobs should define a variable
> "zuul_output_dir", either in site-variables or in their base job. Jobs in
> zuul-jobs can and will depend on that variable existing - it will be
> considered part of the base job interface zuul-jobs expects.
>
> Jobs in zuul-jobs will recognize three specific types of job output:
> * logs
> * artifacts
> * docs
>
> Jobs in zuul-jobs will put job outputs into "{{ zuul_output_dir }}/logs",
> "{{ zuul_ouptut_dir }}/artifacts" and "{{ zuul_output_dir }}/docs" as
> appropriate.
>
> A role will be added to zuul-jobs that can be used in base jobs that will
> ensure those directories all exist.
>
> Compression
> -----------
>
> Deployers may choose to have their base job compress items in {{
> zuul_output_dir }} as part of processing them, or may prefer not to
> depending on whether CPU or network is more precious. Jobs in zuul-jobs
> should just move/copy things into {{ zuul_output_dir }} on the node and
> leave compression, transfer and publication as a base-job operation.
>
> Easy Output Copying
> -------------------
>
> A role will also be added to zuul-jobs to facilitate easy/declarative
> output copying.
>
> It will take as input a dictionary of files/folders named
> 'zuul_copy_output'. The role will copy contents into {{ zuul_output_dir }}
> on the remote node and is intended to be used before output fetching in a
> base job's post-playook.
>
> The input will be a dictionary so that zuul variable merging can be used
> for accumulating.
>
> Keys of the dictionary will be things to copy. Valid values will be the
> type of output to copy:
>
> * logs
> * artifacts
> * docs
> * null   # ansible null, not the string null
>
> null will be used for 'a parent job said to copy this, but this job wants
> to not do so'
>
> The simple content copying role will not be flexible or powerful. People
> wanting more expressive output copying have all of the normal tools for
> moving files around at their disposal. It will obey the following rules:
>
> * All files/folders will be copied if they exist, or ignored if they don't
> * Items will be copied as-if 'cp -a' had been used.
> * Order of files is undefined
> * Conflicts between declared files are an error.
>
> Jobs defined in zuul-jobs should not depend on the {{ zuul_copy_output }}
> variable for content they need copied in place on a remote node. Jobs
> defined in zuul-jobs should instead copy their output to {{ zuul_output_dir
> }} This prevents zuul deployers from being required to put the easy output
> copying role into their base jobs. Jobs defined in zuul-jobs may use the
> role behind the scenes.
>
> Filter Plugin
> -------------
>
> Since the pattern of using a dictionary in job variables to take advantage
> of variable merging is bound to come up more than once, we'll define a
> filter plugin in zuul called 'zuul_list_from_value' (or some better name)
> that will return the list of keys that match a given value. So that given
> the following in a job defintion:
>
> vars:
>   zuul_copy_output:
>     foo/bar.html: logs
>     other/logs/here.html: logs
>     foo/bar.tgz: artifacts
>
> Corresponding Ansible could do:
>
> - copy:
>     src: {{ item }}
>     dest: {{ zuul_log_dir }}
>   with_items: {{ zuul_copy_output | zuul_list_from_value(logs) }}
>
> For OpenStack Today
> ===================
>
> We will define zuul_output_dir to be "{{ ansible_user_dir }}/zuul-output"
> in our site-variables.yaml file.
>
> Implement the following in OpenStack's base job:
>
> We will have the base job will include the simple copying role.
>
> Logs
> ----
>
> Base job post playbook will always grab {{ zuul_output_dir }}/logs from
> nodes, same as today:
> * If there are more than one node, grab it into {{ zuul.executor.log_dir
> }}/{{ inventory_hostname }}.
> * If only one node, grab into  {{ zuul.executor.log_dir }} directly
>
> This will mean moving things like 'fetch-tox-output' logic into the 'tox'
> role itself, so after it's run the appropriate tox logs will be copied to
> {{ zuul_output_dir }}/logs
>
> Artifacts and docs
> ------------------
>
> Base job will always ...
>
> * Grab {{ zuul_output_dir }}/artifacts and {{ zuul_output_dir }}/docs to
> {{ zuul.executor.work_dir }}/artifacts and {{ zuul.executor.work_dir }}/docs
> * Publish docs content to logs.openstack.org/{{
> <http://logs.openstack.org/%7B%7B> zuul_log_path }}/docs - allows for
> easy setting of success-url to 'docs/html" in base jobs regardless of the
> subdir the job puts docs in to (see docs vs. releasenotes)
> * rsync docs content to logs.o.o/{{ zuul_log_path }}/docs after contents
> of {{ zuul_output_dir }}/logs so that explicit docs win if there is overlap
> * Sign each artifact
> * Copy each artifact to {{ zuul.project.short_name }}.{{ zuul.tag }}.{{
> suffix }} OR {{ zuul.project.short_name }}.{{ zuul.branch }}.{{ suffix }}
> depending on if {{ zuul.tag exists }} or not
> * If {{ zuul.change }} is set, upload artifacts to {{ zuul_log_path
> }}/artifacts - to allow people to iterate on artifact upload jobs in check
> and verify that they work
> * If {{ zuul.change }} is not set, upload artifacts to
> tarballs.openstack.org/{{ <http://tarballs.openstack.org/%7B%7B>
> zuul.project.short_name }, same as today.
> Communicate artifact and docs upload paths to child jobs using:
> https://review.openstack.org/#/c/504808/
>
> Artifact Release jobs
> =====================
>
> The release jobs become similar to v2 release jobs, but can be much more
> generalized and reusable. They can also mostly be nodeless jobs (javascript
> npm publication is a notable exception - it will need a node)
>
> Artifacts
> ---------
>
> * Fetch artifacts and signatures from tarballs.openstack.org/{{
> <http://tarballs.openstack.org/%7B%7B> zuul.project.short_name }} - based
> on values set by parent job
> Verify fetched signature is correct
> * Upload artifact to appropriate destination(s) (pypi, npm, maven, etc)
>
> Docs
> ----
>
> * Rsync docs from logs.openstack.org/{{ <http://logs.openstack.org/%7B%7B>
> zuul_log_path }}/docs - based on values set by parent job
> * Publish to release-job-specific destination as the current jobs do
>
> This should allow us to make job/project/content specific publication jobs
> without needing those jobs to duplicate content in the build jobs.
>
> It means we can ALSO put jobs like 'publish-to-pypi' in zuul-jobs without
> needing a second copy in openstack-zuul-jobs because openstack happens to
> builds tarballs differently. The pypi publication step is the same no
> matter how the tarball is produced.
>
>     - jobs:
>         - build-openstack-sphinx-docs
>         - publish-sphinx-docs-to-afs:
>             dependencies:
>                 - build-openstack-sphinx-docs
>
> Rollout
> =======
>
> The steps should be fairly straightforward (and are fine to do post
> rollout)
>
> * Add variable to site-variables
> * Add support to base job for creating directories and then copying
> content back
> * Add easy copy helper role
> * Start transitioning jobs in zuul-jobs and openstack-zuul-jobs to using
> the new interface - can be done one at a time
> * Update usage
>
> If people are ok with the direction generally, we should be able to get
> the enablement pieces in pretty quickly.
>
> Monty
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20171012/e3d470e9/attachment.html>