Open Stack

+1 on this proposal.

On Tue, Oct 10, 2017 at 11:42 PM Monty Taylor <mordred at inaugust.com> wrote:

> Hey everybody!
>
> I'd like to make a proposal for changing how we do logs/artifacts/docs
> collection based on the last few weeks/months of writing things- and of
> having to explain to people how to structure build and publish jobs over
> the last couple of weeks.
>
> tl;dr - I'd like to change the publication interface to be directories
> on the remote node, not directories on the executor
>
> Rationale
> =========
>
> If jobs have to copy files back to the executor as part of the
> publication interface, then the zuul admins can't change the mechanism
> of how artifacts, logs or docs are published without touching a ton of
> potentially in-tree job content.
>
> Doing so should also allow us to stop having a second copy of build
> logic in the artifact release jobs.
>

Having the logic to pull logs in one place allows to have proper isolation
on the executor between logs coming from different nodes in multi-node
environments.

Andrea Frittoli (andreaf)


>
> Implementation
> ==============
>
> Define a root 'output' dir on the remote nodes. Different types of
> output can be collected by putting them into subdirectories of that dir
> on the remote nodes and expecting that base jobs will take care of them.
>
> People using jobs defined in zuul-jobs should define a variable
> "zuul_output_dir", either in site-variables or in their base job. Jobs
> in zuul-jobs can and will depend on that variable existing - it will be
> considered part of the base job interface zuul-jobs expects.
>
> Jobs in zuul-jobs will recognize three specific types of job output:
> * logs
> * artifacts
> * docs
>
> Jobs in zuul-jobs will put job outputs into "{{ zuul_output_dir
> }}/logs", "{{ zuul_ouptut_dir }}/artifacts" and "{{ zuul_output_dir
> }}/docs" as appropriate.
>
> A role will be added to zuul-jobs that can be used in base jobs that
> will ensure those directories all exist.
>
> Compression
> -----------
>
> Deployers may choose to have their base job compress items in {{
> zuul_output_dir }} as part of processing them, or may prefer not to
> depending on whether CPU or network is more precious. Jobs in zuul-jobs
> should just move/copy things into {{ zuul_output_dir }} on the node and
> leave compression, transfer and publication as a base-job operation.
>
> Easy Output Copying
> -------------------
>
> A role will also be added to zuul-jobs to facilitate easy/declarative
> output copying.
>
> It will take as input a dictionary of files/folders named
> 'zuul_copy_output'. The role will copy contents into {{ zuul_output_dir
> }} on the remote node and is intended to be used before output fetching
> in a base job's post-playook.
>
> The input will be a dictionary so that zuul variable merging can be used
> for accumulating.
>
> Keys of the dictionary will be things to copy. Valid values will be the
> type of output to copy:
>
> * logs
> * artifacts
> * docs
> * null   # ansible null, not the string null
>
> null will be used for 'a parent job said to copy this, but this job
> wants to not do so'
>
> The simple content copying role will not be flexible or powerful. People
> wanting more expressive output copying have all of the normal tools for
> moving files around at their disposal. It will obey the following rules:
>
> * All files/folders will be copied if they exist, or ignored if they don't
> * Items will be copied as-if 'cp -a' had been used.
> * Order of files is undefined
> * Conflicts between declared files are an error.
>
> Jobs defined in zuul-jobs should not depend on the {{ zuul_copy_output
> }} variable for content they need copied in place on a remote node. Jobs
> defined in zuul-jobs should instead copy their output to {{
> zuul_output_dir }} This prevents zuul deployers from being required to
> put the easy output copying role into their base jobs. Jobs defined in
> zuul-jobs may use the role behind the scenes.
>
> Filter Plugin
> -------------
>
> Since the pattern of using a dictionary in job variables to take
> advantage of variable merging is bound to come up more than once, we'll
> define a filter plugin in zuul called 'zuul_list_from_value' (or some
> better name) that will return the list of keys that match a given value.
> So that given the following in a job defintion:
>
> vars:
>    zuul_copy_output:
>      foo/bar.html: logs
>      other/logs/here.html: logs
>      foo/bar.tgz: artifacts
>
> Corresponding Ansible could do:
>
> - copy:
>      src: {{ item }}
>      dest: {{ zuul_log_dir }}
>    with_items: {{ zuul_copy_output | zuul_list_from_value(logs) }}
>
> For OpenStack Today
> ===================
>
> We will define zuul_output_dir to be "{{ ansible_user_dir
> }}/zuul-output" in our site-variables.yaml file.
>
> Implement the following in OpenStack's base job:
>
> We will have the base job will include the simple copying role.
>
> Logs
> ----
>
> Base job post playbook will always grab {{ zuul_output_dir }}/logs from
> nodes, same as today:
> * If there are more than one node, grab it into {{ zuul.executor.log_dir
> }}/{{ inventory_hostname }}.
> * If only one node, grab into  {{ zuul.executor.log_dir }} directly
>
> This will mean moving things like 'fetch-tox-output' logic into the
> 'tox' role itself, so after it's run the appropriate tox logs will be
> copied to {{ zuul_output_dir }}/logs
>
> Artifacts and docs
> ------------------
>
> Base job will always ...
>
> * Grab {{ zuul_output_dir }}/artifacts and {{ zuul_output_dir }}/docs to
> {{ zuul.executor.work_dir }}/artifacts and {{ zuul.executor.work_dir
> }}/docs
> * Publish docs content to logs.openstack.org/{{
> <http://logs.openstack.org/%7B%7B> zuul_log_path }}/docs -
> allows for easy setting of success-url to 'docs/html" in base jobs
> regardless of the subdir the job puts docs in to (see docs vs.
> releasenotes)
> * rsync docs content to logs.o.o/{{ zuul_log_path }}/docs after contents
> of {{ zuul_output_dir }}/logs so that explicit docs win if there is overlap
> * Sign each artifact
> * Copy each artifact to {{ zuul.project.short_name }}.{{ zuul.tag }}.{{
> suffix }} OR {{ zuul.project.short_name }}.{{ zuul.branch }}.{{ suffix
> }} depending on if {{ zuul.tag exists }} or not
> * If {{ zuul.change }} is set, upload artifacts to {{ zuul_log_path
> }}/artifacts - to allow people to iterate on artifact upload jobs in
> check and verify that they work
> * If {{ zuul.change }} is not set, upload artifacts to
> tarballs.openstack.org/{{ <http://tarballs.openstack.org/%7B%7B>
> zuul.project.short_name }, same as today.
> Communicate artifact and docs upload paths to child jobs using:
> https://review.openstack.org/#/c/504808/
>
> Artifact Release jobs
> =====================
>
> The release jobs become similar to v2 release jobs, but can be much more
> generalized and reusable. They can also mostly be nodeless jobs
> (javascript npm publication is a notable exception - it will need a node)
>
> Artifacts
> ---------
>
> * Fetch artifacts and signatures from tarballs.openstack.org/{{
> <http://tarballs.openstack.org/%7B%7B>
> zuul.project.short_name }} - based on values set by parent job
> Verify fetched signature is correct
> * Upload artifact to appropriate destination(s) (pypi, npm, maven, etc)
>
> Docs
> ----
>
> * Rsync docs from logs.openstack.org/{{ <http://logs.openstack.org/%7B%7B>
> zuul_log_path }}/docs - based on
> values set by parent job
> * Publish to release-job-specific destination as the current jobs do
>
> This should allow us to make job/project/content specific publication
> jobs without needing those jobs to duplicate content in the build jobs.
>
> It means we can ALSO put jobs like 'publish-to-pypi' in zuul-jobs
> without needing a second copy in openstack-zuul-jobs because openstack
> happens to builds tarballs differently. The pypi publication step is the
> same no matter how the tarball is produced.
>
>      - jobs:
>          - build-openstack-sphinx-docs
>          - publish-sphinx-docs-to-afs:
>              dependencies:
>                  - build-openstack-sphinx-docs
>
> Rollout
> =======
>
> The steps should be fairly straightforward (and are fine to do post
> rollout)
>
> * Add variable to site-variables
> * Add support to base job for creating directories and then copying
> content back
> * Add easy copy helper role
> * Start transitioning jobs in zuul-jobs and openstack-zuul-jobs to using
> the new interface - can be done one at a time
> * Update usage
>
> If people are ok with the direction generally, we should be able to get
> the enablement pieces in pretty quickly.
>
> Monty
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20171011/b52e1ef8/attachment.html>