<div dir="ltr">In general I agree on this proposal (as it's similar to what I and others have talked about implementing in another Zuul installation). I can't speak too much on the OpenStack use side as I don't have the domain knowledge, but I agree it's far easier to train people to "plunk files here that you want archived".</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br></div><div dir="ltr">- jlk</div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Oct 10, 2017 at 3:42 PM, Monty Taylor <span dir="ltr"><<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey everybody!<br>
<br>
I'd like to make a proposal for changing how we do logs/artifacts/docs collection based on the last few weeks/months of writing things- and of having to explain to people how to structure build and publish jobs over the last couple of weeks.<br>
<br>
tl;dr - I'd like to change the publication interface to be directories on the remote node, not directories on the executor<br>
<br>
Rationale<br>
=========<br>
<br>
If jobs have to copy files back to the executor as part of the publication interface, then the zuul admins can't change the mechanism of how artifacts, logs or docs are published without touching a ton of potentially in-tree job content.<br>
<br>
Doing so should also allow us to stop having a second copy of build logic in the artifact release jobs.<br>
<br>
Implementation<br>
==============<br>
<br>
Define a root 'output' dir on the remote nodes. Different types of output can be collected by putting them into subdirectories of that dir on the remote nodes and expecting that base jobs will take care of them.<br>
<br>
People using jobs defined in zuul-jobs should define a variable "zuul_output_dir", either in site-variables or in their base job. Jobs in zuul-jobs can and will depend on that variable existing - it will be considered part of the base job interface zuul-jobs expects.<br>
<br>
Jobs in zuul-jobs will recognize three specific types of job output:<br>
* logs<br>
* artifacts<br>
* docs<br>
<br>
Jobs in zuul-jobs will put job outputs into "{{ zuul_output_dir }}/logs", "{{ zuul_ouptut_dir }}/artifacts" and "{{ zuul_output_dir }}/docs" as appropriate.<br>
<br>
A role will be added to zuul-jobs that can be used in base jobs that will ensure those directories all exist.<br>
<br>
Compression<br>
-----------<br>
<br>
Deployers may choose to have their base job compress items in {{ zuul_output_dir }} as part of processing them, or may prefer not to depending on whether CPU or network is more precious. Jobs in zuul-jobs should just move/copy things into {{ zuul_output_dir }} on the node and leave compression, transfer and publication as a base-job operation.<br>
<br>
Easy Output Copying<br>
-------------------<br>
<br>
A role will also be added to zuul-jobs to facilitate easy/declarative output copying.<br>
<br>
It will take as input a dictionary of files/folders named 'zuul_copy_output'. The role will copy contents into {{ zuul_output_dir }} on the remote node and is intended to be used before output fetching in a base job's post-playook.<br>
<br>
The input will be a dictionary so that zuul variable merging can be used for accumulating.<br>
<br>
Keys of the dictionary will be things to copy. Valid values will be the type of output to copy:<br>
<br>
* logs<br>
* artifacts<br>
* docs<br>
* null # ansible null, not the string null<br>
<br>
null will be used for 'a parent job said to copy this, but this job wants to not do so'<br>
<br>
The simple content copying role will not be flexible or powerful. People wanting more expressive output copying have all of the normal tools for moving files around at their disposal. It will obey the following rules:<br>
<br>
* All files/folders will be copied if they exist, or ignored if they don't<br>
* Items will be copied as-if 'cp -a' had been used.<br>
* Order of files is undefined<br>
* Conflicts between declared files are an error.<br>
<br>
Jobs defined in zuul-jobs should not depend on the {{ zuul_copy_output }} variable for content they need copied in place on a remote node. Jobs defined in zuul-jobs should instead copy their output to {{ zuul_output_dir }} This prevents zuul deployers from being required to put the easy output copying role into their base jobs. Jobs defined in zuul-jobs may use the role behind the scenes.<br>
<br>
Filter Plugin<br>
-------------<br>
<br>
Since the pattern of using a dictionary in job variables to take advantage of variable merging is bound to come up more than once, we'll define a filter plugin in zuul called 'zuul_list_from_value' (or some better name) that will return the list of keys that match a given value. So that given the following in a job defintion:<br>
<br>
vars:<br>
zuul_copy_output:<br>
foo/bar.html: logs<br>
other/logs/here.html: logs<br>
foo/bar.tgz: artifacts<br>
<br>
Corresponding Ansible could do:<br>
<br>
- copy:<br>
src: {{ item }}<br>
dest: {{ zuul_log_dir }}<br>
with_items: {{ zuul_copy_output | zuul_list_from_value(logs) }}<br>
<br>
For OpenStack Today<br>
===================<br>
<br>
We will define zuul_output_dir to be "{{ ansible_user_dir }}/zuul-output" in our site-variables.yaml file.<br>
<br>
Implement the following in OpenStack's base job:<br>
<br>
We will have the base job will include the simple copying role.<br>
<br>
Logs<br>
----<br>
<br>
Base job post playbook will always grab {{ zuul_output_dir }}/logs from nodes, same as today:<br>
* If there are more than one node, grab it into {{ zuul.executor.log_dir }}/{{ inventory_hostname }}.<br>
* If only one node, grab into {{ zuul.executor.log_dir }} directly<br>
<br>
This will mean moving things like 'fetch-tox-output' logic into the 'tox' role itself, so after it's run the appropriate tox logs will be copied to {{ zuul_output_dir }}/logs<br>
<br>
Artifacts and docs<br>
------------------<br>
<br>
Base job will always ...<br>
<br>
* Grab {{ zuul_output_dir }}/artifacts and {{ zuul_output_dir }}/docs to {{ zuul.executor.work_dir }}/artifacts and {{ zuul.executor.work_dir }}/docs<br>
* Publish docs content to <a href="http://logs.openstack.org/%7B%7B" rel="noreferrer" target="_blank">logs.openstack.org/{{</a> zuul_log_path }}/docs - allows for easy setting of success-url to 'docs/html" in base jobs regardless of the subdir the job puts docs in to (see docs vs. releasenotes)<br>
* rsync docs content to logs.o.o/{{ zuul_log_path }}/docs after contents of {{ zuul_output_dir }}/logs so that explicit docs win if there is overlap<br>
* Sign each artifact<br>
* Copy each artifact to {{ zuul.project.short_name }}.{{ zuul.tag }}.{{ suffix }} OR {{ zuul.project.short_name }}.{{ zuul.branch }}.{{ suffix }} depending on if {{ zuul.tag exists }} or not<br>
* If {{ zuul.change }} is set, upload artifacts to {{ zuul_log_path }}/artifacts - to allow people to iterate on artifact upload jobs in check and verify that they work<br>
* If {{ zuul.change }} is not set, upload artifacts to <a href="http://tarballs.openstack.org/%7B%7B" rel="noreferrer" target="_blank">tarballs.openstack.org/{{</a> zuul.project.short_name }, same as today.<br>
Communicate artifact and docs upload paths to child jobs using: <a href="https://review.openstack.org/#/c/504808/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/504808/</a><br>
<br>
Artifact Release jobs<br>
=====================<br>
<br>
The release jobs become similar to v2 release jobs, but can be much more generalized and reusable. They can also mostly be nodeless jobs (javascript npm publication is a notable exception - it will need a node)<br>
<br>
Artifacts<br>
---------<br>
<br>
* Fetch artifacts and signatures from <a href="http://tarballs.openstack.org/%7B%7B" rel="noreferrer" target="_blank">tarballs.openstack.org/{{</a> zuul.project.short_name }} - based on values set by parent job<br>
Verify fetched signature is correct<br>
* Upload artifact to appropriate destination(s) (pypi, npm, maven, etc)<br>
<br>
Docs<br>
----<br>
<br>
* Rsync docs from <a href="http://logs.openstack.org/%7B%7B" rel="noreferrer" target="_blank">logs.openstack.org/{{</a> zuul_log_path }}/docs - based on values set by parent job<br>
* Publish to release-job-specific destination as the current jobs do<br>
<br>
This should allow us to make job/project/content specific publication jobs without needing those jobs to duplicate content in the build jobs.<br>
<br>
It means we can ALSO put jobs like 'publish-to-pypi' in zuul-jobs without needing a second copy in openstack-zuul-jobs because openstack happens to builds tarballs differently. The pypi publication step is the same no matter how the tarball is produced.<br>
<br>
- jobs:<br>
- build-openstack-sphinx-docs<br>
- publish-sphinx-docs-to-afs:<br>
dependencies:<br>
- build-openstack-sphinx-docs<br>
<br>
Rollout<br>
=======<br>
<br>
The steps should be fairly straightforward (and are fine to do post rollout)<br>
<br>
* Add variable to site-variables<br>
* Add support to base job for creating directories and then copying content back<br>
* Add easy copy helper role<br>
* Start transitioning jobs in zuul-jobs and openstack-zuul-jobs to using the new interface - can be done one at a time<br>
* Update usage<br>
<br>
If people are ok with the direction generally, we should be able to get the enablement pieces in pretty quickly.<br>
<br>
Monty<br>
<br>
______________________________<wbr>_________________<br>
OpenStack-Infra mailing list<br>
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstac<wbr>k.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-infra</a></blockquote></div><br></div>