Open Stack

Greetings!

This periodic update is primarily intended as a way to keep
contributors to the OpenStack community apprised of Zuul v3 project
status, including future changes and milestones on our way to use in
production. Additionally, the numerous existing and future users of
Zuul outside of the OpenStack community may find this update useful as
a way to track Zuul v3 development status.

If "changes are coming in the land of Zuul" is new news to you, please
read the section "About Zuul and Zuul v3" towards the end of this
email.

== Zuul v3 project status and updates ==

The biggest recent development is that basic support for GitHub has
merged!  Thanks to Jan, Tobias, Jonathan, Jamie, Jesse, and everyone
else that helped with that years-long effort!  We're still working to
achieve feature parity (notably, cross-repo dependency support hasn't
been implemented yet), but basic operations work and we have a good base
to start from.

We've also landed support for bubblewrap, so that untrusted job content
can run in a restricted environment.  This is a big improvement for
executor security.  Thanks to Clint and others who helped with this!

We merged support for live-streaming interleaved ansible logs and
console logs from all of the hosts in a job.  The streaming protocol is
compatible with finger, so you can easily request the log for a job by
running "finger UUID at executor".  That's handy for using unix tools to
deal with the output (think grep, sed, awk, etc).  To make this
accessible over the web, we are working on a websocket based console
streamer, which uses the finger-compatible endpoints on the backend.
When we're done, we'll have a nice web frontend for easily viewing
console logs linked to from the status page, and finger URLs for users
who want to view or process their logs from a unix shell.  Thanks to
David and Monty for work on this!

We've created some new repositories to hold Zuul jobs and the Ansible
roles that they use.  We're going to try something new here -- we want
to create a standard library of jobs that any Zuul installation (not
just those related to OpenStack) can use.  Flexibility and local
customization of jobs is very important in Zuul v3, but with job
inheritance and Ansible roles, we have two very useful methods of
composition that we can use to share job content so that not everyone
has to reinvent the wheel.  These are the repos we've created and how we
expect to use them:

  openstack-infra/zuul-jobs

    This is what we're calling the "standard library".  We're going to
    put any jobs which we think are not inherently OpenStack-specific.
    For example, jobs to run python unit tests, java builds, go tests,
    autoconf/makefile based projects, etc.

  openstack-infra/openstack-zuul-jobs

    This is where we will put OpenStack-specific jobs (or
    OpenStack-specific variants of standard library jobs).

In the near term, we're going to start populating these repos with what
we need for OpenStack's Zuul, and will probably move things around quite
a bit as we figure out where they should go.  We are also working on a
Sphinx extension (in the openstack-infra/zuul-sphinx repo) to
automatically document all of the jobs and roles in these repos.  We
should have self-documenting jobs with published documentation right
from the start.  Thanks to Paul for his help on this!

Also thanks to Paul for setting up OpenStack's production instance of
Zuul v3 at zuulv3.openstack.org server and our first executor at
ze01.openstack.org.  That's running now, and we're currently working
through some things that we deferred from setting up our dev instance,
notably log publishing.

With the approval of the nodepool drivers spec:

  http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-drivers.html

Tristan has started work on an implementation supporting multiple
backend drivers for nodepool.  This will initially include a driver for
static nodes, and later we will use this to support multiple cloud
technologies:

  http://lists.openstack.org/pipermail/openstack-infra/2017-June/005387.html

Tristan has also proposed a proof-of-concept implementation of a
dashboard for Zuul, which has prompted a conversation about web
frameworks:

  http://lists.openstack.org/pipermail/openstack-infra/2017-June/005402.html

We're working to come to consensus on that so that we can ultimately
converge our webhooks, status page, websocket console streaming, and
dashboard onto one framework.

Upcoming tasks and focus:
* Re-enabling disabled tests: We're continuing to make our way through
the list of remaining tests that need enabling. See the list, which
includes an annotation as to complexity for each test, here:
https://etherpad.openstack.org/p/zuulv3skips
* Github parity
* Log streaming
* Standard jobs
* Set up production zuulv3.openstack.org server
* Full task list and plan is in the Zuul v3 storyboard:
https://storyboard.openstack.org/#!/board/41

Recent changes:
* Zuul v3:
https://review.openstack.org/#/q/status:closed+project:openstack-infra/zuul+branch:feature/zuulv3
* Nodepool:
https://review.openstack.org/#/q/status:closed+project:openstack-infra/nodepool+branch:feature/zuulv3

== About Zuul and Zuul v3 ==
Zuul is a pipeline-oriented project gating system, driving the project
automation necessary to enable a continuous integration environment
for the OpenStack community. It directs the OpenStack community's
testing, running tens of thousands of jobs each day, responding to
events from the code review system and stacking potential changes to
be tested together. Testing and continuous integration for drivers is
also enabled by OpenStack's deployment of Zuul for ~50 third-party
projects.

Zuul v3 is the upcoming, not-yet-in-production version of Zuul
currently under development, bringing (amongst many others) the
following improvements and features:
* simplifying the ability to run jobs in multi-node environments
* improved management of large numbers of jobs and job variations
* support for in-tree job configuration
* ability to define jobs using Ansible (http://github.com/ansible/ansible)

Even though prior versions of Zuul are already in use by numerous
projects and companies not related to OpenStack efforts, a primary
goal of Zuul v3 is to make Zuul a universally useful CI / CD engine
for any size use case. The design of Zuul v3 improves the modularity
of the system, and enables new triggers (such as GitHub) and reporters
(where results are sent) to be more easily integrated with Zuul;
additionally, the ability to execute jobs on non-OpenStack clouds,
such as, well, the other ones :D, as well as non-cloud environments,
will be possible. (These "beyond OpenStack" changes are expected in
Zuul 3.1, which will follow the moment after Zuul v3 is live and in
production for the OpenStack community. Finally, the move from Jenkins
to Ansible, in addition to improving scalability for large
implementations, allows users to write tests using Ansible,
effectively allowing one to "test in the way that you deploy" and
enables simpler writing of test jobs.

OpenStack's current production use of Zuul supports approximately 2500
developers, more than 1500 git repositories, at a rate of more than
2KJPH (kilo-jobs / hour).

For more background and information on the history, and future, of
Zuul, please see:
* Zuul v3 spec:
https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html
* Jim Blair's Zuul v3 talk at OpenStack Summit in Barcelona:
https://www.youtube.com/watch?v=R4EmE1QEvNU
* "There is no Jenkins, only Zuul" (blog post, Jim Blair, 2016/06/16):
http://amo-probos.org/post/18
* "Zuul v3, What's Coming" (blog post, Monty Taylor, 2017/02/28):
http://inaugust.com/posts/whats-coming-zuulv3.html

Significant dates, milestones, and changes directly affecting
contributors in the OpenStack community will be announced on the
appropriate mailing lists for OpenStack (primarily openstack-dev and
openstack-infra), as well as in future versions of this project status
update.

NOTE: At this point, Zuul v3 is NOT yet secure or stable. You should
not, not, please, DO NOT ATTEMPT TO RUN IT IN PRODUCTION YOURSELF yet.
It is still under heavy development, and v3 is not currently used in
production for OpenStack. Zuul v3 content (jobs, etc.) should not
undergo heavy creation as syntax is still under development and
expected to change between now and release date.

== Zuul Community Information ==

Zuul README, with information on contributing to Zuul v3 and our roadmap:
http://git.openstack.org/cgit/openstack-infra/zuul/tree/README.rst?h=feature/zuulv3

Zuul v3 Storyboard, which tracks tasks and status related to operating
Zuul v3 in production for OpenStack:
https://storyboard.openstack.org/#!/board/41

Mailing list: Discussions related to Zuul development, and current /
future operation in production in the OpenStack community, primarily
occur on openstack-infra
(http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra).
(Note that this mailing list encompasses the numerous infrastructure
projects supported by OpenStack's infrastructure team, and is not
exclusively Zuul-focused.)

IRC Discussion:
Join #openstack-infra on Freenode to discuss Zuul's use in OpenStack
or #zuul to get into the weeds on general Zuul topics and v3
development. For those unacquainted with IRC, see this friendly guide
to using IRC: https://wiki.openstack.org/wiki/IRC

Zuul v3 Weekly IRC meeting:
* Mondays at 2200UTC, in #openstack-meeting-alt on Freenode
* Historical meeting logs: http://eavesdrop.openstack.org/meetings/zuul/
* Further information, including upcoming meeting agenda:
https://wiki.openstack.org/wiki/Meetings/Zuul

Thanks,

Jim, with help from Robyn (on behalf of the openstack-infra team)