Open Stack

Hi,

For a while, Zuul has had support for connections to multiple services
for use in triggering jobs, fetching changes, and reporting results.
This has been in use (the github patches work with github, we do support
reporting to gerrit as two different users, we send email over SMTP),
but we have not used the facility to its full potential.  That is
something we want to do in Zuul v3.  Specifically, in addition to all of
the above, we want to be able to have a change to Ansible depend on a
change to shade and be able to test them together.  Ansible is hosted in
github and shade is hosted in gerrit.  By the same token, a third-party
CI operator should be able to point their Zuul at upstream OpenStack and
their own internal Gerrit at the same time.

The pieces are all nearly there for that, but there are a few
assumptions lingering that we need to correct for v3.  They mostly
center around the fact that the "source" for a project (i.e., the
connection over which its source code should be fetched) is specified by
the pipeline.  This was added quite a while ago when we added our first
trigger which did not imply a source (the timer trigger).  By adding the
source to the pipeline, we told Zuul "when the timer goes off, enqueue a
project from gerrit in this pipeline".

That constrains us now, as it is confusing to consider two items from
different connections enqueued in the same pipeline.  There are other
issues too, such as specifying that a job needs an Ansible role from
another Zuul project.

The natural way to correct this seems to be to associate projects with
their connections directly.  Therefore, whenever Zuul encounters a
project (via a trigger, a dependency, or an internal reference) it will
know how to fetch it.

To fully implement this idea necessitates some changes.  With Monty's
help, I have sketched out a plan that should support all of our
use-cases and make this much simpler for developers and users:

1) Associate each source-capable driver with a canonical source-code
   location hostname.  

Our goal is to associate every project with a connection.  Our
connections already have names (like "gerrit") which make perfect sense
as names of pipeline triggers or reporters.  However, they make poor
identifiers for the logical canonical names of source code repos.  If we
were to describe the canonical location for Zuul's source code -- the
place we want users to clone it from -- it would be
git.openstack.org/openstack-infra/zuul.  Adding this attribute allows us
to disambiguate identically named repos from different connections.

Go-lang style project layout and source-code imports are fully-qualified
hostname/path/project identifiers (both on disk and in source code).
For example, "github.com/user/stringutil".  By adopting this facility,
we are better positioned to stage git repos in the Go-lang convention,
and we are using a sensible and common universal identifier for our
projects.

We should default this attribute to the hostname of the connection, but
allow it to be set to another value by the administrator.  So that in
our case, rather than referring to our project locations as
'review.openstack.org/...' we may use 'git.openstack.org.'.

The pipeline and tenant configuration files will continue to reference
the connection name since "trigger: gerrit: ..." makes more sense than
"trigger: git.openstack.org: ...".

2) Use the tenant configuration to associate projects with connections.

The main goal of this exercise is to associate each project with a
connection.  The best place to do that is within the tenant
configuration file.  This is a location where the project-connection
mapping is unambiguous, and it happens early in the configuration
process so that the results are usable by the rest of the
configuration.  The current syntax is:

  - tenant:
      name: openstack
      source:
        gerrit:
          config-repos:
            - openstack-infra/project-config
          project-repos:
            - openstack-infra/zuul

That source section may contain multiple sources (only gerrit in this
case, but could be extended with "github: ...").  Each of the projects
is therefore unambiguously associated with a connection.  Since each
connection has a canonical hostname (via section 1 above), we now also
know the fully qualified canonical location of each source code repo.

While we're on the subject, the terminology config-repos and
project-repos is a bit confusing, especially when we talk about our
config-repo named project-config and the project-repos which have their
own in-project-config.  We started using 'trusted' and 'untrusted'
internally in the launcher to we can keep track of when we're supposed
to deny certain actions.  We should go ahead and rename these
similarly.  So the new syntax would be:

  - tenant:
      name: openstack
      source:
        gerrit:
          trusted-projects:
            - openstack-infra/project-config
          untrusted-projects:
            - openstack-infra/zuul

3) All references to projects may be fully qualified.

Now that we have established a hostname and name for every project, any
time we need to refer to a project we can use the fully qualified form
of the name.  This includes project configuration stanzas, role
references, and potentially even in triggers (for example, an SMTP or
IRC trigger may need a fully qualified form of the project).  For
example:

  - project:
      name: git.openstack.org/openstack/nova

  - job:
      roles:
        - zuul: git.openstack.org/openstack-infra/devstack-gate

Obviously this would get fairly repetitive in our configuration, so we
will make the hostname portion of the name optional, but only if the
project name itself is otherwise unambiguous.

I think this changes will be fairly straightforward and intuitive.  If
they sound good to folks, I can start working on them soon.

-Jim