[openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs

Daniel P. Berrange berrange at redhat.com
Fri Aug 22 14:13:28 UTC 2014

On Fri, Aug 22, 2014 at 02:33:27PM +0200, Thierry Carrez wrote:
> Hi everyone,
> We all know being a project PTL is an extremely busy job. That's because
> in our structure the PTL is responsible for almost everything in a project:
> - Release management contact
> - Work prioritization
> - Keeping bugs under control
> - Communicate about work being planned or done
> - Make sure the gate is not broken
> - Team logistics (run meetings, organize sprints)
> - ...

This is a good list of responsbilities, but I feel like we're missing
something that is possibly a little too fuzzy to describe a bullet
point. In terms of our governance process, PTL is an elected role, so
in that sense the PTL holds an implicit duty to represent the interests
of the project constituents. I'd characterise this as setting the overall
big picture direction, so that the core team (and/or czars) are focusing
their efforts in the right directioon and generally ensuring that the
project is operating in a healthy manner. Perhaps you could class that
as 'team logistics' but I feel that the idea of 'representation the
voters' is is worth calling out explicitly.

Is there any formal writeup of the PTL role's responsibilities ?
With Google so far I only found one paragraph of governance


  "Project Technical Leads (PTLs) lead individual projects. A PTL
   is ultimately responsible for the direction for each project,
   makes tough calls when needed, organizes the work and teams in
   the project and determines if other forms of project leadership
   are needed. The PTL for a project is elected by the body of
   contributors to that particular project."

Anyway, with the idea of elections & representation in mind....

> Enter the Czar system: each project should have a number of liaisons /
> official contacts / delegates that are fully responsible to cover one
> aspect of the project. We need to have Bugs czars, which are responsible
> for getting bugs under control. We need to have Oslo czars, which serve
> as liaisons for the Oslo program but also as active project-local oslo
> advocates. We need Security czars, which the VMT can go to to progress
> quickly on plugging vulnerabilities. We need release management czars,
> to handle the communication and process with that painful OpenStack
> release manager. We need Gate czars to serve as first-line-of-contact
> getting gate issues fixed... You get the idea.
> Some people can be czars of multiple areas. PTLs can retain some czar
> activity if they wish. Czars can collaborate with their equivalents in
> other projects to share best practices. We just need a clear list of
> areas/duties and make sure each project has a name assigned to each.
> Now, why czars ? Why not rely on informal activity ? Well, for that
> system to work we'll need a lot of people to step up and sign up for
> more responsibility. Making them "czars" makes sure that effort is
> recognized and gives them something back. Also if we don't formally
> designate people, we can't really delegate and the PTL will still be
> directly held responsible. The Release management czar should be able to
> sign off release SHAs without asking the PTL. The czars and the PTL
> should collectively be the new "project drivers".
> At that point, why not also get rid of the PTL ? And replace him with a
> team of czars ? If the czar system is successful, the PTL should be
> freed from the day-to-day operational duties and will be able to focus
> on the project health again. We still need someone to keep an eye on the
> project-wide picture and coordinate the work of the czars. We need
> someone to pick czars, in the event multiple candidates sign up. We also
> still need someone to have the final say in case of deadlocked issues.

.... I'm wondering how people come to be czars ? You don't say explicitly,
but reading this it feels like the team of czars would be more or less
self-selecting amongst the project contributors or nominated by the PTL ?

Thus if we took the next step and got rid of the PTL, then we seem to
have entirely removed the idea of democratically elected leadership from
the individual projects. Is this interpretation correct, wrt the idea of
a czar system with PTL role abolished ?

> People say we don't have that many deadlocks in OpenStack for which the
> PTL ultimate power is needed, so we could get rid of them. I'd argue
> that the main reason we don't have that many deadlocks in OpenStack is
> precisely *because* we have a system to break them if they arise. That
> encourages everyone to find a lazy consensus. That part of the PTL job
> works. Let's fix the part that doesn't work (scaling/burnout).

Even if we didn't have deadlocks with a pure czar system, I fear that
we would be loosing something important by no longer having the project
members directly elect their leader. The elections serve to give the
membership a sense of representation & control over the direction of
the project. Without that you have the risk that a self-selected group
of people become defacto controllers of the project, with no formal
system in place to depose them out if they were to stop acting in
interests of the rest of the contributors.

I hope that would never become a real problem for OpenStack, but I
would still feel uncomfortable about changing governance based on
such an assumption. IOW, even if a PTL were to delegate all their
responsibility such that they're merely a figurehead in terms of the
day-to-day functioning of the project, I think it is desirable that
they remain there as a role that is elected & ultimately responsible.

Overall though, I think the idea of the PTL delegating work out to a
team of czars, who are explicitly considered to be working on his/her
behalf, has merit, especially for the larger projects like Nova. It
clearly isn't in our interests long term for our PTLs to get burnt
out by the workload they take on, as that will discourage otherwise
good people from stepping forward for PTL roles. It is mostly the
questions about representation / goverance that I'm interested in
more clarity on.

|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

More information about the OpenStack-dev mailing list