Open Stack

/flame-on
Ok, this is funny to some of us in the community.  The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than "manager".  Sorry, but this needed to be said.
/flame-off

Specific comments in line:

Thierry Carrez wrote:
> 
> Hi everyone,
> 
> We all know being a project PTL is an extremely busy job. That's
> because
> in our structure the PTL is responsible for almost everything in a
> project:
> 
> - Release management contact
> - Work prioritization
> - Keeping bugs under control
> - Communicate about work being planned or done
> - Make sure the gate is not broken
> - Team logistics (run meetings, organize sprints)
> - ...
> 

Point of clarification:  I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it?  It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program.

I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program.  Look at how it is expanding:

Multiple sub-teams for:  LBAAS, DNAAS, GBP, etc.  This model could be extended such that:  
- the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project.
- the final +2/A would be from the Program reviewers to ensure that all integrate nicely together into a single, cohesive program.  
- This would allow sub-projects to have core reviewers, along with the program and be a good separation of duties.  It would also help to increase the number of reviews moving to merged code.
- Taken to a logical stepping stone, you would have project technical leads for each project, and they would make up a program council, with the program technical lead being the chair of the council.

This is a way to offload a good chunk of PTL tactical responsibilities and help them focus more on the strategic.

> They end up being completely drowned in those day-to-day operational
> duties, miss the big picture, can't help in development that much
> anymore, get burnt out. Since you're either "the PTL" or "not the PTL",
> you're very alone and succession planning is not working that great
> either.
> 
> There have been a number of experiments to solve that problem. John
> Garbutt has done an incredible job at helping successive Nova PTLs
> handling the release management aspect. Tracy Jones took over Nova bug
> management. Doug Hellmann successfully introduced the concept of Oslo
> liaisons to get clear point of contacts for Oslo library adoption in
> projects. It may be time to generalize that solution.
> 
> The issue is one of responsibility: the PTL is ultimately responsible
> for everything in a project. If we can more formally delegate that
> responsibility, we can avoid getting up to the PTL for everything, we
> can rely on a team of people rather than just one person.
> 
> Enter the Czar system: each project should have a number of liaisons /
> official contacts / delegates that are fully responsible to cover one
> aspect of the project. We need to have Bugs czars, which are
> responsible
> for getting bugs under control. We need to have Oslo czars, which serve
> as liaisons for the Oslo program but also as active project-local oslo
> advocates. We need Security czars, which the VMT can go to to progress
> quickly on plugging vulnerabilities. We need release management czars,
> to handle the communication and process with that painful OpenStack
> release manager. We need Gate czars to serve as first-line-of-contact
> getting gate issues fixed... You get the idea.
> 
/flame-on
Let's call spades, spades here.  Czar is not only overkill, but the wrong metaphor.
/flame-off

Each position suggested here exists in corporate development projects:
- Bug czar == bug manager/administrator/QA engineer/whatever - someone in charge of making sure bugs get triages, assigned and completed
- Oslo czar == systems engineers/project managers who make sure that the project is in line with the rest of the projects that together make an integrated release.  This position needs to stretch beyond just Oslo to encompass all the cross-project requirements and will likely be its own "committee"
- Gate Czar == integration engineer(manager)/QA engineer(manager)/build-release engineer.  This position would also likely be a liaison to Infra.
- Security Czar == security guru (that name takes me back ;-)
- Release management Czar == Project release manager
- Doc Czar == tech editor
- Tempest Czar == QA engineer(manager)

Yes, programs are now mostly big enough to require coordination and management.  The roles are long defined, so let's just document the definitions and get volunteers.  Each project would be able to decide which roles needed to be filled and whether individuals are capable of filling multiple roles.

These roles need to be transparent in a governance sense and should be increasing communication between projects, community and contributors.

> Some people can be czars of multiple areas. PTLs can retain some czar
> activity if they wish. Czars can collaborate with their equivalents in
> other projects to share best practices. We just need a clear list of
> areas/duties and make sure each project has a name assigned to each.
> 
> Now, why czars ? Why not rely on informal activity ? Well, for that
> system to work we'll need a lot of people to step up and sign up for
> more responsibility. Making them "czars" makes sure that effort is
> recognized and gives them something back. Also if we don't formally
> designate people, we can't really delegate and the PTL will still be
> directly held responsible. The Release management czar should be able
> to
> sign off release SHAs without asking the PTL. The czars and the PTL
> should collectively be the new "project drivers".
> 
> At that point, why not also get rid of the PTL ? And replace him with a
> team of czars ? If the czar system is successful, the PTL should be
> freed from the day-to-day operational duties and will be able to focus
> on the project health again. We still need someone to keep an eye on
> the
> project-wide picture and coordinate the work of the czars. We need
> someone to pick czars, in the event multiple candidates sign up. We
> also
> still need someone to have the final say in case of deadlocked issues.
> 
> People say we don't have that many deadlocks in OpenStack for which the
> PTL ultimate power is needed, so we could get rid of them. I'd argue
> that the main reason we don't have that many deadlocks in OpenStack is
> precisely *because* we have a system to break them if they arise. That
> encourages everyone to find a lazy consensus. That part of the PTL job
> works. Let's fix the part that doesn't work (scaling/burnout).
> 

Perhaps, the crux of the issue with PTLs is that we still need PTLs, but they should be *Project* Technical Leads and the role currently defined as PTL should really become a Program Manager - someone who coordinates all the projects, their issues, needs, communications and leave the deep technical work to the project leads and individual developers.  Consider the project technical leads seeing the trees and the program manager sees the forest.  Let the program manager gather the technical requirements and wishes for the next release and track those, let the team of project technical leads determine the design, architecture, tools, etc.

> --
> Thierry Carrez (ttx)
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Great idea Thierry, just thought I'd propose a slight twist on your concept.
And sorry for the flames, but I'm in Jay Pipe's camp when it comes to needing to ensure inclusiveness, openness and transparency for the community and I think even the name czars are a slippery slope to something less open.

--Rocky Grober