Open Stack

On Fri, Aug 22, 2014 at 8:19 PM, John Dickinson <me at not.mn> wrote:
> I think Anne makes some excellent points about the pattern being proposed being unlikely to be commonly implemented across all the programs (or, at best, very difficult). Let's not try to formalize another "best practice" that works many times and force it to work every time. Here's an alternate proposal:
>
> Let's let PTLs be PTLs and effectively coordinate and manage the activity in their respective projects. And let's get the PTLs together for one or two days every cycle to discuss project issues. Just PTLs, and let's focus on the project management stuff and some cross-project issues.
>
> Getting the PTLs together would allow them to discuss cross-project issues, share frustrations and solutions about what does and doesn't work. Basically, think of it as a mid-cycle meetup, but for PTLs. (Perhaps we could even ask the Foundation to sponsor it.)
>
+100

I think John nails the key point here: PTLs are likely already doing a
lot of what Thierry originally proposed here. I know that I work with
many people in Neutron to help offload things like bug triaging, gate
failure analysis, etc. Without that, Neutron wouldn't scale. In effect
what's proposed here is something we're already doing a lot of. I
don't think forcing this on each project is a good way of doing this,
because each project has different challenges and needs.

Thanks,
Kyle

> --John
>
>
>
>
>
> On Aug 22, 2014, at 6:02 PM, Anne Gentle <anne at openstack.org> wrote:
>
>>
>>
>>
>> On Fri, Aug 22, 2014 at 6:17 PM, Rochelle.RochelleGrober <rochelle.grober at huawei.com> wrote:
>> /flame-on
>> Ok, this is funny to some of us in the community.  The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than "manager".  Sorry, but this needed to be said.
>> /flame-off
>>
>> Specific comments in line:
>>
>> Thierry Carrez wrote:
>> >
>> > Hi everyone,
>> >
>> > We all know being a project PTL is an extremely busy job. That's
>> > because
>> > in our structure the PTL is responsible for almost everything in a
>> > project:
>> >
>> > - Release management contact
>> > - Work prioritization
>> > - Keeping bugs under control
>> > - Communicate about work being planned or done
>> > - Make sure the gate is not broken
>> > - Team logistics (run meetings, organize sprints)
>> > - ...
>> >
>>
>> Point of clarification:  I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it?  It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program.
>>
>>
>> Now Program, formerly Project.
>>
>> I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program.  Look at how it is expanding:
>>
>> Multiple sub-teams for:  LBAAS, DNAAS, GBP, etc.  This model could be extended such that:
>> - the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project.
>> - the final +2/A would be from the Program reviewers to ensure that all integrate nicely together into a single, cohesive program.
>> - This would allow sub-projects to have core reviewers, along with the program and be a good separation of duties.  It would also help to increase the number of reviews moving to merged code.
>> - Taken to a logical stepping stone, you would have project technical leads for each project, and they would make up a program council, with the program technical lead being the chair of the council.
>>
>> This is a way to offload a good chunk of PTL tactical responsibilities and help them focus more on the strategic.
>>
>> > They end up being completely drowned in those day-to-day operational
>> > duties, miss the big picture, can't help in development that much
>> > anymore, get burnt out. Since you're either "the PTL" or "not the PTL",
>> > you're very alone and succession planning is not working that great
>> > either.
>> >
>> > There have been a number of experiments to solve that problem. John
>> > Garbutt has done an incredible job at helping successive Nova PTLs
>> > handling the release management aspect. Tracy Jones took over Nova bug
>> > management. Doug Hellmann successfully introduced the concept of Oslo
>> > liaisons to get clear point of contacts for Oslo library adoption in
>> > projects. It may be time to generalize that solution.
>> >
>> > The issue is one of responsibility: the PTL is ultimately responsible
>> > for everything in a project. If we can more formally delegate that
>> > responsibility, we can avoid getting up to the PTL for everything, we
>> > can rely on a team of people rather than just one person.
>> >
>> > Enter the Czar system: each project should have a number of liaisons /
>> > official contacts / delegates that are fully responsible to cover one
>> > aspect of the project. We need to have Bugs czars, which are
>> > responsible
>> > for getting bugs under control. We need to have Oslo czars, which serve
>> > as liaisons for the Oslo program but also as active project-local oslo
>> > advocates. We need Security czars, which the VMT can go to to progress
>> > quickly on plugging vulnerabilities. We need release management czars,
>> > to handle the communication and process with that painful OpenStack
>> > release manager. We need Gate czars to serve as first-line-of-contact
>> > getting gate issues fixed... You get the idea.
>> >
>> /flame-on
>> Let's call spades, spades here.  Czar is not only overkill, but the wrong metaphor.
>> /flame-off
>>
>> I'm with Rocky on the anti-czar-as-a-word camp. We all like clever names to shed the "corporate" stigma but this word ain't it. Liaison or lead?
>>
>> Also wanted to point to https://wiki.openstack.org/wiki/PTLguide as it's quite nice.
>>
>> I think PTLs tend to find help when they absolutely are ready to fall over, and I'm all for a plan that helps us not fall over. I've had people step up for bug triaging, gate work, tests, and oslo, sometimes one person did three or four at once. I certainly can't do it all for cross-project. Based on what I've seen, I doubt that we can add this much formality to this across 20+ programs. It's the "bigger more integrated project" vs. "smaller more focused project" difference that won't let us do a pattern here. We can certainly document the responsibilities and let programs know there are some best practices.
>>
>> Anne
>>
>>
>> Each position suggested here exists in corporate development projects:
>> - Bug czar == bug manager/administrator/QA engineer/whatever - someone in charge of making sure bugs get triages, assigned and completed
>> - Oslo czar == systems engineers/project managers who make sure that the project is in line with the rest of the projects that together make an integrated release.  This position needs to stretch beyond just Oslo to encompass all the cross-project requirements and will likely be its own "committee"
>> - Gate Czar == integration engineer(manager)/QA engineer(manager)/build-release engineer.  This position would also likely be a liaison to Infra.
>> - Security Czar == security guru (that name takes me back ;-)
>> - Release management Czar == Project release manager
>> - Doc Czar == tech editor
>> - Tempest Czar == QA engineer(manager)
>>
>> Yes, programs are now mostly big enough to require coordination and management.  The roles are long defined, so let's just document the definitions and get volunteers.  Each project would be able to decide which roles needed to be filled and whether individuals are capable of filling multiple roles.
>>
>> These roles need to be transparent in a governance sense and should be increasing communication between projects, community and contributors.
>>
>> > Some people can be czars of multiple areas. PTLs can retain some czar
>> > activity if they wish. Czars can collaborate with their equivalents in
>> > other projects to share best practices. We just need a clear list of
>> > areas/duties and make sure each project has a name assigned to each.
>> >
>> > Now, why czars ? Why not rely on informal activity ? Well, for that
>> > system to work we'll need a lot of people to step up and sign up for
>> > more responsibility. Making them "czars" makes sure that effort is
>> > recognized and gives them something back. Also if we don't formally
>> > designate people, we can't really delegate and the PTL will still be
>> > directly held responsible. The Release management czar should be able
>> > to
>> > sign off release SHAs without asking the PTL. The czars and the PTL
>> > should collectively be the new "project drivers".
>> >
>> > At that point, why not also get rid of the PTL ? And replace him with a
>> > team of czars ? If the czar system is successful, the PTL should be
>> > freed from the day-to-day operational duties and will be able to focus
>> > on the project health again. We still need someone to keep an eye on
>> > the
>> > project-wide picture and coordinate the work of the czars. We need
>> > someone to pick czars, in the event multiple candidates sign up. We
>> > also
>> > still need someone to have the final say in case of deadlocked issues.
>> >
>> > People say we don't have that many deadlocks in OpenStack for which the
>> > PTL ultimate power is needed, so we could get rid of them. I'd argue
>> > that the main reason we don't have that many deadlocks in OpenStack is
>> > precisely *because* we have a system to break them if they arise. That
>> > encourages everyone to find a lazy consensus. That part of the PTL job
>> > works. Let's fix the part that doesn't work (scaling/burnout).
>> >
>>
>> Perhaps, the crux of the issue with PTLs is that we still need PTLs, but they should be *Project* Technical Leads and the role currently defined as PTL should really become a Program Manager - someone who coordinates all the projects, their issues, needs, communications and leave the deep technical work to the project leads and individual developers.  Consider the project technical leads seeing the trees and the program manager sees the forest.  Let the program manager gather the technical requirements and wishes for the next release and track those, let the team of project technical leads determine the design, architecture, tools, etc.
>>
>> > --
>> > Thierry Carrez (ttx)
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> Great idea Thierry, just thought I'd propose a slight twist on your concept.
>> And sorry for the flames, but I'm in Jay Pipe's camp when it comes to needing to ensure inclusiveness, openness and transparency for the community and I think even the name czars are a slippery slope to something less open.
>>
>> --Rocky Grober
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>