Open Stack

On 11/11/15 13:39 -0600, Matt Riedemann wrote:
>
>
>On 11/11/2015 8:51 AM, Flavio Percoco wrote:
>>On 09/11/15 21:30 -0600, Matt Riedemann wrote:
>>>On 11/9/2015 9:12 PM, Matthew Treinish wrote:
>>>>On Mon, Nov 09, 2015 at 10:54:43PM +0000, Kuvaja, Erno wrote:
>>>>>>On Mon, Nov 09, 2015 at 05:28:45PM -0500, Doug Hellmann wrote:
>>>>>>>Excerpts from Matt Riedemann's message of 2015-11-09 16:05:29 -0600:
>>>>>>>>
>>>>>>>>On 11/9/2015 10:41 AM, Thierry Carrez wrote:
>>>>>>>>>Hi everyone,
>>>>>>>>>
>>>>>>>>>A few cycles ago we set up the Release Cycle Management team which
>>>>>>>>>was a bit of a frankenteam of the things I happened to be leading:
>>>>>>>>>release management, stable branch maintenance and vulnerability
>>>>>>management.
>>>>>>>>>While you could argue that there was some overlap between those
>>>>>>>>>functions (as in, "all these things need to be released") logic
>>>>>>>>>was not the primary reason they were put together.
>>>>>>>>>
>>>>>>>>>When the Security Team was created, the VMT was spinned out of the
>>>>>>>>>Release Cycle Management team and joined there. Now I think we
>>>>>>>>>should spin out stable branch maintenance as well:
>>>>>>>>>
>>>>>>>>>* A good chunk of the stable team work used to be stable point
>>>>>>>>>release management, but as of stable/liberty this is now done by
>>>>>>>>>the release management team and triggered by the project-specific
>>>>>>>>>stable maintenance teams, so there is no more overlap in tooling
>>>>>>>>>used there
>>>>>>>>>
>>>>>>>>>* Following the kilo reform, the stable team is now focused on
>>>>>>>>>defining and enforcing a common stable branch policy[1], rather
>>>>>>>>>than approving every patch. Being more visible and having more
>>>>>>>>>dedicated members can only help in that very specific mission
>>>>>>>>>
>>>>>>>>>* The release team is now headed by Doug Hellmann, who is focused
>>>>>>>>>on release management and does not have the history I had with
>>>>>>>>>stable branch policy. So it might be the right moment to refocus
>>>>>>>>>release management solely on release management and get the stable
>>>>>>>>>team its own leadership
>>>>>>>>>
>>>>>>>>>* Empowering that team to make its own decisions, giving it more
>>>>>>>>>visibility and recognition will hopefully lead to more resources
>>>>>>>>>being dedicated to it
>>>>>>>>>
>>>>>>>>>* If the team expands, it could finally own stable branch health
>>>>>>>>>and gate fixing. If that ends up all falling under the same roof,
>>>>>>>>>that team could make decisions on support timeframes as well,
>>>>>>>>>since it will be the primary resource to make that work
>>>>>>>>
>>>>>>>>Isn't this kind of already what the stable maint team does? Well,
>>>>>>>>that and some QA people like mtreinish and sdague.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>So.. good idea ? bad idea ? What do current stable-maint-core[2]
>>>>>>>>>members think of that ? Who thinks they could step up to lead that
>>>>>>team ?
>>>>>>>>>
>>>>>>>>>[1]
>>>>>>>>>http://docs.openstack.org/project-team-guide/stable-branches.html
>>>>>>>>>[2] https://review.openstack.org/#/admin/groups/530,members
>>>>>>>>>
>>>>>>>>
>>>>>>>>With the decentralizing of the stable branch stuff in Liberty [1] it
>>>>>>>>seems like there would be less use for a PTL for stable branch
>>>>>>>>maintenance - the cats are now herding themselves, right? Or at
>>>>>>>>least that's the plan as far as I understood it. And the existing
>>>>>>>>stable branch wizards are more or less around for help and answering
>>>>>>questions.
>>>>>>>
>>>>>>>The same might be said about releasing from master and the release
>>>>>>>management team. There's still some benefit to having people
>>>>>>>dedicated
>>>>>>>to making sure projects all agree to sane policies and to keep up
>>>>>>>with
>>>>>>>deliverables that need to be released.
>>>>>>
>>>>>>Except the distinction is that relmgt is actually producing
>>>>>>something. Relmgt
>>>>>>has the releases repo which does centralize library releases, reno
>>>>>>to do the
>>>>>>release notes, etc. What does the global stable core do? Right now
>>>>>>it's there
>>>>>>almost entirely to just add people to the project specific stable
>>>>>>core teams.
>>>>>>
>>>>>>-Matt Treinish
>>>>>
>>>>>
>>>>>I'd like to move the discussion from what are the roles of the
>>>>>current stable-maint-core and more towards what the benefits would
>>>>>be having a stable-maint team rather than the -core group alone.
>>>>>
>>>>>Personally I think the stable maintenance should be quite a lot more
>>>>>than unblocking gate and approving people allowed to merge to the
>>>>>stable branches.
>>>>>
>>>>
>>>>Sure, but that's not we're talking about here are we? The other
>>>>tasks, like
>>>>backporting changes for example, have been taken on by project teams.
>>>>Even in
>>>>your other email you mentioned that you've been doing backports and
>>>>other tasks
>>>>that you consider stable maint in a glance only context. That's
>>>>something we
>>>>changed in kilo which ttx referenced in [1] to enable that to happen,
>>>>and it was
>>>>the only way to scale things.
>>>>
>>>>The discussion here is about the cross project effort around stable
>>>>branches,
>>>>which by design is a more limited scope now. Right now the cross
>>>>project effort
>>>>around stable branch policy is really 2 things (both of which ttx
>>>>already
>>>>mentioned):
>>>>
>>>>1. Keeping the gates working on the stable branches
>>>>2. Defining and enforcing stable branch policy.
>>>>
>>>>The only lever on #2 is that the global stable-maint-core is the only
>>>>group
>>>>which has add permissions to the per project stable core groups.
>>>>(also the
>>>>stable branch policy wiki, but that rarely changes) We specifically
>>>>shrunk it to
>>>>these 2 things in. [1] Well, really 3 things there, but since we're
>>>>not doing
>>>>integrated stable point releases in the future its now only 2.
>>>>
>>>>This is my whole argument that creating a new team doesn't do
>>>>anything. Living
>>>>under rel-mgt, some other project, or creating a new one isn't going
>>>>to change
>>>>the fact that cross-project stable maint is the same 2 tasks which
>>>>basically
>>>>nobody cares about, which TBH your email is just an indication of.
>>>>Even if we
>>>>wanted to grow to something beyond these 2 tasks they would still be
>>>>the core of
>>>>whatever it becomes and a lack of people caring about them will just
>>>>block any
>>>>potential scope growth.
>>>>
>>>>Frankly, this is the problem with any ML or open discussion about
>>>>stable branch
>>>>maint. Everyone has an opinion about everything but lacks actual
>>>>context or
>>>>never steps up to work on the cross-project side of this. I don't
>>>>think creating
>>>>a new team that has no repos or other artifacts and therefore no
>>>>stackalytics
>>>>credit (and by extension corporate chest thumping) is magically going
>>>>to create
>>>>people actually working on this.
>>>>
>>>>The logical follow on idea to the above is to create a stable branch
>>>>policy doc
>>>>repo and a project team to own that. But, I still don't think that
>>>>solves the
>>>>problem, especially since the real priority issue in this space is #1
>>>>which
>>>>at most is a handful of us bothering to look at that, (well that and
>>>>the stable
>>>>policy barely ever changes) which honestly really isn't the majority
>>>>of the
>>>>stable-maint-core group.  We also have a similar problem with gate
>>>>issues on
>>>>master and it's mostly the same people looking at these things there
>>>>too.
>>>>
>>>>IMHO the only way for creating a separate project team to be useful
>>>>is to
>>>>re-centralize more responsibilities of stable maint, which is
>>>>something I don't
>>>>think we should do because we'll just hit the same scaling issues we
>>>>had before
>>>>kilo which prompted that change in the first place. It's the same
>>>>problem every
>>>>horizontal, cross project, or whatever you want to call it effort has
>>>>with
>>>>OpenStack's growth and why most have moved to the distributed self
>>>>service
>>>>models.
>>>>
>>>>But, if the goal is just to not make Doug responsible for #2, which
>>>>is something
>>>>ttx was primarily doing before, then I guess it's fine but we should
>>>>be honest
>>>>about it and just make ttx the new team leader. :) I honestly don't
>>>>think
>>>>anything else discussed will change by explicitly making it a
>>>>separate team.
>>>>
>>>>-Matt Treinish
>>>>
>>>>>>
>>>>>>>>
>>>>>>>>[1]
>>>>>>>>http://lists.openstack.org/pipermail/openstack-dev/2015-November/078
>>>>>>>>281.html
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>__________________________________________________________________________
>>>>>>>>
>>>>>>>>OpenStack Development Mailing List (not for usage questions)
>>>>>>>>Unsubscribe:
>>>>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>Matty T got all P&V in Tokyo! :)
>>>
>>>I pretty much agree with everything said above. I don't think we make
>>>many policy changes, the stable maint core team is already pretty
>>>decentralized as it is. We rarely seem to communicate much except when
>>>the (1) gate is wedged or (2) we're coming up on a release and need a
>>>push for reviews.
>>
>>And this is perhaps one of the problems. I don't think decentralizing
>>the stable team is enough to have proper stable maintenance. It's a
>>great step forward and I don't think the proposal is to remove that.
>>
>>Just like for the releases (especially for projects following
>>coordinated releases), stable branches could benefit from having
>>someone looking after them (for those projects that apply to the tag
>>ttx mentioned in one of his previous emails) and coordinate releases,
>>status, etc.
>>
>>I think there's more to it than enforcing policies and keeping the
>>gate sane. What about projects w/o backports? Are we encouraging ppl
>>to do enough backports to stable branches?
>>
>>>
>>>The small group of gate thumpers raise hell when (1) comes up, and the
>>>stable branch champion coordinates (2).
>>>
>>>I've asked this in other threads before, but do we need a PTL for
>>>this? I'm assuming ttx doesn't want this since he's asking for
>>>volunteers. Doug doesn't want it because of release mgmt duties. But
>>>what does a stable branch PTL do? If we have policy changes on stable
>>>that need discussing, can't we come together and discuss those as
>>>needed as a group?
>>
>>Here's a couple of things that I'd love to see the volunteer for that
>>job to do:
>>
>>- Help coordinating stable releases when they are needed.
>
>But isn't this why we decentralized the stable release stuff? So you 
>don't need a single person or small team herding all the cats?

Perhaps, but still some coordination is needed. I believe.

>
>>- Help creating tools to monitor existing backports.
>
>I'm not sure a PTL is needed for this. And I don't think a tool is 
>needed, we have gerrit.  With decentralized stable maint to the 
>projects (including releasing), if I'm a nova person that cares about 
>stable (and I am), then I can find out what's open for stable/liberty 
>backports using gerrit:

I meant something that would also help enforcing the policies and what
not. I know what you can do w/ gerrit.

>>- Help creating tools that could help identifying patches that might
>>  be worth backporting.
>
>Again, I'm not sure a PTL is needed for this, just some hackers. I 
>could see value in writing a script that scans launchpad for things 
>like 'liberty-backport-potential' where the fix is merged in mitaka 
>but not proposed to stable/liberty.
>
>https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/liberty,n,z
>
>Easy peasy. We actually have these same dashboards linked into our 
>weekly nova meeting agenda just for awareness.

Again, I know what you can do with gerrit. What I was suggesting is a
script that would help scan launchpads as you mentioned.

I'm not saying you need a PTL to work on that but, we've seen
(especially with the growth of the number of liaisons) that someone
that can act as a representative is, most of the time, helpful. :)

>
>>- Be a contact for whenever things break in stable branches and
>>  coordinate with folks that can help fixing those issues.
>
>We have the #openstack-stable channel, people can ask for help there. 
>We have the mailing list where stable blocking issues are regularly 
>brought forward to the wider audience. I guess being PTL would mean 
>there is a lightning rod when something goes bad, but I'd think the 
>majority of the time they are going to defer to the PTL of whatever 
>project is blocked and get them to work on it, i.e. if trove 
>stable/liberty is blocked b/c of some issue in their own CI tooling.

Again, it's not a matter of support but coordination, tooling and
enforcement.

Flavio

>
>>
>>Not sure if all the above points are worth it but I do see this as an
>>useful role to have.
>>
>>Cheers,
>>Flavio
>>
>>>And for gate issues, the people that generally work on those are doing
>>>it because it benefits them, either because it keeps them from having
>>>to maintain forks out of tree or because it gets other changes landed
>>>that are needed for their product/customers. I don't see a new team
>>>making a difference in attracting new people to really dig into the
>>>ugly job of unwedging the gate on stable - it takes quite a bit of
>>>time to get up to speed. Tony is a great example of someone that has
>>>kind of come out of left field and said he wants to help with this and
>>>has stuck with it long enough that he makes a difference and it's
>>>worth spending time mentoring him on the stable gate issues, but those
>>>people are far and few between.
>>>
>>>--
>>>
>>>Thanks,
>>>
>>>Matt Riedemann
>>>
>>>
>>>__________________________________________________________________________
>>>
>>>OpenStack Development Mailing List (not for usage questions)
>>>Unsubscribe:
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>__________________________________________________________________________
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>-- 
>
>Thanks,
>
>Matt Riedemann
>
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151112/c05cc2e7/attachment.pgp>