Open Stack

On Aug 26, 2014, at 2:01 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:

> 
> 
> 
> On Wed, Aug 20, 2014 at 2:25 AM, Eoghan Glynn <eglynn at redhat.com> wrote:
> 
> 
> > > > Additional cross-project resources can be ponied up by the large
> > > > contributor companies, and existing cross-project resources are not
> > > > necessarily divertable on command.
> > >
> > > Sure additional cross-project resources can and need to be ponied up, but I
> > > am doubtful that will be enough.
> >
> > OK, so what exactly do you suspect wouldn't be enough, for what
> > exactly?
> >
> >
> > I am not sure what would be enough to get OpenStack back in a position where
> > more developers/users are happier with the current state of affairs. Which
> > is why I think we may want to try several things.
> >
> >
> >
> > Is it the likely number of such new resources, or the level of domain-
> > expertise that they can be realistically be expected bring to the
> > table, or the period of time to on-board them, or something else?
> >
> >
> > Yes, all of the above.
> 
> Hi Joe,
> 
> In coming to that conclusion, have you thought about and explicitly
> rejected all of the approaches that have been mooted to mitigate
> those concerns? 
> 
> Is there a strong reason why the following non-exhaustive list
> would all be doomed to failure:
> 
>  * encouraging projects to follow the successful Sahara model,
>    where one core contributor also made a large contribution to
>    a cross-project effort (in this case infra, but could be QA
>    or docs or release management or stable-maint ... etc)
> 
>    [this could be seen as essentially offsetting the cost of
>     that additional project drawing from the cross-project well]
> 
>  * assigning liaisons from each project to *each* of the cross-
>    project efforts
> 
>    [this could be augmented/accelerated with one of the standard
>     on-boarding approaches, such as a designated mentor for the
>     liaison or even an immersive period of secondment]
> 
>  * applying back-pressure via the board representation to make
>    it more likely that the appropriate number of net-new
>    cross-project resources are forthcoming
> 
>    [c.f. Stef's "we're not amateurs or volunteers" mail earlier
>     on this thread]
> 
> All of these are good ideas and I think we should try them. I am just afraid this won't be enough.
> 
> Imagine for a second, that the gate is is always stable, and none of the existing cross project efforts are short staffed. OpenStack would still has a pretty poor user experience and return errors in production. Our 'official' CLIs are poor, our logs are cryptic, we have scaling issues (by number of nodes), people are concerned about operational readiness [0], upgrades are very painful, etc. Solving the issue of scaling cross project efforts is not enough, we still have to solve a whole slew of usability issues. 

These are indeed problems, and AFAICT, we don’t really have a structure in place to solve some of them directly. There’s the unified CLI project, which is making good progress. The SDK project started this cycle as well. I don’t have the impression, though, that either of those has quite the traction we need to fully replace the in-project versions, yet. Sean has some notes for making logging better, but I don’t think there’s a team working on those changes yet either.

The challenge with most of these cross-project initiatives is that they need every project to contribute resources, at least in the form of reviews if not code, but every project also has its own priorities. Would the situation be improved if we had a more formal way for the TC to say to projects, “this cycle we need you to dedicate resources to work on X with this cross-project team, even if that means deprioritizing something else”, similar to what we’ve done recently with the gap analysis?

Doug

> 
> [0] http://robhirschfeld.com/2014/08/04/oscon-report/
> 
>  
> 
> I really think we need to do better than dismissing out-of-hand
> the idea of beefing up the cross-project efforts. If it won't
> work for specific reasons, let's get those reasons out onto
> the table and make a data-driven decision on this.
> 
> > And which cross-project concern do you think is most strained by the
> > current set of projects in the integrated release? Is it:
> >
> > * QA
> > * infra
> > * release management
> > * oslo
> > * documentation
> > * stable-maint
> >
> > or something else?
> >
> >
> > Good question.
> >
> > IMHO QA, Infra and release management are probably the most strained.
> 
> OK, well let's brain-storm on how some of those efforts could
> potentially be made more scalable.
> 
> Should we for example start to look at release management as a
> program onto itself, with a PTL *and* a group of cores to divide
> and conquer the load?
> 
> (the hands-on rel mgmt for the juno-2 milestone, for example, was
>  delegated - is there a good reason why such delegation wouldn't
>  work as a matter of course?)
> 
> Should QA programs such as grenade be actively seeking new cores to
> spread the workload?
> 
> (until recently, this had the effective minimum of 2 cores, despite
>  now being a requirement for integrated projects)
> 
> Could the infra group potentially delegate some of the workload onto
> the distro folks?
> 
> (given that it's strongly in their interest to have their distro
>  represented in the CI gate.
> 
> None of the above ideas may make sense, but it doesn't feel like
> every avenue has been explored here. I for one don't feel entirely
> satisfied that every potential solution to cross-project strain was
> fully thought-out in advance of the de-integration being presented
> as the solution.
> 
> Just my $0.02 ...
> 
> Cheers,
> Eoghan
> 
> [on vacation with limited connectivity]
> 
> > But I also think there is something missing from this list. Many of the projects
> > are hitting similar issues and end up solving them in different ways, which
> > just leads to more confusion for the end user. Today we have a decent model
> > for rolling out cross-project libraries (Oslo) but we don't have a good way
> > of having broader cross project discussions such as: API standards (such as
> > discoverability of features), logging standards, aligning on concepts
> > (different projects have different terms and concepts for scaling and
> > isolating failure domains), and an overall better user experience. So I
> > think we have a whole class of cross project issues that we have not even
> > begun addressing.
> >
> >
> >
> > Each of those teams has quite different prerequisite skill-sets, and
> > the on-ramp for someone jumping in seeking to make a positive impact
> > will vary from team to team.
> >
> > Different approaches have been tried on different teams, ranging from
> > dedicated project-liaisons (Oslo) to shared cores (Sahara/Infra) to
> > newly assigned dedicated resources (QA/Infra). Which of these models
> > might work in your opinion? Which are doomed to failure, and why?
> >
> > So can you be more specific here on why you think adding more cross-
> > project resources won't be enough to address an identified shortage
> > of cross-project resources, while de-integrating projects would be?
> >
> > And, please, can we put the proverbial strawman back in its box on
> > this thread? It's all well and good as a polemic device, but doesn't
> > really move the discussion forward in a constructive way, IMO.
> >
> > Thanks,
> > Eoghan
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev