Open Stack

On Wed, 22 Aug 2018 09:49:13 -0400, Doug Hellmann wrote:
> Excerpts from melanie witt's message of 2018-08-21 15:05:00 -0700:
>> On Tue, 21 Aug 2018 16:41:11 -0400, Doug Hellmann wrote:
>>> Excerpts from melanie witt's message of 2018-08-21 12:53:43 -0700:
>>>> On Tue, 21 Aug 2018 06:50:56 -0500, Matt Riedemann wrote:
>>>>> At this point, I think we're at:
>>>>>
>>>>> 1. Should placement be extracted into it's own git repo in Stein while
>>>>> nova still has known major issues which will have dependencies on
>>>>> placement changes, mainly modeling affinity?
>>>>>
>>>>> 2. If we extract, does it go under compute governance or a new project
>>>>> with a new PTL.
>>>>>
>>>>> As I've said, I personally believe that unless we have concrete plans
>>>>> for the big items in #1, we shouldn't hold up the extraction. We said in
>>>>> Dublin we wouldn't extract to a new git repo in Rocky but we'd work up
>>>>> to that point so we could do it in Stein, so this shouldn't surprise
>>>>> anyone. The actual code extraction and re-packaging and all that is
>>>>> going to be the biggest technical issue with all of this, and will
>>>>> likely take all of stein to complete it after all the bugs are shaken out.
>>>>>
>>>>> For #2, I think for now, in the interim, while we deal with the
>>>>> technical headache of the code extraction itself, it's best to leave the
>>>>> new repo under compute governance so the existing team is intact and we
>>>>> don't conflate the people issue with the technical issue at the same
>>>>> time. Get the hard technical part done first, and then we can move it
>>>>> out of compute governance. Once it's in its own git repo, we can change
>>>>> the core team as needed but I think it should be initialized with
>>>>> existing nova-core.
>>>>
>>>> I'm in support of extracting placement into its own git repo because
>>>> Chris has done a lot of work to reduce dependencies in placement and
>>>> moving it into its own repo would help in not having to keep chasing
>>>> that. As has been said before, I think all of us agree that placement
>>>> should be separate as an end goal. The question is when to fully
>>>> separate it from governance.
>>>>
>>>> It's true that we don't have concrete plans for affinity modeling and
>>>> shared storage modeling. But I think we do have concrete plans for vGPU
>>>> enhancements (being able to have different vGPU types on one compute
>>>> host and adding support for traits). vGPU support is an important and
>>>> highly sought after feature for operators and users, as we witnessed at
>>>> the last Summit in Vancouver. vGPU support is currently using a flat
>>>> resource provider structure that needs to be migrated to nested in order
>>>> to do the enhancement work, and that's how the reshaper work came about.
>>>> (Reshaper work will migrate a flat resource provider structure to a
>>>> nested one.)
>>>>
>>>> We have the nested resource provider support in placement but we need to
>>>> integrate the Nova side, leveraging the reshaper code. The reshaper code
>>>> is still going through code review, then next we have the integration to
>>>> do. I think things are bound to break when we integrate it, just because
>>>> nothing is ever perfect, as much as we scrutinize it and the real test
>>>> is when we start using it for real. I think going through this
>>>> integration would be best done *before* extraction to a new repo. But
>>>> given that there is never a "good" time to extract something to a new
>>>> repo, I am OK with the idea of doing the extraction first, if that is
>>>> what most people want to do.
>>>>
>>>> What I'm concerned about on the governance piece is how things look as
>>>> far as project priorities between the two projects if they are split.
>>>> Affinity modeling and shared storage support are compute features
>>>> OpenStack operators and users need. Operators need affinity modeling in
>>>> the placement is needed to achieve parity for affinity scheduling with
>>>> multiple cells. That means, affinity scheduling in Nova with multiple
>>>> cells is susceptible to races and does *not* work as well as the
>>>> previous single cell support. Shared storage support is something
>>>> operators have badly needed for years now and was envisioned to be
>>>> solved with placement.
>>>>
>>>> Given all of that, I'm not seeing how *now* is a good time to separate
>>>> the placement project under separate governance with separate goals and
>>>> priorities. If operators need things for compute, that are well-known
>>>> and that placement was created to solve, how will placement have a
>>>> shared interest in solving compute problems, if it is not part of the
>>>> compute project?
>>>>
>>>
>>> Who are candidates to be members of a review team for the placement
>>> repository after the code is moved out of openstack/nova?
>>>
>>> How many of them are also members of the nova-core team?
>>
>> I assume you pose this question in the proposed situation I described
>> where placement is a repo under compute. I expect the review team to be
> 
> No, not at all. I'm trying to understand how you think a completely
> separate team is going to cause problems. Because it seems like at
> least a large portion, if not all, of the contributors want it, and
> I need to have a very good reason for denying their request, if we
> do. Right now, I understand that there are concerns, but I don't
> understand why.

I have been trying to explain why over several replies to this thread. 
Fracturing a group is not something anyone does to foster cooperation 
and shared priorities and goals. It is my job and responsibility to 
ensure that features that operators and users need, get delivered to 
them. We have a list of specific features that we know operators and 
users need, right now. And fracturing the group is something that will 
negatively impact our ability to implement them. Working farther apart 
will negatively impact our ability to coordinate work that is closely 
coupled. The last thing I want to do right now is build walls. I do not 
want there to be separate goal-setting and priorities discussions -- I 
want us to work together, coordinating as one group.

At the very least, separating the groups is something to be taken very 
seriously and is not reversible. I want to be very sure it will be 
better for operators and users relying on us, if the groups were 
separated. I haven't heard any reasons why it would be, yet.

Thus, I would like to see an incremental approach, given the outstanding 
work items we know we have. I would like to see us make changes and 
*try* to create an environment where people can work together in one 
group, before we go to the extreme option and separate everything 
completely. I would like to extract the code into its own repo under 
compute, with the expectation that it is to evolve *independently* of 
Nova code, and with a subset of nova-core, plus a new placement-core 
team with placement experts added to it. And we see how that goes. Then, 
we reassess. If everyone gives that an earnest try, and it is not 
working for people, then we will know for sure that we tried to make 
changes and stay together as one group, and it didn't work, and 
separating into two groups in the middle of the work items we have, is 
what we have to do.

>>> What do you think those folks are more interested in working on than the
>>> things you listed as needing to be done to support the nova use cases?
>>
>> I'm not thinking of anything specific here. At a high level, I don't see
>> how separating into two separate groups under separate leadership helps
>> us deliver the listed things for operators and users. I tend to think
>> that a unified group will be more successful at that.
> 
> OK. At the same time, I'm trying to understand why you have a hard
> time believing a new team's priorities would not be aligned with
> the nova team's priorities if, as it seems, a large percentage of
> that new team would be made up of the same exact people.

I think it's about context. If two separate projects do their own 
priority and goal setting, separately, I think they will naturally be 
more different than they would be if they were one project. Currently, 
we agree on goals and priorities together, in the compute context. If 
placement has its own separate context, the priority setting and goal 
planning will be done in the context of placement. In two separate 
groups, someone who is a member of both the Nova and Placement teams 
would have to persuade Placement-only members to agree to prioritize a 
particular item. This may sound subtle, but it's a notable difference in 
how things work when it's one team vs two separate teams. I think having 
shared context and alignment, at this point in time, when we have 
outstanding closely coupled nova/placement work to do, is critical in 
delivering for operators and users who are depending on us.

>>> What can they do to reassure you that they will work on the items
>>> nova needs, regardless of the governance structure?
>>
>> If they were separate groups, I don't see why the leadership of
>> placement would necessarily share goals and priorities with compute. I
>> think that is why it's much more difficult to get things done with two
>> separate groups, in general.
> 
> Most of the teams in the community seem to have a relatively easy time
> coming to common agreement on priorities and goals, even when they are
> not so closely related as the nova and placement teams would be. So I
> guess I still don't see what the problem would be, and am looking for
> more details, either about the concerns you all have or ways to
> alleviate them.

The "relatively easy time" sounds a bit vague to me. My experience with 
cross-project work (multi-attach) has been that someone has to serve as 
a champion of a feature or effort, and persuade other team to care about 
it and work together on it. If the team is sufficiently persuaded, they 
agree to collaborate on a thing. Then, they go into their silos and do 
some work on it. Then, the champion checks in on the teams after some 
time passes. The champion gathers problems and blockers and sets up a 
meeting between the two teams or takes the messages back and forth 
themselves. The teams do some more work in their silos. The champion 
checks in again after some time, setting up a meeting to re-synchronize 
the teams. And this continues until the feature is complete, if the 
champion has been around that long.

This can work, but it's a much different experience than being one team. 
Being that we are still in the thick of it, and need to work on the 
items I have mentioned before in my other replies, I don't think *now* 
is a good time to separate groups. Separating now will hurt operators, 
users, and customers. Separating now will make already challenging work 
even more challenging, with more barriers to climb over and more effort 
to stay connected. That is why I'm concerned.

>> I want to reiterate again that the only thing I care about here is
>> delivering functionality that operators and users need. vGPUs, in
>> particular, has been highly sought after at a community-wide level, not
>> just from the compute community. I want to deliver the features that
>> people are depending on and IMHO, being a unified group helps that. I
>> don't see how being two separate groups helps that.
> 
> I don't think doing those things is mutually exclusive with solving,
> or at least addressing, the underlying trust and self-governance
> issues here, though. In fact, I think dealing with *that* is going
> to make us all more effective at delivering the software, in the
> long term because we will have cleared up what the expectations
> between the two teams is.

To be clear, there are not underlying trust issues with most of the 
team. A trust issue was expressed by one member of the team and I think 
it's unfair to apply it to everyone else.

Aside from that, it has always been difficult to add folks to nova-core 
because of the large scope and expertise needed to approve code across 
all of Nova. Extracting the placement code into its own repo alleviates 
the problem of the massive scope and gives us the ability to have a 
placement-core team. And I expect the placement repo to evolve 
independently of Nova code, as it's been intended to be a separate 
project, eventually.

My hope is that making these changes will improve the experience in the 
placement subteam and compute as a whole. I hope that the idea of a 
compromise to try it out will be amenable to everyone. I don't take the 
idea of separating the groups completely, at this point in time, with 
outstanding closely coupled nova/placement work, lightly. I think it 
should be taken very seriously and done with care. And taking a step, 
trying it out, and then reassessing, seems like the most prudent way to 
go about it, IMO.

Best,
-melanie