Open Stack

On 11/10/2016 6:15 AM, Chris Dent wrote:
> On Tue, 8 Nov 2016, Chris Friesen wrote:
>
>> That said, I don't know that there's an easy solution to this in nova
>> due to the fact that it's a distributed system with a central shared
>> data store. Enhancing a sched filter might require new data, which
>> means modifying the DB model and the DB migrations, and updating the
>> InstanceNUMATopology object and tweaking nova-api to request
>> additional attrs, and it might have implications for upgrade, etc.
>
> The symptoms that you are describing here are pretty much the crux
> of the biscuit.

I really hesitate to even respond to this but I feel the need to, but at 
the same time feel as though I'm going to regret it, and it won't help 
change anyone's mind, and I'll probably get in trouble from the 
Foundation or TC or some other group that monitors what PTLs/cores/etc 
say in public. But here goes!

>
> As much as I think having additional cores or subsystem maintainers
> with +2 but not +W would probably be useful, I don't think it will
> do much to fix the fundamental problem -- the crux above -- so the
> concerns that people have about the risks will continue, will continue
> to be valid and can't just be dismissed.
>
> That's a technology crux. There's also a social crux.
>
> It was very clear in BCN that there is significant disagreement in
> the community over the volume of code that nova ought to be accepting.
> Even in an ideal world where a lot of the roadblocks that people are
> experiencing aren't there, there would be disagreement between at least
> two (idealized, not literal) camps: One that says any code which doesn't
> break stuff and is vaguely in the domain of what nova is about should be
> merged and another that says nova should have a clear and focused
> vision and contributions that don't fit that vision should be rejected.

Believe it or not, those of us that say no to things aren't super happy 
saying no to people all of the time. It gets pretty soul-crushing after 
awhile. However, for those of us that have worked on Nova for several 
years and have seen where saying, 'sure, what the hell, this magical 
little vendor snowflake can't possibly harm us, right? btw, don't worry 
about maintaining this or CI testing it...we'll trust you to do that out 
of tree and report/fix those problems 4 years from now' - that creates 
an innate hesitation to more and more magical unicorn vendor-specific 
changes. Because it's those kinds of things that get in the way of us 
being able to make broader necessary changes later, like with the 
placement service/scheduler stuff, and capabilities in the API. With so 
much random custom use case code, it's really hard to know what changes 
will break something else you thought was unrelated.

Also, we have consistently had operators give feedback at summits that 
basically say, 'nova is the most boring project in my deployment, and 
that's a good thing' - that's about the best compliment we can get: 
stability matters.

And for that matter, the OpenStack community has said that 
interoperability matters, and as most have seen we've been working on 
removing plugin and extension points over the last several releases to 
get to that end. Having said that, it's an entirely whole other topic 
unto itself and I have mixed feelings about it personally, i.e. if I'm 
building a private devops cloud and don't care about 
defcore/refstack/interoperability needed for marketing a product, then 
what's wrong with me having configuration and extension point freedom 
coming out of my ears? That's a valid point, I understand that. I also 
understand that we don't test any of that stuff upstream and we break 
it, and then get people shouting at us for breaking things that broke 
their snowflake configuration.

>
> At the moment both of these cruxes seem to be treated as things that
> are too hard to change, or at least too hard discuss.

Chris, I keep hearing this, especially from you. I don't know what needs 
to happen to move past this. We talked about this at length at the BCN 
summit. We talked about new contributors and ways to help there at the 
ATX summit. So it's not like no one is willing to discuss these things - 
we obviously have already. But saying things like 'no one is willing to 
even discuss this' is really frustrating to continue hearing, and what's 
even more frustrating is I get the feeling that if I even mention that, 
it's going to somehow justify the point. "See, he said he's tired of 
talking about this, which means he doesn't want to talk about it."

>
> On the tech crux some people claim that the decomposition (and the
> related boundary hardening) that would lead to safe contracts between
> subsystems have already gone as far as they can or taking them
> further requires such a huge amount of effort that given resources
> and other commitments it can't happen. That smacks of a lack of
> imagination and hope.

I also don't agree with this. I think the sentiment at the summit, if 
we're talking about the same thing, was that we've done a lot of 
boundary hardening already. There is obviously room for improvement - 
but to get there, you're going to have to wade through a LOT of custom 
code added by vendors in years past and make sure not to break that. 
That's totally fine as a goal, but are you personally going to take up 
that challenge and start working on it? If not, then I care less about 
hearing about it over and over and continuing to talk about this. We 
have priorities each release, and we have several cores working on those 
priorities, and with the HPE and Mirantis layoffs we have even fewer 
dedicated people able to work on some of these bigger sweeping changes - 
and those people had the background on what dragons lay in wait when you 
take on some of these big systematic changes. So the point is we have to 
prioritize, and I put out of tree enablement less of a priority compared 
to the other things we need to get done that operators actually care about.

My main point here is, if you see the opportunity and justification for 
the work to completely change something, then by all means, work on that 
and keep everyone involved. But I'm tired of hearing how unwilling 
people in nova, and by people of course I mean the core team, are to 
listening to ideas for positive change and then being thrown under the 
bus when we don't have all of the time or priority to nurture all of 
those changes.

>
> On the social crux, the impression I got is that people are either
> too exhausted, too busy, too angry, or too scared to actually talk
> things out and reach a conclusion and instead bitch behind each other's
> backs about the lack of agreement and some other core who will either
> reject or accept code they shouldn't be. We need to be honest about
> this lack of trust if we want to make it better. And we need resolve
> the vision of the project, in a concrete fashion.

What's really baffling to me is when I'm in the hallway with you and 
Matt Booth and others talking about this stuff face to face in person, 
and explaining my side of the issue, and I'm listening to yours, we seem 
to all be nodding our heads and agreeing, or at least understanding a 
bit better. But then when we get back to the world it's all us vs them 
and 'a lot of people are saying nova sucks and they are ready to give up'.

First, the 'speaker of the people' thing gets tiring. We're all working 
on the same thing. If people have issues I welcome them to air it out, 
but they need to listen to and understand the reply, and try to 
understand why we have to say no to things at times, or that we can't 
prioritize something.

Second, at least from my view, I get the feeling that there are a few 
people beating this drum and complaining about not getting enough 
attention and we need to stop everything we're doing whenever they say 
something and I just have a hard time with that. What I value most in 
nova contributors are people that are doing the dirty work, like 
triaging and fixing bugs, fixing technical debt, trying to take on the 
dirtier issues to help us move forward, and probably most of all - lots 
of quality code reviews. At the end of the day quality code reviews are 
going to make you core, and that's what's going to move code through the 
system. But people would rather complain about not being core, or the 
lack of cores, than actually bust their ass to help out and make core.

>
> Until at least one of those cruxes is resolved it's going to be
> really hard to make substantial progress and many of the strategies
> we try are going to be tactical band-aids.
>
> mdbooth's proposal is somewhat different from some of the other
> options because implementing it requires a leap of faith over
> both cruxes into what could be a self-reinforcing environment: we will
> trust one another because we've said we will, and we will build the
> stronger contracts because we need them in order to be able to trust
> people, and we will talk about it all (better than we do now) because it
> won't work if we don't.
>
> That might work. It could be a mess, but it is better than an
> alternative which is feeling increasingly likely: People go elsewhere.

And finally, the hostage taking approach of everyone going elsewhere is 
really wearing on me. If developers have a choice of projects to work on 
in OpenStack and for whatever reason find it impossible to work in Nova, 
then I frankly think they should move on, for the benefit of both parties.

You know other reasons that people are going to go elsewhere? Because 
their company took them off OpenStack, or fired them, because they were 
working on OpenStack and they decided as a technology it didn't work 
well enough for them and they are going to invest resources elsewhere. 
That's why I find it a major distraction to be talking about all of the 
ideals of how things would work when instead we have a ton of work to 
already get done and we're understaffed, and the future is unsure 
regardless - so while I'm here I want to get the major changes in that 
we've been working on for years as a project, like cells v2, placement, 
and moving off nova-network.

>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

There, now I feel equally terrible and relieved. I really don't want to 
be anyone's barrier, and I feel great when I'm able to help someone out, 
new contributor or not. I don't really know what else to say. I know all 
of this probably doesn't mean anything and there will be a massive reply 
thread discounting each point and exquisite detail about how I'm wrong - 
that's fine, it's what I get for taking the bait to respond in detail to 
something like this. But I felt the need to do so.

-- 

Thanks,

Matt Riedemann