Open Stack

Hi Eugene,

Apologies for the delay in my response. It took me a while to figure out
how to say what I wanted to say here as diplomatically as possible. In the
end, I decided I couldn't be diplomatic about some of the things that I
think needed to be said. I don't play games of politics very well at the
best of times, so I do apologize that I will undoubtedly offend some of you
with what I say in this e-mail.

So just to get this off my chest, here's the elephant in the room I think
needs to be addressed up front:

Prior to last Thursday morning's IRC meeting, for the last couple of months
we've been having a spirited and productive discussion around both user
requirements, operator requirements, use cases, definitions of terms,
product roadmap and api and object model revisions all having to do with
Neutron LBaaS. These discussions have mostly been on the mailing list, with
supplemental materials created on the OpenStack wiki, on google documents,
and google spreadsheets. On occasion, references to blueprint documents
have been made, but little discussion of these blueprints has been
happening, and certainly not among the people on the mailing list. (Most of
the stuff in the blueprints is just referencing mailing list discussions,
or gerrit "discussions" where the only things commenting are CI systems
anyway.) Further, the IRC meetings have usually been too short to get much
done, and sub-project leadership has specifically been encouraging
follow-up discussions to happen on the mailing list.

A lot of good things have come from the discussion including, as far as I
can tell, the first-ever actual feature usage data survey from operators'
user bases. (As far as I can tell prior to this, decisions about product
direction prior to this were being made based on what "someone thinks we'll
need" instead of what end-users are asking for and operators are actually
doing.) I believe part of the reason we've seen the amount of enthusiasm we
have, as well as the re-involvement of organizations that had previously
"gone off to do their own thing" as far as load balancing is concerned is
*because* this discussion has been no-holds barred, no suggestion is off
the table, and no opinion unheard. Ideas have stood on their own merit.

As of Thursday morning, however, I suddenly see a huge re-emphasis on
moving back to using the blueprint system, using gerrit, and otherwise
forcing this discussion back into the usual "Neutron workflow" (which is
neither friendly to newcomers, nor, as far as I can tell, really very
useful for discussing potentially sweeping changes--  incremental changes,
sure, but what if we really need to rebuild a big chunk of this from the
ground up? How does that discussion happen in such a granular system? What
is the point of having code-level discussions when the higher level design
is what is off?)

Further, I'm detecting an air both in that IRC meeting and in your response
to my API proposal below that certain things are "off the table" for
discussion. Specifically, I'm detecting pressure from, well, you
apparently, that features like L7 and SSL were decided upon months ago,
that work is already well underway on these features, and therefore the
window has already closed on changing direction for Juno. (Nevermind the
fact that we've only seen serious participation from so many cloud
providers in the last 4 weeks, and nevermind the fact that the usage survey
shows that development priorities for this project are clearly off.)

Granted, I understand that a system for coordinating so many contributors
(blueprint and gerrit) is necessary to make forward progress. What I have a
problem with is the sudden and shocking nature with which this
participation is now apparently being forced as an attempt to stifle the
discussion that has been going on (and which, I think, has been enormously
productive). In contemplating "why now?" and "why like this?" for the
timing of this, I can only make two possible guesses as to the motivation:

1. I'm guessing Neutron LBaaS is suddenly getting pressure from Neutron
core to "get people in line," or none of the changes we're proposing will
be approved. And if Neutron LBaaS needs the cooperation of Neutron core in
order to get features pushed through, then the LBaaS sub-project is stuck
in the mud at the whim of Neutron core leadership. Comments to the effect
of "we're generating so much documentation and material so quickly that
nobody in the main neutron project can keep up" as made in the IRC meeting,
and the fact that we've had many object model proposals rejected by Neutron
core with rather vague justification seem to lend credibility to this guess.

The only thing I'll point out here is that if you try to put the brakes on
this literally right after we suddenly have the willing and enthusiastic
participation of so many big players in this space, what makes you think
they won't take displeasure at this dictatorial approach to sub project
leadership and go off to do their own thing again? (Thus robbing this sub
project of ideas, real-world data, development and testing resources, and
well, knocking the wind out of its sails.)

I don't expect Neutron core developers to necessarily have to keep up with
what is going on with all the nuances of discussion around the LBaaS
sub-project, leastwise not until we're asking for changes which affect
Neutron core itself. I suppose it's inevitable that once we know the
direction we're going with the API and object model changes, that this
discussion would move into blueprints and garret. If saying that we've
reached that point is all Neutron LBaaS leadership were trying to convey on
Thursday morning... sheesh did that message come across wrong! (And is
premature, I would say-- because we certainly aren't in agreement as to
what the new API and corresponding object model should be! That's literally
what we're trying to decide right now!)

2. I get the impression that perhaps those who have been most involved in
coding Neutron LBaaS up until now are sensing that:
A. Based on some of the proposals being made, the feature requirements that
are actually important to users and operators alike, and the "nothing is
off the table" approach to the discussion until now... they are starting to
realize that a large portion of Neutron LBaaS might need to be rewritten
from the ground up.
B. With so many new and powerful voices being heard, it's becoming apparent
that this project probably won't be coded by a small, somewhat insular
group anymore.

To this I would say:
1. We actually do appreciate the work that has been done to make Neutron
LBaaS what it is today. We wouldn't be having these discussions if it
weren't for your hard work!
2. The danger of writing code for an idea before requirements and use cases
are thoroughly understood is that you may end up coding in the wrong
direction. And even when you do a good job planning these things, you're
still often going to be scrapping large chunks of code that was written and
tested when new requirements and new technologies emerge. This is the
nature of software development when you're breaking new ground. It is
impossible to get everything right the first time.
3. ALL of the organizations involved in this discussion in the last month
have their own load balancing solutions in place, and ALL of these
solutions are going to need to have large portions of them re-written in
order to work with whatever model we ultimately decide upon in this group.
So it's not like you're the only ones who have to look back on years' worth
of work and live with the fact that you're going to have to scrap or
rewrite large chunks of it from the ground up.
4. It's important to keep lines of communication open for all
participants-- yes, people need to justify their opinions and show good
reasons for objections when work is already underway. But you ignore the
guy shouting about the iceberg off the starboard bow at your own peril.
5. Transparency in communication is important to the health of open source
projects like this. And that fact doesn't change, if suddenly 3/4ths of the
developers working on the project are from one new organization. :/

So, having said this, hopefully its apparent where some of my further
comments on this review of my API proposal come from.

Comments in-line:

On Fri, Apr 25, 2014 at 4:03 AM, Eugene Nikanorov
<enikanorov at mirantis.com>wrote:

> Hi Stephen,
>
> Thanks for the great document. As I promised, I'll try to make a few
> action items out if it.
> First of all, I'd like to say that the API you have proposed is very close
> to what is proposed in the blueprint
> https://review.openstack.org/#/c/89903/ with several differences I'd like
> to address here and make them action items.
>

Ok, the above blueprint was uploaded on April 23rd, literally the day I
sent my API proposal to the mailing list. And this was after it was known
via the action items in the previous week's IRC meeting that my team and I
would be working hard over the next week to produce an API proposal
revision, based on the excellent work of the Rackspace team from the
previous week.

I can understand having a "plan B" in case I missed a deadline there (after
all, I wasn't all that confident we'd get it done in time, but I worked
through the weekend and pulled some very long days to get it done)-- but
it's pretty offensive to me to realize that the action items from the
previous week's meeting apparently weren't being taken seriously. That not
only was there apparently never an intent to seriously consider my
proposal, but now, if I want to be heard, I'm going to have to familiarize
myself with your proposal and fight tooth and nail to be heard to fix any
brokenness I will almost certainly find in it. And I have to do this with
very little confidence that my voice is going to be heard.

Gee. Thanks.


> So, first of all, I think that API described in the doc seems to account
> for all cases we had in mind, i didn't check on case-by-cases basis, it's
> just a first glance impression looking at REST resource set and their
> attributes.
>

As an aside, let's us please, please, please get these use cases out of
being "in mind" and into a unified, digestible, revision-controlled
document. I guarantee you haven't thought of some of the use cases I have
in mind.


> General idea of the whole API/obj model improvement is that we create a
> baseline for all advanced features/usecases that we have in the roadmap.
> Which means that those features then can be added incrementally.
> Incrementally means that resources or attributes might be added, but
> workflow remains and backward compatibility is preserved. That was not the
> case with multiple listeners and L7.
>

I would argue that that wasn't the case for SSL either in the old model.
And even the model I proposed doesn't address HA yet. It's assumed that the
object model proposed won't affect the ability of vendors to implement HA
in whatever way makes sense for their organization... but we don't really
know that until we see some rough models on that front.

Anyway, I think the point remains that the whole reason for the API and
object model revision was to be more forward thinking, specifically about
those features that are currently sorely lacking from Neutron LBaaS, and
which both users and operators clearly need before the solution can be
deployed on a wide scale.

And yes, of course this is a design that will necessarily be implemented
incrementally! First priority should be to deliver at least the same level
of functionality that the old api / object model delivers. But, again, I
think it's a really good idea to be forward thinking about these sorts of
things. If I'm planning a road trip from Denver to New York, I like to
consider at a high level at least what kind of route will take me there (so
I can be sure that I don't realize 500 miles into the trip that I've been
driving toward L.A. the whole time).


> So, a couple on general comments:
>
> 1. Whole discussion about API/obj model improvement had the goal of
> allowing multiple pools and multiple listeners.
>

This is news to me. If we aren't going to be thinking about SSL and L7
also, at least, then there really was no reason to propose drastic changes
to the API / object model. And just what have we been discussing on this
mailing list and in the IRC meetings for the last couple months? Again, I
think it's a little unfair to criticize the API proposal I've made because
it delivers features that people have been asking for for a long time (and
that are already present, functional, and well understood in other load
balancer implementations.)

Also, please understand that "multiple pools" is L7! Can anyone name a use
case where multiple back-end pools are used on a single listener that
doesn't involve some L7 logic in there somewhere?


> For that purpose loadbalancer instance might be an extra. The good thing
> about 'loadbalancer' is that it can be introduced in the API in incremental
> way.
>

Right. You might also notice that in the diagram, it's grayed out and there
are notes indicating that work needs to be done to flesh out the concept
based on discussion around HA functionality and operator concerns. Again,
this is an attempt by me to be more forward-thinking.

There is only one attribute (of a VIP) that references the "load balancer"
concept as defined in the glossary (also linked). Yes, it can be removed
from this API proposal and added back in again later when / if the
discussion around HA actually happens.


> So, VIP+listeners itself is already quite flexible construct (where VIP is
> a root object playing loadbalancer role) that addresses our immediate needs.
> So I'd like to extract loadbalancer API and corresponding use cases in
> another blueprint.
> You know that loadbalancer concept has raised very heated discussions so
> it makes sense to continue discussing it separately, keeping in mind that
> introducing loadbalancer is not very complex and it may be done on top of
> the VIPs/listeners API
>

Also, I notice this is a pretty stark 180-degree turn around on your
position on this from previous weeks' discussions. It's OK to change one's
mind on these things, but again, this leads me to believe that this is
being forced from outside of the Neutron LBaaS sub-project.


> 2. SSL-related objects. SSL is rather big deal, both from API and object
> model, it was a separate blueprint in Icehouse and i think it makes sense
> to work separately on it.
> What I mean is that ssl don't affect core API (VIPs/listeners/pools) other
> than adding some attributes to listeners.
>

Ok, again, from my perspective this is a stark change from how this has
been discussed in previous weeks on this mailing list and in the IRC
meetings. I thought it hadn't been decided exactly how SSL was going to be
done yet. (There was very little discussion of this on the mailing list,
and certainly nothing conclusive. And even the blueprint references a
gerrit patch "discussion" that essentially is just CI systems checking in.
In fact the only actual design discussion of this in months appears to have
been happening exclusively on the mailing list:
http://stackalytics.com/report/blueprint/neutron/lbaas-ssl-termination )

If you want to claim that "silence implies consent" with the design, this
seems disingenuous to me when much more broad design discussions (which
ultimately affect the SSL design) are happening at the same time. It's like
arguing about how the decks are going to be laid out when we haven't even
decided which kind of ship we're building yet.


> 3. L7 is also a separate work, it will not be accounted in 'API
> improvement' blueprint. You can sync with Samuel for this as we already
> have pretty detailed blueprints on that.
>

Please provide an example of how "multiple pools" will actually be used
without L7.

And again: "Let's move the discussion off the mailing list where it's been
happening over to the system where apparently others have been pressing
forward without the obvious knowledge or consent of the people having the
discussion. Oh, and by the way, because we've already written code here, a
lot of what you propose is tacitly no longer up for discussion."


> 4. Attribute differences in REST resources.
> This falls into two categories:
> - existing attributes that should belong to one or another resource,
>

The devil's in the details here. I was very specific about which attributes
belong to which objects because getting this wrong has serious implications
for use cases. (For example, if "neutron_subnet" is an attribute of a pool,
then this implies all members of the pool must be in the same
neutron_subnet. And this breaks the use case that was described on the
mailing list during the SSL re-encryption discussion where some members are
in the same subnet, and some are across the internet on the other side of a
VPN or otherwise.)

You also need to define which attributes are immutable after creation of
the object, and which can be changed with later updates (and how). This
also has implications for use cases.

If your proposal doesn't define this level of detail, then your proposal
isn't addressing the use cases and is therefore incomplete.


> - attributes that should be added (e.g. they didn't exist in current API)
>

Right. As I said last week, my intention was to try to address as many
concerns and feature requests from the mailing list discussions, wiki
pages, and google documents / spreadsheets as possible. I was hoping to
take a stab at HA as well, but the result of that discussion so far is that
we're nowhere near having any idea how to accomplish this in a way that is
going to be generically possible for all operators.

I mean: You all do realize that if there's a key missing feature that one
operator absolutely needs in order to continue their business operations,
that isn't addressed in our proposals... then that operator is NOT going to
use our product, right? And that doesn't mean you need to offer all
features out of the gate-- but you ought to have put some thought into how
you're going to do it when the time comes. This proposal is trying to be
that plan for how we're eventually going to do it. It will, of course, be
developed incrementally.

It is true that I haven't had the time to fill out all the use cases we
thought about, or that became apparent from mailing list discussions as we
were writing our API revision proposal--  our main drive was to get the
proposal out the door. My plan was (and still is) to back-fill these use
cases in the right place (which I'm no longer sure is the google doc that
Samuel created, or the gerrit system) once we got the API proposal out. So
I do apologize that I'm making reference to stuff that has been considered
but not shared thus far and realize that not having it in the shared
document weakens my position. I had to sleep.


> The first class is better to be addressed in the blueprint review. The
> second class could be a small action items/blueprints or even bugs.
>
Example:
>   1) "custom_503" - that attribute deserves it's own miniblueprint, I'd
> keep it out of scope of 'API improvement' work.
>  2) ipv6_subnet_id/addressv6 - that IMO also deserves it's own
> miniblueprint (whole thing about ipv6 support)
>

Yep, custom_503 is extremely minor, and its use case is "User would like to
have custom error 503 page displayed on HTTP / HTTPS requests when no
servers are available in the back-end pool(s)."

IPv6 discussions are quite a bit more involved-- but I will note that at
least one major vendor mentioned this as a high priority item, (and it
turns out it's easy to address in the model) which is why we threw it in
this proposal.


> So, I'd like to make the following action items out of the document:
>
> 1. Extract 'core API' - VIPs/Listeners/Pools/Members/Healthmonitors.
> This action item is actually the blueprint that I've filed and that's what
> I'm going to implement
>

"that's what I'm going to implement"--  ie. Discussion is closed on this.


> 2. Work on defining single-call API that goes along with single object
> core API (above)
> Your document already does a very good job on this front.
>

Thank you. It's nice to know some of my work is being paid attention to.


> 3. Extract 'Loadbalancer' portion of API into additional Doc/blueprint. I
> deserves it's own discussion and use cases.
> I think separating it will also help to reduce discussion contention.
>

Fine, though I would say that justifying the "VIP-centric" approach becomes
more difficult if we're not starting to think about how we're going to
address operator concerns (like, say, HA) and how these potentially
intersect and/or interface with user concerns.


> 4. Work with https://review.openstack.org/#/c/89903/ to define proper
> attribute placement of existing attributes
>

Given that you totally ignored (or planned to ignore) the work you knew I
was doing on this API proposal in writing the above blueprint...  does my
opinion on the attribute placement even matter at this point?


> 5. Define a set of attributes that are missing in proposed API and make a
> list of work items based on that.
> (I assume that there also could be some, that may make sense to include in
> proposal)
>

This stems directly from use cases. Please also point out which attributes
are mutable after creation and which are not. I'm still of the (perhaps
misguided) opinion that if I share these use cases, someone is actually
going to pay attention to them, so my plan is to work on fleshing these out
before I do anything regarding arguing over attribute placement.


> I think following this list will actually help us to make iterative
> progress and also to work on items in parallel.
>
>
Thanks again for the great document!
>

Right.

Thanks,
Stephen


-- 
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140427/8b6916f8/attachment.html>