Open Stack

Hi Eugene,

Responses inline:

On Sun, Apr 27, 2014 at 6:35 AM, Eugene Nikanorov
<enikanorov at mirantis.com>wrote:
>
> Ok, the above blueprint was uploaded on April 23rd, literally the day I
>> sent my API proposal to the mailing list. And this was after it was known
>> via the action items in the previous week's IRC meeting that my team and I
>> would be working hard over the next week to produce an API proposal
>> revision, based on the excellent work of the Rackspace team from the
>> previous week.
>>
>> I can understand having a "plan B" in case I missed a deadline there
>> (after all, I wasn't all that confident we'd get it done in time, but I
>> worked through the weekend and pulled some very long days to get it done)--
>> but it's pretty offensive to me to realize that the action items from the
>> previous week's meeting apparently weren't being taken seriously.
>>
>
> I'm not sure which of my actions exactly has offended you, was it
> submitting a blueprint to neutron-spec?
> I've been working on the several proposals, including the one on review
> since the start of Icehouse, preparing wikis, docs, and even an attempt to
> actually implement the proposal, so I'm not sure what exactly I did wrong.
> I was not thinking about beating anyone to submit blueprint first, it was
> on launchpad since the end of havana anyway and i've just renewed it and
> followed the new process.
>

You knew from the action items that came out of the IRC meeting of April 17
that my team would be working on an API revision proposal. You also knew
that this proposal was to be accompanied by an object model diagram and
glossary, in order to clear up confusion. You were in that meeting, you saw
the action items being created. Heck, you even added the "to prepare API
for SSL and L7" directive for my team yourself!

The implied but not stated assumption about this work was that it would be
fairly evaluated once done, and that we would be given a short window (ie.
about a week) in which to fully prepare and state our proposal.

Your actions, though, were apparently to produce your own version of the
same in blueprint form without notifying anyone in the group that you were
going to be doing this, let alone my team. How could you have given my API
proposal a fair shake prior to publishing your blueprint, if both came out
on the same day? (In fact, I'm lead to believe that you and other Neutron
LBaaS developers hadn't even looked at my proposal before the meeting on
4/24, where y'all started determining product direction, apparently by
edict.)

Therefore, looking honestly at your actions on this and trying to give you
the benefit of the doubt, I still must assume that you never intended to
seriously consider our proposal. (Or at the very least somehow forgot we
were working on it, despite the threads I started on the mailing list about
SSL re-encryption and HA features that had to do with it.) If you're
handing out "make work" action items, what's the point of me participating
in this process at all?

Even if we ultimately agree on much of the design and road map, you still
effectively ignored anything I was going to say by making what are
essentially product road map proposals (ie. blueprints) prior to my API
proposal being available (let alone discussed in group).

Do you now understand why I find this offensive? Can you also understand
how others, seeing how this was handled, might now be reluctant to
participate?


> That not only was there apparently never an intent to seriously consider
>> my proposal, but now, if I want to be heard, I'm going to have to
>> familiarize myself with your proposal and fight tooth and nail to be heard
>> to fix any brokenness I will almost certainly find in it. And I have to do
>> this with very little confidence that my voice is going to be heard.
>>
>
> I think our disconnect comes from the fact that whole discussion that
> started in the end of Havana from fixing the API in the way that L7 &
> multiple listeners are possible, came to the discussion of 'what lbaas API
> of the dream should look like'.
> Your document does great job of addressing the latter, but it's hardly a
> single action item/single blueprint/single patch.
>

Right, so *if* we decide to go with my proposal, we need to first decide
which parts we're actually going to go with--  I don't expect my proposal
to be complete or perfect by any means, and we need to have honest
discussion of this first. Then, once we've more-or-less come to a consensus
on this overall direction, it makes sense to think about how to split up
the work into digestible, parallelize-able chunks that can be tackled by
the various interested parties working on this project.  (My team actually
wanted to propose a road map and attach it to the proposal, but there
simply wasn't time if we wanted to get the API out before the next IRC
meeting in enough time for people to have had a chance to look at it.)

Why embark on this process at all if we don't have any real idea of what
the end-goal looks like?

I also don't yet see how one goes about having a high-level design
discussion where potentially a lot of sweeping changes are considered using
the blueprint or gerrit system alone. That's what I thought the mailing
list and wiki were for! (But I admit that I'm still not very familiar with
the blueprint or gerrit systems. I don't see yet how anyone gets holistic
views of problems and designs using these systems specifically because
they're so granular.)


> General idea of the whole API/obj model improvement is that we create a
>>> baseline for all advanced features/usecases that we have in the roadmap.
>>> Which means that those features then can be added incrementally.
>>> Incrementally means that resources or attributes might be added, but
>>> workflow remains and backward compatibility is preserved. That was not the
>>> case with multiple listeners and L7.
>>>
>>
>> I would argue that that wasn't the case for SSL either in the old model.
>> And even the model I proposed doesn't address HA yet.
>>
> The model shouldn't immediately address HA. You just have to make sure
> that once you decided to add HA, you don't have to redesign the rest of it
> (but that probably requires some ideas on how to add HA)
>

Right-- direction around HA is still too nebulous for object model
discussions. But you need to have some clue as to what it might look like,
and how to separate those concerns from the proposed object model, lest we
have to go through another extreme API / object model revision when HA
features are introduced. Again, I've been calling this "being forward
thinking."

So, a couple on general comments:
>>>
>>> 1. Whole discussion about API/obj model improvement had the goal of
>>> allowing multiple pools and multiple listeners.
>>>
>>
>> This is news to me. If we aren't going to be thinking about SSL and L7
>> also, at least, then there really was no reason to propose drastic changes
>> to the API / object model.
>>
> Stephen, L7 and multiple listeners were the reasons to revise the model.
> Let's just separate the work of improving existing API/obj model from
> additional action items, as i said above. L7 and SSL could be worked on in
> parallel, but only after deal with Pools/Vips/Listeners is fixed.
> Think of it from implementation perspective: how can one work on SSL if
> there still not fixed whether we have listeners or just VIPs? same for L7.
>

Right--  so paraphrasing what I think you're saying: "Let's consider how
we're going to tackle the SSL and L7 problems at a later date, once we've
fixed the object model to be compatible with how we're going to tackle SSL
and L7." This implies you've already considered how to tackle SSL and L7 in
making your object model change proposal!

My point is: Unless you have already considered how to do SSL and L7, then
any object model change proposal is essentially pointless. Why make these
changes unless we already know we're going to do SSL and L7 in some way
that is compatible with the proposed changes?

Evaluating these things *necessarily* must happen at the same time (ie.
holistically) or the object model proposal is pointless and unjustified.
Actually implementing the design will happen first with the core object
model changes, and then the SSL and L7 features.

Maybe that's where the confusion is coming from? I'm saying "How can you
justify drastic changes to underlying data structures unless you're taking
a holistic view of the problem?" And I think you're saying "Practically
speaking, first you have to implement the core object model changes before
you can work in parallel on the SSL and L7 implementations."  I agree on
the order in which we'll eventually write the code. But we need to evaluate
the overall design as including SSL and L7!

And just what have we been discussing on this mailing list and in the IRC
>> meetings for the last couple months? Again, I think it's a little unfair to
>> criticize the API proposal I've made because it delivers features that
>> people have been asking for for a long time (and that are already present,
>> functional, and well understood in other load balancer implementations.)
>>
> I don't remember criticizing it, I actually think it is a great and
> detailed description, good API; in the part that is about
> VIP/listeners/Pools it is nearly the same as the blueprint on spec-review,
> other parts such as L7 stuff also makes sense to me, but that's certainly
> what I didn't plan to address in my bp, which has much much more limited
> scope.
>

You're right-- perhaps I'm interpreting comments to the effect of "you're
going beyond the scope of the original proposal" as criticism.

But I think I've already made my point that not considering things like SSL
and L7 in any proposal means there's no clear indication that core object
model changes will be compatible with what needs to happen for SSL and L7.


> Also, please understand that "multiple pools" is L7! Can anyone name a use
>> case where multiple back-end pools are used on a single listener that
>> doesn't involve some L7 logic in there somewhere?
>>
> Right, it is L7. And I'm not trying to add L7, I'm just getting rid of
> 'Pool as the root object' that has prevented L7 from being implemented,
> that is in scope of my proposal. Actually adding L7 is out of scope of it.
>

Ok, so I'm hearing you say that L7 is out of scope, but changes which
enable the possibility of L7 are not. I would say that the latter cannot be
done without considering how former is likely to be done. So while actually
implementing L7 can be done separately, evaluating the proposed object
model / API design changes cannot be.


> For that purpose loadbalancer instance might be an extra. The good thing
>>> about 'loadbalancer' is that it can be introduced in the API in incremental
>>> way.
>>>
>>
>> Right. You might also notice that in the diagram, it's grayed out and
>> there are notes indicating that work needs to be done to flesh out the
>> concept based on discussion around HA functionality and operator concerns.
>> Again, this is an attempt by me to be more forward-thinking.
>>
>> There is only one attribute (of a VIP) that references the "load
>> balancer" concept as defined in the glossary (also linked). Yes, it can be
>> removed from this API proposal and added back in again later when / if the
>> discussion around HA actually happens.
>>
>>
>>> So, VIP+listeners itself is already quite flexible construct (where VIP
>>> is a root object playing loadbalancer role) that addresses our immediate
>>> needs.
>>> So I'd like to extract loadbalancer API and corresponding use cases in
>>> another blueprint.
>>> You know that loadbalancer concept has raised very heated discussions so
>>> it makes sense to continue discussing it separately, keeping in mind that
>>> introducing loadbalancer is not very complex and it may be done on top of
>>> the VIPs/listeners API
>>>
>>
>> Also, I notice this is a pretty stark 180-degree turn around on your
>> position on this from previous weeks' discussions. It's OK to change one's
>> mind on these things, but again, this leads me to believe that this is
>> being forced from outside of the Neutron LBaaS sub-project.
>>
> Not exactly. I was advocating approaches #2 and #3 from the beginning (
> https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstance/Discussion) because both of them address requirements for multiple tcp endpoints and
> L7 (structurally they are close).
> I agree that #2 is even more flexible, but the discussion about the
> loadbalancer instance has been kept for more than release cycle for now
> without much result; since #3 addresses the requirements also - I'm ok with
> that.
>

Ok, that still seems to me like you're taking that stance because people in
Neutron core see the word "load balancer" and assume it's an implementation
detail (without actually defining what an implementation detail is, nor
apparently looking at the glossary we made largely to address the confusion
around this term).

Would it help if I called it an "efkalb" in my proposal instead?


> 2. SSL-related objects. SSL is rather big deal, both from API and object
>> model, it was a separate blueprint in Icehouse and i think it makes sense
>> to work separately on it.
>>
>>> What I mean is that ssl don't affect core API (VIPs/listeners/pools)
>>> other than adding some attributes to listeners.
>>>
>>
>> Ok, again, from my perspective this is a stark change from how this has
>> been discussed in previous weeks on this mailing list and in the IRC
>> meetings. I thought it hadn't been decided exactly how SSL was going to be
>> done yet.
>>
> Something could be called 'decided' when it lands into the code
> repository. There has been some work already done on SSL on both design and
> implementation front though.
>

Work that puts the cart before the horse, IMO. Sorry, but if someone is
going to write large chunks of code without even having a way to evaluate
whether it meets requirements (because there were no stated requirements,
no design, etc.) it shouldn't come as a surprise to anyone that potentially
large chunks of this will need to be re-done once the requirements are
better understood.

If you're wondering why you haven't seen code from my team yet, this is
why! We don't like spinning our wheels when we aren't sure we're even
pointed in the right direction yet.


> (There was very little discussion of this on the mailing list, and
>> certainly nothing conclusive. And even the blueprint references a gerrit
>> patch "discussion" that essentially is just CI systems checking in. In fact
>> the only actual design discussion of this in months appears to have been
>> happening exclusively on the mailing list:
>> http://stackalytics.com/report/blueprint/neutron/lbaas-ssl-termination )
>>
>> If you want to claim that "silence implies consent" with the design, this
>> seems disingenuous to me when much more broad design discussions (which
>> ultimately affect the SSL design) are happening at the same time. It's like
>> arguing about how the decks are going to be laid out when we haven't even
>> decided which kind of ship we're building yet.
>>
> I'm not sure that I'm arguing here at all. As you probably noticed, I
> didn't participate much in ssl-related discussions.
> Speaking of SSL - we have a few few project-wise issues such as lack of
> secure storage, lack of secure messaging, requirement to have opensource
> impl of SSL API (which yet to be added).
> There's also a patch on review that is worth looking at, because someone
> has already put efforts into both design and code.
> And before throwing away all those efforts we need to be sure it
> completely not suitable with the rest of API.
>

Right, so thus far the SSL work is being done by people happy to press
forward with it, even though we're not really in agreement on how it really
fits into the rest of the design. If someone wants to start coding early,
I'm certainly not going to tell them not to, but they should realize that
when the design discussion catches up to them, it's actually pretty likely
they're going to have to rewrite a bunch of stuff (or others will rewrite
it, potentially making their code obsolete where their design doesn't sync
with the overall design and requirements).

Also, how can this evaluation be done in a fair and independent way if
there is no requirements document? It's only after Jorge from Rackspace
started this document that we even started to have requirements around SSL,
and this was started well after the coding work on SSL features was well
underway:  https://wiki.openstack.org/wiki/Neutron/LBaaS/requirements

(FWIW, my design proposal does not yet address everything in that
requirements document, either, eh.)

Also for clarification: I'm not trying to knock those who do code early
(producing POCs and whatnot). Sometimes the only way to come up with
specific requirements is to see what can be done by trying out different
ways to solve a problem. But it seems unrealistic to me to expect initial
exploratory efforts in this regard to end up in final product. Whenever I
see this done, where assumptions about how the pieces fit together are not
re-evaluated once exploratory POCs have yielded more knowledge or where
said POCs end up becoming the final product, I see significant design flaws
that get baked into final products. Why do y'all think it's been so hard to
get "VIP" and "Listener" and "Pool" separated into separate entities?
Because someone baked them all together in the initial version of this
product because it worked for the first simple use cases, and that decision
is only now being re-evaluated now that we know more, and at great cost.


>
>
>> 3. L7 is also a separate work, it will not be accounted in 'API
>>> improvement' blueprint. You can sync with Samuel for this as we already
>>> have pretty detailed blueprints on that.
>>>
>>
>> Please provide an example of how "multiple pools" will actually be used
>> without L7.
>>
> As said above, the bp targets just the baseline for L7: removing 'root
> object' role from the Pool, to actually allow multiple pools in
> configuration when L7 rules are added.
>

Right. Already discussed this above.


> And again: "Let's move the discussion off the mailing list where it's been
>> happening over to the system where apparently others have been pressing
>> forward without the obvious knowledge or consent of the people having the
>> discussion. Oh, and by the way, because we've already written code here, a
>> lot of what you propose is tacitly no longer up for discussion."
>>
>



>
>>
>>> 4. Attribute differences in REST resources.
>>> This falls into two categories:
>>> - existing attributes that should belong to one or another resource,
>>>
>>
>> The devil's in the details here. I was very specific about which
>> attributes belong to which objects because getting this wrong has serious
>> implications for use cases. (For example, if "neutron_subnet" is an
>> attribute of a pool, then this implies all members of the pool must be in
>> the same neutron_subnet. And this breaks the use case that was described on
>> the mailing list during the SSL re-encryption discussion where some members
>> are in the same subnet, and some are across the internet on the other side
>> of a VPN or otherwise.)
>>
> Right. I should have said it's a work item for me - to look closely
> through your doc and address the differences. Also, that is exactly the
> kind of work item where gerrit helps a lot.
>
>
>>
>> You also need to define which attributes are immutable after creation of
>> the object, and which can be changed with later updates (and how). This
>> also has implications for use cases.
>>
> Well, this is usually defined in the code, I can add this to the spec as
> well. I hoped to avoid duplicate work.
>

Code is the poorest form of documentation. It captures the "how" and "what"
but not the "why" of decisions.


>
>
>>
>
>> If your proposal doesn't define this level of detail, then your proposal
>> isn't addressing the use cases and is therefore incomplete.
>>
> I'm working on that.
>
>
>>
>>
>>> - attributes that should be added (e.g. they didn't exist in current API)
>>>
>>
>> Right. As I said last week, my intention was to try to address as many
>> concerns and feature requests from the mailing list discussions, wiki
>> pages, and google documents / spreadsheets as possible. I was hoping to
>> take a stab at HA as well, but the result of that discussion so far is that
>> we're nowhere near having any idea how to accomplish this in a way that is
>> going to be generically possible for all operators.
>>
>> I mean: You all do realize that if there's a key missing feature that one
>> operator absolutely needs in order to continue their business operations,
>> that isn't addressed in our proposals... then that operator is NOT going to
>> use our product, right?
>>
>
>
>> And that doesn't mean you need to offer all features out of the gate--
>> but you ought to have put some thought into how you're going to do it when
>> the time comes. This proposal is trying to be that plan for how we're
>> eventually going to do it. It will, of course, be developed incrementally.
>>
> So, what are we arguing about? I'm just doing the small part that fixes
> relationship issue with the obj model.
>

We're arguing about whether we should be evaluating (so-called) advanced
features in our design discussion (which I would say, is vital, if we plan
on actually offering them someday). I think you're confusing this with an
implementation roadmap discussion.

My proposal is a *design discussion*. It is not about how we're going to go
about actually coding anything yet. We need to get the design right before
we can decide how to divide and conquer this problem!


> So, I'd like to make the following action items out of the document:
>>>
>>> 1. Extract 'core API' - VIPs/Listeners/Pools/Members/Healthmonitors.
>>> This action item is actually the blueprint that I've filed and that's
>>> what I'm going to implement
>>>
>>
>> "that's what I'm going to implement"--  ie. Discussion is closed on this.
>>
> I'm not sure I get your point here. Just reiterating: 'root object' moved
> to VIP, listeners added - that's what i'm addressing.
> Once we have that, you can actually develop patches on everything else and
> be sure that you have not redone everything if underlying design is changed.
>

Look, I agree that splitting out VIP, Listener, and Pool to be their own
objects, and moving the "root" of the object tree to the VIP is probably
the right thing to do. But there are valuable contributors in this
discussion who are still not on-board with this idea. So why is this
fundamental design discussion being closed early? That was the point of my
comment.


> 2. Work on defining single-call API that goes along with single object
>>> core API (above)
>>> Your document already does a very good job on this front.
>>>
>>
>> Thank you. It's nice to know some of my work is being paid attention to.
>>
>>
>>> 3. Extract 'Loadbalancer' portion of API into additional Doc/blueprint.
>>> I deserves it's own discussion and use cases.
>>> I think separating it will also help to reduce discussion contention.
>>>
>>
>> Fine, though I would say that justifying the "VIP-centric" approach
>> becomes more difficult if we're not starting to think about how we're going
>> to address operator concerns (like, say, HA) and how these potentially
>> intersect and/or interface with user concerns.
>>
> Well, I'd also would like to hear from those, who don't want neutron to
> provide 'virtualized appliance' functionality.
> I think that meanwhile we may treat VIP as 'loadbalancer' object and apply
> HA features directly to it.
>

This is such a huge can of worms, this is why I didn't attempt to address
it in our design proposal.

Also, per a previous e-mail thread, I'm all for banning the use of the term
"load balancer" to describe any single object or primitive. There are many
valid definitions of the term "load balancer" in our industry, and to avoid
confusion I'd like to avoid all of them when referring to specific objects
or primitives.

If someone asks, "How can you be building a 'load balancer as a service'
product if it doesn't actually contain anything that can be called a load
balancer?" Then the response should be, "The whole thing is the load
balancer. It has many parts, and none can appropriately be called a 'load
balancer' on its own."


> 4. Work with https://review.openstack.org/#/c/89903/ to define proper
>>> attribute placement of existing attributes
>>>
>>
>> Given that you totally ignored (or planned to ignore) the work you knew I
>> was doing on this API proposal in writing the above blueprint...  does my
>> opinion on the attribute placement even matter at this point?
>>
> Sorry again for offensive phrasing, i didn't mean that. That was meant to
> be is a work item more for me than for you, but i hoped to get some inline
> comments on gerrit.
>

Again, for the most part it wasn't actually your phrasing that I found
offensive. I hope you understand that now. Thank you for the apology in any
case. I have enjoyed working with y'all on this project and hope to
continue to do so.


> At the end I'd like to emphasize that I was not planning to implement
> everything and account for all cases.
> The scope is to add listeners and move 'root object' role from Pool to the
> VIP, that's it, and that's what is on spec review.
>

Right. And I think I've illustrated my case that evaluating a partial
design (one that claims not to consider the next couple of logical steps at
least) is problematic at best, and that evaluating a design should be done
holistically, and that said evaluation should have little to do with
discussions about how to actually go about coding it until we're closer to
consensus on the design.

Stephen


-- 
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140427/eac681b7/attachment.html>