Open Stack

Replying inline.

> -----Original Message-----
> From: Monty Taylor [mailto:mordred at inaugust.com]
> Sent: Friday, April 17, 2015 7:53 PM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [Nova][Neutron] Linuxbridge as the default in
> DevStack [was: Status of the nova-network to Neutron migration work]
> 
> On 04/17/2015 06:48 PM, Rochelle Grober wrote:
> > I know the DevStack issue seems to be solved, but I had to
> > respond.....inline
> >
> > From: Fox, Kevin M [mailto:Kevin.Fox at pnnl.gov] Sent: Friday, April 17,
> > 2015 12:28 To: OpenStack Development Mailing List (not for usage
> > questions) Subject: Re: [openstack-dev] [Nova][Neutron] Linuxbridge as
> > the default in DevStack [was: Status of the nova-network to Neutron
> > migration work]
> >
> > No, the complaints from ops I have heard even internally, which I
> > think is being echo'd here is "I understand how linux bridge works, I
> > don't opensvswitch". and "I don't want to be bothered to learn to
> > debug openvswitch because I don't think we need it".
> >
> > If linux bridge had feature parity with openvswitch, then it would be
> > a reasonable argument or if the users truly didn't need the extra
> > features provided by openvswitch/naas. I still assert though, that
> > linux bridge won't get feature parity with openvswitch and the extra
> > features are actually critical to users (DVR/NaaS), so its worth
> > switching to opevnswitch and learning how to debug it. Linux Bridge is
> > a nonsolution at this point.
> 
> I'm sorry, but with all due respect - I believe that sounds very much like
> sticking fingers in ears and not paying attention to the very real needs of
> users.

No, when you have complex software, with multiple classes of users, it is almost impossible to please all your users, in every way. Sime times, you must make hard decisions to make one users experience a little less good for the benefit of the whole community. /me channels Spock here...

If it makes the Ops life a little harder, but for every Op that has to learn how to debug openvswitch, 100 users don't have to deal with the difference between nova-network and neutron api's and software built on top of OpensStack that only works with one of them, I think that's worth the tradeoff. Its unfortunate, but necessary. Ops have to learn new things all the time. Its in the job description.

I currently Operate 3 different OpenStack clouds, so I'm not just trying to push work on others and not myself. I paid the learning curve cost.
 
> Let me tell you some non-features I encounter currently:
> 
> - Needing Floating IPs to get a public address
> 
> This is touted as "the right way to do it" - but it's actually a terrible
> experience for a user. The clouds I have access to that just give me a direct
> DHCP address are much more useful.

Another case of short term pain for long term gain.

Its nice to be able to not use them, up until you realize you needed it, don't have it, and its too late to deal with it.

Ip addresses are stateful creatures. You attach dns entries to them. Some users contact them directly. They are a window into your machine. The cloud is all about scaling. If you can't just move an ip from vm to vm,  you force them to become pets. Without them, you're operating much like in the virtualization days before cloud.

> In fact, we should delete floating ips - they are a non-feature that make life
> harder. Literally no user of a cloud has ever wanted them, although we've
> learned to deal with them.

Terrible idea.

The first time I moved a floating ip from vm 1 to vm2 to do a rolling update that took under a second, it paid off. And my users benefited. Or the times I deleted vm's and launched new vm's in their place, and no data was lost and no one noticed.

Cloud is a very different way to do things and if you don't understand it well, can be confused with traditional virtualization. It too, is worth the learning curve to understand how to do things the Cloud Way. You don't know you want it, until you go through the learning curve and understand why they really make sense. To keep state where it belongs, out of the vm.

This is the hart of the issue we're discussing. People are wanting to force the cloud software to function not as a cloud, but more like what they are familiar with. But that's a bad idea. You gut the very features that make Cloud awesome.

> - SDN
> 
> I understand this is important for people, so let's keep it around - but having
> software routers essentially means that it's a scaling bottleneck. In the
> cloud Infra uses that has SDN, we have to create multiple software routers
> to handle the scaling issues. On the other hand, direct routing / linuxbridge
> does NOT have this problem, because the network packets are routed
> directly.

Only if you gut the network stack by making it flat and making NaaS optional.

But for NaaS, OpenVSwitch backend does support scaled routing. Its called DVR. The linux bridge agent does not. And at the current rate of development, the Linux Bridge is likely not to. If NaaS really is critical, then OpenVswitch over linux bridge pays dividends.

> We should not delete SDN like we should delete floating IPs, because there
> are real users who have real uses cases and SDN helps them.
> However, it should be an opt-in feature for a user that is an add on.

Then app developers can't rely on it being there, and users can't have as much software readily available to launch in their tenant, weakening the ecosystem.
 
> vexxhost is getting this right right now - you automatically get a DHCP'd
> direct routed IP on each VM you provision, but if you decide you need fancy,
> you can opt in to create a private network.

Then how do you deal with a hypervisor dieing and dns records pointing to that ip? You encourage the vm a pet. This seems fine until it happens. Then it hurts.

> 
> - DVR
> 
> I'm an end user. I do not care about this at all. DVR is only important if you
> have bought in to software routers. It's a solution to a problem that would
> go away if things worked like networks.

I'm a cloud user too. I don't directly care about it either. Other then needing to ensure when I use NaaS, it scales. The how is irrelevant to me. If it's done with a Cisco neutron plugin, that's fine. I don't have to care. The thing that sucks is going from cloud to cloud, and having to write two sets of templates, one for nova-network  and one for neutron since the api's are different. The user is being forced to care. This is bad.
 
> >:/ So is keeping nova-network around
> > forever. :/ But other then requiring some more training for ops
> >folks, I think Neutron can suit the rest of the use cases these days
> >nova-network provided over neutron. The sooner we can put the
> >nova-network issue to bed, the better off the ecosystem will be. It
> >will take a couple of years for the ecosystem to settle out to
> >deprecating it, since a lot of clouds take years to upgrade and
> >finally put the issue to bed. Lets do that sooner rather then later  so
> >a couple of years from now, we're done. :/
> 
> I'm about to deploy a cloud, I'm going to run neutron, and I'm not going to
> run openvswitch because I do not need it. I will run the equiv of flatdhcp.

Its good that your using neutron. Its unfortunate for the community that this fracturing is occurring.

App developers have at least 3 targets. Nova-network, neutron with flat network, neutron with tenant networks. It's a lot of effort to write and debug one template, let alone 3. :/  Still, I'd prefer 2 over 3 any day. :/

> If neutron doesn't have it, I will write it, because it's that important that it
> exist.

So be it. One of the great things about open source is you can do whatever you want.

Oddly, this flexibility is also its Achilles heel. In the application space, it's a great thing. In an Operating System, it tends to hurt the flexibility of the things built on top. This is why Linux ultimately won over the BSD's. Linux stayed relatively fork free, while the BSD's are quite divergent. The lack of divergence helped Linux app developers.

Another example, take cellphones. Linux was early to that party, and lost, since so many different linux implementations targeted the phone space with different api's. Everyone followed their self interests, and the ecosystem on top never materialized since there were too many ways to do everything and nothing worked the same.

Google takes the same Linux kernel, puts a bit of userspace on top, calls it android and encourages an app ecosystem on top, and bam. The OS becomes the number one phone OS in terms of users. They do stuff to try and minimize forks and divergent functionality, and the whole ecosystem benefits from it. I'm not saying everything Google has done there has been good, but the general idea of app ecosystem encouragement is good.

OpenStack is an operating system and needs to encourage a wealth of users/apps on top of it. The main way to do that is to make sure your abstractions are clean enough that the stuff under the hood don't matter to the cloud user/app developer. But with nova-network/neutron, it does. Same with FlatDHCP. :/

> 
> If you take that ability away from me, you will be removing working feature
> and replacing them with things that make my user experience worse.
> 
> Let's not do that. Let's listen to the people who are using this thing as end
> users. Let's understand their experience and frustration. And let's not chase
> pie-in-the-sky theory of how it "should" work in the face of what a ton of
> people are asking and even begging for. FlatDHCP is perfect for the 80%
> case. The extra complexity of the additional things if you don't actually need
> them is irresponsible.

I think we're at a philosophical impasse here.  Unfortunately I don't think we're going to agree. And that's ok. That's the beauty of open source. :)

Thanks,
Kevin

> 
> >
> > [Rockyg] Kevin, the problem is that the extra features *aren't*
> > critical to the deployers and/or users of many of openstack
> > deployments.  And since they are not critical, the deployers won't
> > *move* to using neutron that requires them to learn all this new
> > "stuff" that thjey don't need.  By not providing a simple path to a
> > flatDHCP implementation, you will get existing users refusing to
> > upgrade rather than take a bunch of extraneous stuff from Neutron
> > because the OpenStack project deprecated "their network." So, likely
> > two things will happen: 1) the deployments that are already you there
> > configured with nova-network and flatDHCP will stop upgrading with the
> > last nova-network release and 2) if there isn't a simple equivalent by
> > then in neutron or some other openstack project, someone will fork to
> > keep the flatDHCP solution moving forward.
> >
> > You can lead a devops to pizza, but you can't make it eat soylent
> > green pizza.  And that's how you lose some of the community and
> > perhaps spur either Neutron's or OpenStack's successor open source
> > project(s).
> >
> > KISS is still in effect.  It seems Neutron is abstracting away the
> > current network complexities for developers and endusers at the
> > expense of tossing it all on the shoulders of the deployer/admins.
> > Until you abstract some of that complexity out of the deployment path,
> > either through good coding, useful templates, configuration and
> > management tools, etc., you're going to continue to get pushback from
> > the devops and they will continue to claim parity doesn't exist *for
> > them*.
> >
> > Something I learned a while ago - the sysadmins control the system and
> > stick with minor changes and/or single system by system upgrades until
> > they are either tempted with something shiny/fun/cool/sexy/powerful or
> > coerced by management to change.
> > Until you can demonstrate a *benefit* to them to move to the neutron
> > paradigm for their flatDHCP network, you won't get them to move.
> > They'll take a learning ramp-up, for either less work or better
> > control, but they won't take it for more work.
> >
> > --Rocky
> >
> > ________________________________ From: Kevin Benton
> > [blak111 at gmail.com] Sent: Friday, April 17, 2015 11:49 AM To:
> > OpenStack Development Mailing List (not for usage questions) Subject:
> > Re: [openstack-dev] [Nova][Neutron] Linuxbridge as the default in
> > DevStack [was: Status of the nova-network to Neutron migration work] I
> > definitely understand that. But what is the major complaint from
> > operators? I understood that quote to imply it was around Neutron's
> > model of self-service networking.
> >
> > If the main reason the remaining Nova-net operators don't want to use
> > Neutron is due to the fact that they don't want to deal with the
> > Neutron API, swapping some implementation defaults isn't really going
> > to get us anywhere on that front.
> >
> > It's an important distinction because it determines what actionable
> > items we can take (e.g. what Salvatore mentioned in his email about
> > defaults). Does that make sense?
> >
> > On Fri, Apr 17, 2015 at 11:33 AM, Jeremy Stanley
> > <fungi at yuggoth.org<mailto:fungi at yuggoth.org>> wrote: On 2015-04-17
> > 10:55:19 -0700 (-0700), Kevin Benton wrote:
> >> I understand. What I'm saying is that switching to Linux bridge will
> >> not change the networking model to 'just connect everything to a
> >> simple flat network'. All of the complaints about self-service
> >> networking will still hold.
> >
> > And conversely, swapping simple bridge interfaces for something else
> > still means problems are harder to debug, whether or not you're stuck
> > with self-service networking features you're not using. -- Jeremy
> > Stanley
> >
> >
> ___________________________________________________________________
> ___
> > ____
> >
> >
> OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe<http://O
> > penStack-dev-request at lists.openstack.org?subject:unsubscribe>
> >
> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> > -- Kevin Benton
> >
> >
> >
> >
> ___________________________________________________________________
> ___
> > ____
> >
> >
> OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> 
> ___________________________________________________________________
> _______
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev