[openstack-dev] [tc] [all] OpenStack moving both too fast and too slow at the same time

Matt Riedemann mriedemos at gmail.com
Sat May 6 22:15:19 UTC 2017

On 5/5/2017 6:44 PM, Fox, Kevin M wrote:
> Note, when I say OpenStack below, I'm talking about nova/glance/cinder/neutron/horizon/heat/octavia/designate. No offence to the other projects intended. just trying to constrain the conversation a bit... Those parts are fairly comparable to what k8s provides.
> I think part of your point is valid, that k8s isn't as feature rich in some ways, (networking for example), and will get more complex in time. But it has a huge amount of functionality for significantly less effort compared to an OpenStack deployment with similar functionality today.
> I think there are some major things different between the two projects that are really paying off for k8s over OpenStack right now. We can use those as learning opportunities moving forward or the gap will continue to widen, as will the user migrations away from OpenStack. These are mostly architectural things.
> Versions:
>  * The real core of OpenStack is essentially version 1 + iterative changes.
>  * k8s is essentially the third version of Borg. Plenty of room to ditch bad ideas/decisions.
> That means OpenStack's architecture has essentially grown organically rather then being as carefully thought out. The backwards compatibility has been a good goal, but its so hard to upgrade most places burn it down and stand up something new anyway so a lot of work with a lot less payoff then you would think. Maybe it is time to consider OpenStack version 2...
> I think OpenStack's greatest strength is its standardized api's. Thus far we've been changing the api's over time and keeping the implementation mostly the same... maybe we should consider keeping the api the same and switch some of the implementations out... It might take a while to get back to where we are now, but I suspect the overall solution would be much better now that we have so much experience with building the first one.
> k8s and OpenStack do largely the same thing. get in user request, schedule the resource onto some machines and allow management/lifecycle of the thing.
> Why then does k8s scalability goal target 5000 nodes and OpenStack really struggle with more then 300 nodes without a huge amount of extra work? I think its architecture. OpenStack really abuses rabbit, does a lot with relational databases that maybe are better done elsewhere, and forces isolation between projects that maybe is not the best solution.
> Part of it I think is combined services. They don't have separate services for cinder-api/nova-api,neutron-api/heat-api/etc. Just kube-apiserver. same with the *-schedulers, just kube-scheduler. This means many fewer things to manage for ops, and allows for faster communication times (lower latency). In theory OpenStack could scale out much better with the finer grained services. but I'm not sure thats really ever shown true in practice.
> Layered/eating own dogfood:
>  * OpenStack assumes the operator will install "all the things".
>  * K8s uses k8s to deploy lots of itself.
> You use kubelet with the same yaml file format normally used to deploy stuff to deploy etcd/kube-apiserver/kube-scheduler/kube-controller-manager to get a working base system.
> You then use the base system to launch sdn, ingress, service-discovery, the ui, etc.
> This means the learning required is substantially less when it comes to debugging problems, performing upgrades, etc, because its the same for the most part for k8s as it is for any other app running on it. The learning costs is way lower.
> Versioning:
>  * With OpenStack, Upgrades are hard, mismatched version servers/agents are hard/impossible.
>  * K8s, they support the controllers being 2 versions ahead of the clients.
> Its hard to bolt this on after the fact, but its also harder when you have multiple communications channels to do it with. Having to do it in http, sql, and rabbit messages makes it so much harder. Having only one place talk to the single datastore (etcd) makes that easier, as is only having one place everything interacts with the servers kube-apiserver.
> Some amount of distribution:
>  * OpenStack components are generally expected to come from distro's.
>  * K8s, Core pieces like kube-apiserver are distributed prebuilt and ready to go in container images if you chose to use them.
> Minimal silo's:
>  * the various OpenStack projects are very silo'ed.
>  * Most of the k8s subsystems currently are all tightly integrated with each other and are managed by the same teams.
> This has lead to architectural decisions that take more of the bigger picture into account. Under OpenStack's current model, each project does their own thing without too much design into how it all comes together in the end. A lot of problems spring from this. I'm sure k8s will get more and more silo'ed as it grows and matures. But right now, the lack of silo's really are speeding its development.
> Anyway, I think I'm done rambling for now... I do hope we can reflect on some of the differences between the projects and see if we can figure out how to pull in some of the good ideas from k8s.
> Thanks,
> Kevin

Am I stating the obvious when I say that in the beginning there were two 
projects in OpenStack: Nova and Swift. Cinder and Glance sprang forth 
from Nova. We had nova-network (we still do). Neutron didn't come from 
Nova and it's taken us a few years to finally get to the place where we 
deprecated nova-network and made Neutron the default networking backend 
for Nova.

Yes things could move quick and dirty when it was essentially a single 
project. We're still paying for a lot of quick and dirty technical debt 
within Nova, and a lot of old code we've either deprecated or are 
working on deprecating (like rxtx_factor in flavors and personality 
files to name a couple but there are really a ton of examples - quotas, 
bdm v1, cells v1, etc).

Splitting nova-volume out to Cinder still left a *ton* of tight-coupling 
between projects. That's a major reason it's taken so long to get volume 
multi-attach supported in Nova, because the interfaces between the 
projects were just bad (lots of shared data and knowledge about the 
internals of each project rather than coding to clean REST API 
interfaces). We're working on correcting that now, but it takes time.

Frankly I'm fine with the projects doing their own thing as long as 
there is a clean API (with microversions for changes) between them.

It's an interesting idea to think what it would be like if everything 
was a couple of projects, but it also scares me. :)

It's also fun to think about just dropping backward compatibility on the 
API and do a brand new major version, but we talked about that with Nova 
v3 for how many releases? At least 3 before deciding that microversions 
was the best way forward. We're going to be talking about raising 
minimum microversions in the API in Boston and I'm eager to get feedback 
on that idea.




More information about the OpenStack-dev mailing list