[openstack-dev] [all][tc] Languages vs. Scope of "OpenStack"

Erno Kuvaja ekuvaja at redhat.com
Tue May 24 00:07:50 UTC 2016

On Tue, May 24, 2016 at 12:28 AM, Gregory Haynes <greg at greghaynes.net>

> On Mon, May 23, 2016, at 05:24 PM, Morgan Fainberg wrote:
> On Mon, May 23, 2016 at 2:57 PM, Gregory Haynes <greg at greghaynes.net>
> wrote:
> On Fri, May 20, 2016, at 07:48 AM, Thierry Carrez wrote:
> > John Dickinson wrote:
> > > [...]
> > >> So the real question we need to answer is... where does OpenStack
> > >> stop, and where does the wider open source community start ? If
> > >> OpenStack is purely an "integration engine", glue code for other
> > >> lower-level technologies like hypervisors, databases, or distributed
> > >> block storage, then the scope is limited, Python should be plenty
> > >> enough, and we don't need to fragment our community. If OpenStack is
> > >> "whatever it takes to reach our mission", then yes we need to add one
> > >> language to cover lower-level/native optimization, because we'll
> > >> need that... and we need to accept the community cost as a
> > >> consequence of that scope choice. Those are the only two options on
> > >> the table.
> > >>
> > >> I'm actually not sure what is the best answer. But I'm convinced we,
> > >> as a community, need to have a clear answer to that. We've been
> > >> avoiding that clear answer until now, creating tension between the
> > >> advocates of an ASF-like collection of tools and the advocates of a
> > >> tighter-integrated "openstack" product. We have created silos and
> > >> specialized areas as we got into the business of developing time-
> > >> series databases or SDNs. As a result, it's not "one community"
> > >> anymore. Should we further encourage that, or should we focus on
> > >> what the core of our mission is, what we have in common, this
> > >> integration engine that binds all those other open source projects
> > >> into one programmable infrastructure solution ?
> > >
> > > You said the answer in your question. OpenStack isn't defined as an
> > > integration engine[3]. The definition of OpenStack is whatever it
> > > takes to fulfill our mission[4][5]. I don't mean that as a tautology.
> > > I mean that we've already gone to the effort of defining OpenStack.
> It's
> > > our mission statement. We're all about building a cloud platform upon
> > > which people can run their apps ("cloud-native" or otherwise), so we
> > > write the software needed to do that.
> > >
> > > So where does OpenStack stop and the wider community start? OpenStack
> > > includes the projects needed to fulfill its mission.
> >
> > I'd totally agree with you if OpenStack was developed in a vacuum. But
> > there is a large number of open source projects and libraries that
> > OpenStack needs to fulfill its mission that are not in "OpenStack": they
> > are external open source projects we depend on. Python, MySQL, libvirt,
> > KVM, Ceph, OpenvSwitch, RabbitMQ... We are not asking that those should
> > be included in OpenStack, and we are not NIHing replacements for those
> > in OpenStack either.
> >
> > So it is not as clear-cut as you present it, and you can approach this
> > dependency question from two directions.
> >
> > One is community-centric: "anything produced by our community is
> > OpenStack". If we are missing a lower-level piece to achieve our mission
> > and are developing it ourselves as a result, then it is OpenStack, even
> > if it ends up being a message queue or a database.
> >
> > The other approach is product-centric: "lower-level pieces are OpenStack
> > dependencies, rather than OpenStack itself". If we are missing a
> > lower-level piece to achieve our mission and are developing it as a
> > result, it could be developed on OpenStack infrastructure by members of
> > the OpenStack community but it is not "OpenStack the product", it's an
> > OpenStack *dependency*. It is not governed by the TC, it can use any
> > language and tool deemed necessary.
> >
> > On this second approach, there is the obvious question of where
> > "lower-level" starts, which as you explained above is not really
> > clear-cut. A good litmus test for it could be whenever Python is not
> > enough. If you can't develop it effectively with the language that is
> > currently sufficient for the rest of OpenStack, then developing it as an
> > OpenStack dependency in whatever language is appropriate might be the
> > solution...
> >
> > That is what I mean by 'scope': where does "OpenStack" stop, and where
> > do "OpenStack dependencies" start ? It is a lot easier and a lot less
> > community-costly to allow additional languages in OpenStack dependencies
> > (we already have plenty there).
> >
> I strongly agree with both of the points about what OpenStack is defined
> as. We are  a set of projects attempting to fulfill our mission. In
> doing so, we try to use outside dependencies to help us as much as
> possible. Sometimes we cannot find an outside dependency to satisfy a
> need whether due to optimization needs, licensing issues, usability
> problems, or simply because an outside project doesn't exist. That is
> when things become less clear-cut and we might need to develop software
> not purely/directly related to fulfilling our mission.
> In the product-centric approach I worry that we are going to be paying
> many of the costs which the existing TC resolutions hoped to prevent. We
> still will have to maintain and debug these dependencies and if the
> OpenStack community is the only ones doing so then the language
> fragmentation costs will come in to play. Our goal should be to only pay
> these costs when necessary, and I am not sure this type of distinction
> helps us do that.
> Additionally, I am not sure how this would solve the swift case. The
> object server is the component that has been rewritten so either it
> would need to live outside of OpenStack with the rest of Swift being
> part of OpenStack, or all of Swift would need to move outside. Neither
> of these seems ideal.
> This makes me wonder - does this decision need to be an all-or-nothing
> thing? Can we say that the swift object server is a special case, let it
> be written in another language, and leave it at that? My thinking is -
> there is only value in us deciding on and allowing an additional
> language for solving these issues generally if there are other places
> that require this new language. It has been mentioned that Gnocchi could
> potentially be a case in the future but until that problem actually
> comes up there isn't much benefit to us deciding on another language
> anyone can use and there are plenty of downsides to doing so.
> I really do not want to "special case" swift. It really doesn't go with
> the spirit of inclusion.
> I am not sure how inclusion is related to special casing. Inclusion here
> implies that some group is not being accepted in to our community. The
> excluded group here would be one writing software not in python, which is
> the same groups being excluded currently. This is an argument against the
> status quo, not against special casing.
> I expect we will see a continued drive for whichever additional languages
> are supported once it's in place/allowed.
> That is the problem. The social and debugging costs of adding another
> language are a function of how much that other language is used. If one
> component of one project is written in another language then these costs
> should be fairly low. I agree that once we allow another language we should
> expect many projects to begin using it, and IMO most if not all of these
> cases except swift will not be warranted. Thus, we will be paying the costs
> in development resources from several standpoints without cause.
> I also worry "special casing" drives at a core thing we've been trying to
> dissuade: Swift is not a special case and is part of OpenStack. It should
> be considered as such and adding more onto that line just divides the
> community further.
> I think that filling the role of "native optimizations that are
> inappropriate in python" is something we will continue to have requests for
> especially if one project has blessing to use this
> native-tool-optimization-thing.
> This is my main disagreement with the all-or-nothing premise - we seem to
> have a bad case of fallacy of composition. AFAIK we do not have other
> components of OpenStack which are needing to do a read on a large number of
> on disk files in parallel from python, let alone do it in a highly
> performant way. Just because we have identified one non trivial problem for
> a language does not mean we have lots more to come. It is entirely possible
> that another project will have this same problem or we will identify
> another problem in the future which requires a similar solution, but I
> don't know why we expect to or even want to solve that issue now before
> that case is clearly defined. Hopefully we don't have the need to write
> more databases as part of OpenStack...
> So in short, lets not special case a project here.
> Cheers,
> Greg
> Cheers,
> Greg
I do like the special casing part. Lets special case the deviation from
Python to those projects that needs it. This whole discussion is not if we
accept language apart from Python or not as we do already.

Why I do like the special casing Swift now and Gnocchi next week rather
than opening this up; purely because I don't want to see in 3 months 7 new
projects joining writing their service and API on node.js because
JavaScript is now generally acceptable language as we don't special case
these special needs (Why should we special case one project if we can't do
that with another). Also I think TripleO was still mainly written in bash
when it joined, we did not need to add bash for officially approved
languages when that happened and no-one tried to convince the community
that we should do more of that.

I'd be more than happy to keep Python as our official development language
and grant/trust TC with the judgement to deviate from that where needed.
Also where possible I'd like to see us staying with the languages that can
be interfaced directly with Python. Don't want to see Perl either just
because one can optimize the processing of the strings superior compared to
Python. Otherwise I'm sure that some point someone comes with the use case
where Python nor Golang is not optimal and we have this discussion again.
Soon enough having full salad of languages listed, used and the movement
between projects becomes more and more difficult for the contributors.

TL;DR Please agree that these tight optimizations are special case and lets
not change our official line that OpenStack is developed in Python apart
from some special cases. ;)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160524/18a5fd7c/attachment.html>

More information about the OpenStack-dev mailing list