Open Stack

OpenStack is more then the sum of its various pieces. Often features need to cross more then one project. Cross project work is already extremely hard without having to change languages between. Language change should be done very carefully and deliberately.

Thanks,
Kevin

________________________________
From: Geoff O'Callaghan
Sent: Monday, May 23, 2016 5:59:13 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][tc] Languages vs. Scope of "OpenStack"

Surely openstack is defined by it’s capabilities and interfaces rather than it’s internals.  Given the simplistic view that openstack is a collection of micro services connected by well defined api’s does it really matter what code is used inside that micro service (or macro service )?   If there is a community willing to bring and support code in language ‘x’,  isn’t that better than worrying about the off chance of existing developers wishing to move projects and not knowing the target language?    Is there a fear that we’ll end up with a fork of nova (or others) written in rust ?
If we’re not open to evolving then we’ll die.

Just throwing in a different perspective.

Sorry about the top-post but it applies in general to the below discussion
Tks
Geoff

On 24 May 2016, at 9:28 AM, Gregory Haynes <greg at greghaynes.net<mailto:greg at greghaynes.net>> wrote:

On Mon, May 23, 2016, at 05:24 PM, Morgan Fainberg wrote:


On Mon, May 23, 2016 at 2:57 PM, Gregory Haynes <greg at greghaynes.net<mailto:greg at greghaynes.net>> wrote:
On Fri, May 20, 2016, at 07:48 AM, Thierry Carrez wrote:
> John Dickinson wrote:
> > [...]
> >> So the real question we need to answer is... where does OpenStack
> >> stop, and where does the wider open source community start ? If
> >> OpenStack is purely an "integration engine", glue code for other
> >> lower-level technologies like hypervisors, databases, or distributed
> >> block storage, then the scope is limited, Python should be plenty
> >> enough, and we don't need to fragment our community. If OpenStack is
> >> "whatever it takes to reach our mission", then yes we need to add one
> >> language to cover lower-level/native optimization, because we'll
> >> need that... and we need to accept the community cost as a
> >> consequence of that scope choice. Those are the only two options on
> >> the table.
> >>
> >> I'm actually not sure what is the best answer. But I'm convinced we,
> >> as a community, need to have a clear answer to that. We've been
> >> avoiding that clear answer until now, creating tension between the
> >> advocates of an ASF-like collection of tools and the advocates of a
> >> tighter-integrated "openstack" product. We have created silos and
> >> specialized areas as we got into the business of developing time-
> >> series databases or SDNs. As a result, it's not "one community"
> >> anymore. Should we further encourage that, or should we focus on
> >> what the core of our mission is, what we have in common, this
> >> integration engine that binds all those other open source projects
> >> into one programmable infrastructure solution ?
> >
> > You said the answer in your question. OpenStack isn't defined as an
> > integration engine[3]. The definition of OpenStack is whatever it
> > takes to fulfill our mission[4][5]. I don't mean that as a tautology.
> > I mean that we've already gone to the effort of defining OpenStack. It's
> > our mission statement. We're all about building a cloud platform upon
> > which people can run their apps ("cloud-native" or otherwise), so we
> > write the software needed to do that.
> >
> > So where does OpenStack stop and the wider community start? OpenStack
> > includes the projects needed to fulfill its mission.
>
> I'd totally agree with you if OpenStack was developed in a vacuum. But
> there is a large number of open source projects and libraries that
> OpenStack needs to fulfill its mission that are not in "OpenStack": they
> are external open source projects we depend on. Python, MySQL, libvirt,
> KVM, Ceph, OpenvSwitch, RabbitMQ... We are not asking that those should
> be included in OpenStack, and we are not NIHing replacements for those
> in OpenStack either.
>
> So it is not as clear-cut as you present it, and you can approach this
> dependency question from two directions.
>
> One is community-centric: "anything produced by our community is
> OpenStack". If we are missing a lower-level piece to achieve our mission
> and are developing it ourselves as a result, then it is OpenStack, even
> if it ends up being a message queue or a database.
>
> The other approach is product-centric: "lower-level pieces are OpenStack
> dependencies, rather than OpenStack itself". If we are missing a
> lower-level piece to achieve our mission and are developing it as a
> result, it could be developed on OpenStack infrastructure by members of
> the OpenStack community but it is not "OpenStack the product", it's an
> OpenStack *dependency*. It is not governed by the TC, it can use any
> language and tool deemed necessary.
>
> On this second approach, there is the obvious question of where
> "lower-level" starts, which as you explained above is not really
> clear-cut. A good litmus test for it could be whenever Python is not
> enough. If you can't develop it effectively with the language that is
> currently sufficient for the rest of OpenStack, then developing it as an
> OpenStack dependency in whatever language is appropriate might be the
> solution...
>
> That is what I mean by 'scope': where does "OpenStack" stop, and where
> do "OpenStack dependencies" start ? It is a lot easier and a lot less
> community-costly to allow additional languages in OpenStack dependencies
> (we already have plenty there).
>

I strongly agree with both of the points about what OpenStack is defined
as. We are  a set of projects attempting to fulfill our mission. In
doing so, we try to use outside dependencies to help us as much as
possible. Sometimes we cannot find an outside dependency to satisfy a
need whether due to optimization needs, licensing issues, usability
problems, or simply because an outside project doesn't exist. That is
when things become less clear-cut and we might need to develop software
not purely/directly related to fulfilling our mission.

In the product-centric approach I worry that we are going to be paying
many of the costs which the existing TC resolutions hoped to prevent. We
still will have to maintain and debug these dependencies and if the
OpenStack community is the only ones doing so then the language
fragmentation costs will come in to play. Our goal should be to only pay
these costs when necessary, and I am not sure this type of distinction
helps us do that.

Additionally, I am not sure how this would solve the swift case. The
object server is the component that has been rewritten so either it
would need to live outside of OpenStack with the rest of Swift being
part of OpenStack, or all of Swift would need to move outside. Neither
of these seems ideal.

This makes me wonder - does this decision need to be an all-or-nothing
thing? Can we say that the swift object server is a special case, let it
be written in another language, and leave it at that? My thinking is -
there is only value in us deciding on and allowing an additional
language for solving these issues generally if there are other places
that require this new language. It has been mentioned that Gnocchi could
potentially be a case in the future but until that problem actually
comes up there isn't much benefit to us deciding on another language
anyone can use and there are plenty of downsides to doing so.


I really do not want to "special case" swift. It really doesn't go with the spirit of inclusion.

I am not sure how inclusion is related to special casing. Inclusion here implies that some group is not being accepted in to our community. The excluded group here would be one writing software not in python, which is the same groups being excluded currently. This is an argument against the status quo, not against special casing.

I expect we will see a continued drive for whichever additional languages are supported once it's in place/allowed.

That is the problem. The social and debugging costs of adding another language are a function of how much that other language is used. If one component of one project is written in another language then these costs should be fairly low. I agree that once we allow another language we should expect many projects to begin using it, and IMO most if not all of these cases except swift will not be warranted. Thus, we will be paying the costs in development resources from several standpoints without cause.

I also worry "special casing" drives at a core thing we've been trying to dissuade: Swift is not a special case and is part of OpenStack. It should be considered as such and adding more onto that line just divides the community further.

I think that filling the role of "native optimizations that are inappropriate in python" is something we will continue to have requests for especially if one project has blessing to use this native-tool-optimization-thing.


This is my main disagreement with the all-or-nothing premise - we seem to have a bad case of fallacy of composition. AFAIK we do not have other components of OpenStack which are needing to do a read on a large number of on disk files in parallel from python, let alone do it in a highly performant way. Just because we have identified one non trivial problem for a language does not mean we have lots more to come. It is entirely possible that another project will have this same problem or we will identify another problem in the future which requires a similar solution, but I don't know why we expect to or even want to solve that issue now before that case is clearly defined. Hopefully we don't have the need to write more databases as part of OpenStack...

So in short, lets not special case a project here.

Cheers,
Greg

Cheers,
Greg
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160524/f43c235e/attachment.html>