Open Stack

Hello to All.

This message is not about arguing weather OpenStack needs Go and other
language.

This is a good discussion. So, the main question here is "Go along with
Python for OpenStack" and the problem is to support Go code starting for
necessity of skilled Go developers to infrastructure for CI/CD, etc.

Correct me if i'm wrong, none of the messages above were stating about
support Go-extensions for Python (C extensions were mentioned couple
times). Starting Go v1.5 it is possible to develop extension for Python [1]
(lib that helps to develop extensions [2])
An idea is in:
  - "If think that your project is an exceptional one (swift, designate,
etc.) and you really think that Golang is what you need,
     then why can't you develop your own Go-extensions  and write Python
libs that are utilizing that code
     and then add that new python dependency to your project?"
  - distribute your go-extensions (*.so files) and DEB/RPM for further
consumption in DevStack for example (like we do for multiple components -
Kafka, Cassandra, MySQL, RMQ, etc.)

As i can see such behaviour would allow Python to be main lang for
OpenStack development, we wouldn't have an overhead for building new
infrastructure for Go and would allow projects to use Go for developing
their extensions out of OpenStack Big Tent.

[1] https://blog.filippo.io/building-python-modules-with-go-1-5/
[2] https://github.com/sbinet/go-python

Kind regards,
Denys Makogon


2016-05-25 14:21 GMT+03:00 Flavio Percoco <flavio at redhat.com>:

> On 25/05/16 06:48 -0400, Sean Dague wrote:
>
> [snip]
>
>
> 4. Do we want to be in the business of building data plane services that
>> will all run into python limitations, and will all need to be rewritten
>> in another language?
>>
>> This is a slightly different spin on the question Thierry is asking.
>>
>> Control Plane services are very unlikely to ever hit a scaling concern
>> where rewriting the service in another language is needed for
>> performance issues. These are orchestrators, and the time spent in them
>> is vastly less than the operations they trigger (start a vm, configure a
>> switch, boot a database server). There was a whole lot of talk in the
>> threads of "well that's not innovative, no one will want to do just
>> that", which seems weird, because that's most of OpenStack. And it's
>> pretty much where all the effort in the containers space is right now,
>> with a new container fleet manager every couple of weeks. So thinking
>> that this is a boring problem no one wants to solve, doesn't hold water
>> with me.
>>
>> Data Plane services seem like they will all end up in the boat of
>> "python is not fast enough". Be it serving data from disk, mass DNS
>> transfers, time series database, message queues. They will all
>> eventually hit the python wall. Swift hit it first because of the
>> maturity of the project and they are now focused on this kind of
>> optimization, as that's what their user base demands. However I think
>> all other data plane services will hit this as well.
>>
>> Glance (which is partially a data plane service) did hit this limit, and
>> the way it is largely mitigated by folks is by using Ceph and exposing
>> that
>> directly to Nova so now Glance is only in the location game and metadata
>> game, and Ceph is in the data plane game.
>>
>
> Sorry for nitpicking here but Glance's API keeps being a data API. Sure it
> stores locations and sure you can do fancy things with those locations
> but, as
> far as end users go, it's still a data API. It is not be used as
> intensively as
> Swift's, though. Ceph's driver allows for fancier things to be done but
> there
> are deployments which don't use Ceph.
>
> I believe it'd be better to separate data services that *own* the data from
> those that integrate other backends. Swift owns the data. You upload it to
> swift, it stores the data using its own strategies and it serves it.
> Glance gets
> the data, puts it in some other store and then you can either access it
> (not
> always) directly from the store or have Glance serving it back.
>
>
> When it comes to doing data plan services in OpenStack, I'm quite mixed.
>> The technology concerns for data plane
>> services are quite different. All the control plane services kind of
>> look and feel the same. An API + worker model, a DB for state, message
>> passing / rpc to put work to the workers. This is a common pattern and
>> is something which even for all the project differences, does end up
>> kind of common between parts. Projects that follow this model are
>> debuggable as a group not too badly.
>>
>> 5. Where does Swift fit?
>>
>> This I think has always been a tension point in the community (at least
>> since I joined in 2012). Swift is an original service of OpenStack, as
>> it started as Swift and Nova. But they were very different things. Swift
>> is a data service, Nova was a control plane. Much of what is now
>> OpenStack is Nova derivative in some way (some times direct extractions
>> (Glance, Cinder, Ironic), some times convergent paths (Neutron). And
>> then with that many examples, lots of other things built in similar ways.
>>
>> Swift doesn't use common oslo components. That actually makes debugging
>> it quite different compared to the rest of OpenStack. The lack of
>> oslo.log means structured JSON log messages to Elastic Search, are not
>> a thing. Swift has a very different model in it's service split.
>> Swift doesn't use global requirements. Swift ensures it can run without
>> Keystone, because their goal is Swift everywhere, whether or not it's
>> part of the rest of OpenStack.
>>
>> These are all fine goals, but they definitely have led to tensions on
>> all sides.
>>
>> And I think part of the question is "are these tensions that need to be
>> solved" or "is this data that this thing is different". Which isn't to
>> say that Swift is bad, it's just definitively different than much of the
>> ecosystem. Maybe Swift should be graduated beyond OpenStack, because
>> it's scope cross cuts much differently. Ceph isn't part of OpenStack,
>> but it's in 50% of installs. libvirt isn't part of OpenStack, but it's
>> in 90% of installs. And in both of those cases OpenStack is one of the
>> biggest drivers of their use.
>>
>> Which, gets contentious because people feel like this is kicking
>> something out. And that I can understand. There is a lot of emotion
>> wrapped up in labels and who gets to be on the the OpenStack home page.
>> I wish there wasn't. Good software should get deployed because it is
>> good and solves a need, not because of labels. I'm not sure Swift users
>> really care that Swift is OpenStack. They care that Swift is Swift. And
>> Swift being Swift, but not being OpenStack would open up degrees of
>> freedom in Swift being more Swift centric without the same friction
>> from the rest of of the ecosystem.
>>
>> ...
>>
>> Honestly, we probably need to just address #5 first and foremost. It's
>> been dodged around for a long time, the tensions haven't gotten any
>> better.
>> We have just decided the standoff around Swift being different is going
>> to remain.
>> The global requirements proposal is a good example of that
>> https://review.openstack.org/#/c/88736/ - 434 revisions later.
>>
>
> Do we? I don't think this is a *swift* problem. I think this is a more
> generic
> problem that came as part of the big tent adoption. Swift has been around
> longer
> than any other project, sure. But I don't believe "fixing swift" will
> actually
> get us anywhere but to the next discussion about the shiny new data
> service that
> needs Go and we don't know whether it's OpenStack or not.
>
> If we don't organize "the tent" properly, I don't think we'll be able to do
> something about projects similar to Swift. Let's solve the issue of what
> this
> tent should look like now, what layers we need (if we need any), what other
> grouping we should have (if we must have them) before we start pushing
> projects
> aside.
>
> So, I believe we should address #4 first but from a different perspective.
> That
> is, we're *already* in the business of building these projects because
> there's
> nothing preventing them from being part of the tent. Let's identify what
> these
> projects are, let's reorganize the tent accordingly and find a proper
> place for
> these projects. Is that place the tent? or is it somewhere else? Dunno yet.
>
> From that we have one data point on #4. Are we going to be building a
>> set of data plane services that can't be in python? If Swift wasn't the
>> leading
>> example here would we be having the conversation at all?
>>
>
> This would have happened anyway. There are other services that could have
> similar requirements as Swift. Designate is one. I think Zaqar could also
> have
> such requirements in the API but it'd still be possible to delay such
> scenario.
>
>
> Flavio
>
> --
> @flaper87
> Flavio Percoco
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160525/0a3e813d/attachment.html>