Open Stack

On 10/05/2016 01:01, Gregory Haynes wrote:
>
> On Mon, May 9, 2016, at 03:54 PM, John Dickinson wrote:
>> On 9 May 2016, at 13:16, Gregory Haynes wrote:
>>>
>>> This is a bit of an aside but I am sure others are wondering the same
>>> thing - Is there some info (specs/etherpad/ML thread/etc) that has more
>>> details on the bottleneck you're running in to? Given that the only
>>> clients of your service are the public facing DNS servers I am now even
>>> more surprised that you're hitting a python-inherent bottleneck.
>>
>> In Swift's case, the summary is that it's hard[0] to write a network
>> service in Python that shuffles data between the network and a block
>> device (hard drive) and effectively utilizes all of the hardware
>> available. So far, we've done very well by fork()'ing child processes,
>> using cooperative concurrency via eventlet, and basic "write more
>> efficient code" optimizations. However, when it comes down to it,
>> managing all of the async operations across many cores and many drives
>> is really hard, and there just isn't a good, efficient interface for
>> that in Python.
>
> This is a pretty big difference from hitting an unsolvable performance
> issue in the language and instead is a case of language preference -
> which is fine. I don't really want to fall in to the language-comparison
> trap, but I think more detailed reasoning for why it is preferable over
> python in specific use cases we have hit is good info to include /
> discuss in the document you're drafting :). Essentially its a matter of
> weighing the costs (which lots of people have hit on so I won't) with
> the potential benefits and so unless the benefits are made very clear
> (especially if those benefits are technical) its pretty hard to evaluate
> IMO.
>
> There seemed to be an assumption in some of the designate rewrite posts
> that there is some language-inherent performance issue causing a
> bottleneck. If this does actually exist then that is a good reason for
> rewriting in another language and is something that would be very useful
> to clearly document as a case where we support this type of thing. I am
> highly suspicious that this is the case though, but I am trying hard to
> keep an open mind...

The way this component works makes it quite difficult to make any major
improvement.

MiniDNS (the component) takes data and sends a zone transfer every time 
a recordset gets updated. That is a full (AXFR) zone transfer, so every
record in the zone gets sent to each of the DNS servers that end users
can hit.

This can be quite a large number - ns[1-6].example.com. may well be
tens or hundreds of servers behind anycast IPs and load balancers.

In many cases, internal zones (or even external zones) can be quite
large - I have seen zones that are 200-300Mb. If a zone is high traffic
(like say cloud.example.com. where a record is added / removed for
each boot / destroy, or the reverse DNS zones for a cloud), there can
be a lot of data sent out from this component.

We are a small development team, and after looking at our options, and 
judging the amount of developer hours we had available, a different
language was the route we decided on. I was going to go implement a few
POCs and see what was most suitable.

Golang was then being proposed as a new "blessed" language, and as it
was a language that we had a pre-existing POC in we decided to keep
this within the potential new list of languages.

As I said before, we did not just randomly decide this. We have been
talking about it for a while, and at this summit we dedicated an entire
session to it, and decided to do it.


>>
>> Initial results from a golang reimplementation of the object server in
>> Python are very positive[1]. We're not proposing to rewrite Swift
>> entirely in Golang. Specifically, we're looking at improving object
>> replication time in Swift. This service must discover what data is on
>> a drive, talk to other servers in the cluster about what they have,
>> and coordinate any data sync process that's needed.
>>
>> [0] Hard, not impossible. Of course, given enough time, we can do
>>   anything in a Turing-complete language, right? But we're not talking
>>   about possible, we're talking about efficient tools for the job at
>>   hand.
>
> Sorry to be a pedant but that is just plain false - there are plenty of
> intractable problems.
>
>>
>> [1] http://d.not.mn/python_vs_golang_gets.png and
>>   http://d.not.mn/python_vs_golang_puts.png
>>
>>
>> --John
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>