Open Stack

On Mon, 9 May 2016 14:17:40 -0500
Edward Leafe <ed at leafe.com> wrote:

> Whenever I hear claims that Python is “too slow for X”, I wonder
> what’s so special about X that makes it so much more demanding than,
> say, serving up YouTube.

In case of Swift, the biggest issue was the scheduler. As RAX people
found, although we (OpenStack Swift community of implementors) were able
to obtain an acceptable baseline performance, a bad drive dragged down
the whole node. Before Hummingbird, dfg (and redbo) screwed around with
fully separate processes that provided isolation, but that was not
scaling well. So, there was an endless parade of solutions on the base
of threads. Some patches went in, some did not. At some points things
were so bad that dfg posted a patch, which maintained a scoring board
in an SQLite file. They were willing to add a bunch of I/O to every
request just to avoid the worst case that Python forced upon them.
The community (that is basically John, Sam, and I) put brakes on that.
But only at that point redbo created Hummingbird, which solved the
issue for them.

Once Hummingbird went into production, they found that it was easy to
polish and it could be much faster. Some of the benchmarks were
beating Python by 80 times. CPU consumption went way down, too.
But all that was secondary in the adoption of Go. If not a significant
scalability crisis in the field, Swift in Go would not have happened.

Scott Simpson gave a preso at Vancouver Summit that had some details and
benchmarks. Google is no help finding it online, unfortunately. Only
finds the panel discussion. Maybe someone had it saved.

-- Pete