Open Stack

There have been threads in the past about the slowness of the "openstack"
client tool such as this one by Sean last year:

  http://lists.openstack.org/pipermail/openstack-dev/2015-April/061317.html

Sean mentioned a 1.5s fixed overhead on openstack client, and mentions it
is significantly slower than the equivalent nova command. In my testing
I don't see any real speed difference between openstack & nova client
programs, so maybe that differential has been addressed since Sean's
original thread, or maybe nova has got slower.

Overall though, I find it is way too sluggish considering it is running
on a local machine with 12 cpus and 30 GB of RAM.

I had a quick go at trying to profile the tools with cprofile and analyse
with KCacheGrind as per this blog:

  https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara

And notice that in profiling 'nova help' for example, the big sink appears
to come from the 'pkg_resource' module and its use of pyparsing. I didn't
spend any real time to dig into this in detail, because it got me wondering
whether we can easily just avoid the big startup penalty by not having to
startup a new python interpretor for each command we run.

I traced devstack and saw it run 'openstack' and 'neutron' commands approx
140 times in my particular configuration. If each one of those has a 1.5s
overhead, we could potentially save 3 & 1/2 minutes off devstack execution
time.

So as a proof of concept I have created an 'openstack-server' command
which listens on a unix socket for requests and then invokes the
OpenStackShell.run / OpenStackComputeShell.main / NeutronShell.run
methods as appropriate.

I then replaced the 'openstack', 'nova' and 'neutron' commands with
versions that simply call to the 'openstack-server' service over the
UNIX socket. Since devstack will always recreate these commands in
/usr/bin, I simply put my replacements in $HOME/bin and then made
sure $HOME/bin was first in the $PATH

You might call this 'command line as a service' :-)

Anyhow, with my devstack setup a traditional install takes

  real	21m34.050s
  user	7m8.649s
  sys	1m57.865s

And when using openstack-server it only takes

  real	17m47.059s
  user	3m51.087s
  sys	1m42.428s

So that has cut 18% off the total running time for devstack, which
is quite considerable really.

I'm attaching the openstack-server & replacement openstack commands
so you can see what I did. You have to manually run the openstack-server
command ahead of time and it'll print out details of every command run
on stdout.

Anyway, I'm not personally planning to take this experiment any further.
I'll probably keep using this wrapper in my own local dev env since it
does cut down on devstack time significantly. This mail is just to see
if it'll stimulate any interesting discussion or motivate someone to
explore things further.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
-------------- next part --------------
#!/usr/bin/python

import socket
import sys
import os
import os.path
import json

server_address = "/tmp/openstack.sock"

sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

try:
    sock.connect(server_address)
except socket.error, msg:
    print >>sys.stderr, msg
    sys.exit(1)


def send(sock, doc):
    jdoc = json.dumps(doc)
    sock.send('%d\n' % len(jdoc))
    sock.sendall(jdoc)

def recv(sock):
    length_str = ''

    char = sock.recv(1)
    if len(char) == 0:
        print >>sys.stderr, "Unexpected end of file"
        sys.exit(1)

    while char != '\n':
        length_str += char
        char = sock.recv(1)
        if len(char) == 0:
            print >>sys.stderr, "Unexpected end of file"
            sys.exit(1)

    total = int(length_str)
        
    # use a memoryview to receive the data chunk by chunk efficiently
    jdoc = memoryview(bytearray(total))
    next_offset = 0
    while total - next_offset > 0:
        recv_size = sock.recv_into(jdoc[next_offset:], total - next_offset)
        next_offset += recv_size
    try:
        doc = json.loads(jdoc.tobytes())
    except (TypeError, ValueError), e:
        raise Exception('Data received was not in JSON format')
    return doc

try:
    env = {}
    passenv = ["CINDER_VERSION",
               "OS_AUTH_URL",
               "OS_IDENTITY_API_VERSION",
               "OS_NO_CACHE",
               "OS_PASSWORD",
               "OS_PROJECT_NAME",
               "OS_REGION_NAME",
               "OS_TENANT_NAME",
               "OS_USERNAME",
               "OS_VOLUME_API_VERSION"]
    for name in passenv:
        if name in os.environ:
            env[name] = os.environ[name]

    cmd = {
        "app": os.path.basename(sys.argv[0]),
        "env": env,
        "argv": sys.argv[1:]
    }
    send(sock, cmd)

    doc = recv(sock)
    if doc["stdout"] != '':
        print >>sys.stdout, doc["stdout"]
    if doc["stderr"] != '':
        print >>sys.stderr, doc["stderr"]
    sys.exit(doc["status"])
finally:
    sock.close()
-------------- next part --------------
#!/usr/bin/python

import socket
import sys
import os
import json
from cStringIO import StringIO

from openstackclient import shell as osc_shell
from novaclient import shell as nova_shell
from neutronclient import shell as neutron_shell

server_address = "/tmp/openstack.sock"

try:
    os.unlink(server_address)
except OSError:
    if os.path.exists(server_address):
	raise

sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
print >>sys.stderr, 'starting up on %s' % server_address
sock.bind(server_address)

# Listen for incoming connections
sock.listen(1)

def send(sock, doc):
    jdoc = json.dumps(doc)
    sock.send('%d\n' % len(jdoc))
    sock.sendall(jdoc)

def recv(sock):
    length_str = ''
    char = sock.recv(1)
    while char != '\n':
        length_str += char
        char = sock.recv(1)

    total = int(length_str)
        
    # use a memoryview to receive the data chunk by chunk efficiently
    jdoc = memoryview(bytearray(total))
    next_offset = 0
    while total - next_offset > 0:
        recv_size = sock.recv_into(jdoc[next_offset:], total - next_offset)
        next_offset += recv_size
    try:
        doc = json.loads(jdoc.tobytes())
    except (TypeError, ValueError), e:
        raise Exception('Data received was not in JSON format')
    return doc

while True:
    csock, client_address = sock.accept()
    try:
        doc = recv(csock)

        print >>sys.stderr, "%s %s" % (doc["app"], doc["argv"])
        oldenv = {}
        for name in doc["env"].keys():
            oldenv[name] = os.environ.get(name, None)
            os.environ[name] = doc["env"][name]

        try:
            old_stdout = sys.stdout
            old_stderr = sys.stderr
            my_stdout = sys.stdout = StringIO()
            my_stderr = sys.stderr = StringIO()

            class Exit(BaseException):
                def __init__(self, status):
                    self.status = status

            def noexit(stat):
                raise Exit(stat)

            sys.exit = noexit

            if doc["app"] == "openstack":
                sh = osc_shell.OpenStackShell()
                ret = sh.run(doc["argv"])
            elif doc["app"] == "nova":
                sh = nova_shell.OpenStackComputeShell()
                ret = sh.main(doc["argv"])
            elif doc["app"] == "neutron":
                sh = neutron_shell.NeutronShell(
                    neutron_shell.NEUTRON_API_VERSION)
                ret = sh.run(doc["argv"])
            else:
                print >>sys.stderr, "Unknown application %s" %doc["app"]
                ret = 1
        except Exit as e:
            ret = e.status
        finally:
            sys.stdout = old_stdout
            sys.stderr = old_stderr
            
            for name in oldenv.keys():
                if oldenv[name] is None:
                    del os.environ[name]
                else:
                    os.environ[name] = oldenv[name]
                    
        send(csock, {
            "stdout": my_stdout.getvalue(),
            "stderr": my_stderr.getvalue(),
            "status": ret,
        })

    except BaseException as e:
        print >>sys.stderr, e
        pass
    finally:
	csock.close()