[openstack-dev] [neutron][rootwrap] Performance considerations, sudo?
Monty Taylor
mordred at inaugust.com
Thu Mar 20 11:32:16 UTC 2014
On 03/20/2014 05:31 AM, Miguel Angel Ajo wrote:
>
>
> On 03/19/2014 10:54 PM, Joe Gordon wrote:
>>
>>
>>
>> On Wed, Mar 19, 2014 at 9:25 AM, Miguel Angel Ajo <majopela at redhat.com
>> <mailto:majopela at redhat.com>> wrote:
>>
>>
>>
>> An advance on the changes that it's requiring to have a
>> py->c++ compiled rootwrap as a mitigation POC for havana/icehouse.
>>
>>
>> https://github.com/mangelajo/__shedskin.rootwrap/commit/__e4167a6491dfbc71e2d0f6e28ba93b__c8a1dd66c0
>>
>>
>> <https://github.com/mangelajo/shedskin.rootwrap/commit/e4167a6491dfbc71e2d0f6e28ba93bc8a1dd66c0>
>>
>>
>> The current translation output is included.
The C++ code it's generating is inefficient and subject to some issues.
I'd be a bit concerned about it and would want to make some fixes.
Also - I'm sure you know this, but it's labeled as experimental. As
something that is going to run as root, I'd want to actually audit all
of the code it does. WHICH MEANS - we'd be auditing/examining C++ code.
If we're going to do that, I'd rather just convert it to C++ by hand,
audit that, and deal with it. Either way, we'll need a substantial
amount of infra support - such as multi-platform compilation testing,
valgrind, etc.
Not a blocker - but things to think about. If you're really serious
about bringing C++ into the mix - let's just do it.
(ps. if we're going to do that, we could also just write a sudo plugin)
>> It looks like doable (almost killed 80% of the translation problems),
>> but there are two big stones:
>>
>> 1) As Joe said, no support for Subprocess (we're interested in
>> popen),
>> I'm using a dummy os.system() for the test.
>>
>> 2) No logging support.
>>
>> I'm not sure on how complicated could be getting those modules
>> implemented for shedkin.
>>
>>
>> Before sorting out of we can get those working under shedskin, any
>> preliminary performance numbers from neutron when using this?
>>
>
> Sure, totally agree.
>
> I'm trying to put up a conversion without 1 & 2, to run a benchmark on
> it, and then I'll post the results.
>
> I suppose, we couldn't use it in neutron itself without Popen support
> (not sure) but at least I could get an estimate out of the previous
> numbers and the new ones.
>
> Best,
> Miguel Ángel.
>>
>>
>> On 03/18/2014 09:14 AM, Miguel Angel Ajo wrote:
>>
>> Hi Joe, thank you very much for the positive feedback,
>>
>> I plan to spend a day during this week on the
>> shedskin-compatibility
>> for rootwrap (I'll branch it, and tune/cut down as necessary) to
>> make
>> it compile under shedskin [1] : nothing done yet.
>>
>> It's a short-term alternative until we can have a rootwrap
>> agent,
>> together with it's integration in neutron (for Juno). As, for the
>> compiled rootwrap, if it works, and if it does look good
>> (security wise)
>> then we'd have a solution for Icehouse/Havana.
>>
>> help in [1] is really welcome ;-) I'm available in
>> #openstack-neutron
>> as 'ajo'.
>>
>> Best regards,
>> Miguel Ángel.
>>
>> [1] https://github.com/mangelajo/__shedskin.rootwrap
>> <https://github.com/mangelajo/shedskin.rootwrap>
>>
>> On 03/18/2014 12:48 AM, Joe Gordon wrote:
>>
>>
>>
>>
>> On Tue, Mar 11, 2014 at 1:46 AM, Miguel Angel Ajo Pelayo
>> <mangelajo at redhat.com <mailto:mangelajo at redhat.com>
>> <mailto:mangelajo at redhat.com <mailto:mangelajo at redhat.com>>>
>> wrote:
>>
>>
>> I have included on the etherpad, the option to write
>> a sudo
>> plugin (or several), specific for openstack.
>>
>>
>> And this is a test with shedskin, I suppose that in
>> more complicated
>> dependecy scenarios it should perform better.
>>
>> [majopela at redcylon tmp]$ cat <<EOF >test.py
>> > import sys
>> > print "hello world"
>> > sys.exit(0)
>> > EOF
>>
>> [majopela at redcylon tmp]$ time python test.py
>> hello world
>>
>> real 0m0.016s
>> user 0m0.015s
>> sys 0m0.001s
>>
>>
>>
>> This looks very promising!
>>
>> A few gotchas:
>>
>> * Very limited library support
>>
>> https://code.google.com/p/__shedskin/wiki/docs#Library___Limitations
>>
>> <https://code.google.com/p/shedskin/wiki/docs#Library_Limitations>
>> * no logging
>> * no six
>> * no subprocess
>>
>> * no *args support
>> *
>>
>> https://code.google.com/p/__shedskin/wiki/docs#Python___Subset_Restrictions
>>
>>
>> <https://code.google.com/p/shedskin/wiki/docs#Python_Subset_Restrictions>
>>
>> that being said I did a quick comparison with great results:
>>
>> $ cat tmp.sh
>> #!/usr/bin/env bash
>> echo $0 $@
>> ip a
>>
>> $ time ./tmp.sh foo bar> /dev/null
>>
>> real 0m0.009s
>> user 0m0.003s
>> sys 0m0.006s
>>
>>
>>
>> $ cat tmp.py
>> #!/usr/bin/env python
>> import os
>> import sys
>>
>> print sys.argv
>> print os.system("ip a")
>>
>> $ time ./tmp.py foo bar > /dev/null
>>
>> min:
>> real 0m0.016s
>> user 0m0.004s
>> sys 0m0.012s
>>
>> max:
>> real 0m0.038s
>> user 0m0.016s
>> sys 0m0.020s
>>
>>
>>
>> shedskin tmp.py && make
>>
>>
>> $ time ./tmp foo bar > /dev/null
>>
>> real 0m0.010s
>> user 0m0.007s
>> sys 0m0.002s
>>
>>
>>
>> Based in these results I think a deeper dive into making
>> rootwrap
>> supportshedskin is worthwhile.
>>
>>
>>
>>
>>
>> [majopela at redcylon tmp]$ shedskin test.py
>> *** SHED SKIN Python-to-C++ Compiler 0.9.4 ***
>> Copyright 2005-2011 Mark Dufour; License GNU GPL
>> version 3 (See
>> LICENSE)
>>
>> [analyzing types..]
>> ******************************__**100%
>> [generating c++ code..]
>> [elapsed time: 1.59 seconds]
>> [majopela at redcylon tmp]$ make
>> g++ -O2 -march=native -Wno-deprecated -I.
>> -I/usr/lib/python2.7/site-__packages/shedskin/lib
>> /tmp/test.cpp
>> /usr/lib/python2.7/site-__packages/shedskin/lib/sys.cpp
>> /usr/lib/python2.7/site-__packages/shedskin/lib/re.cpp
>>
>>
>> /usr/lib/python2.7/site-__packages/shedskin/lib/builtin.__cpp -lgc
>> -lpcre -o test
>> [majopela at redcylon tmp]$ time ./test
>> hello world
>>
>> real 0m0.003s
>> user 0m0.000s
>> sys 0m0.002s
>>
>>
>> ----- Original Message -----
>> > We had this same issue with the dhcp-agent. Code was
>> added that
>> paralleled
>> > the initial sync here:
>> https://review.openstack.org/#__/c/28914/
>> <https://review.openstack.org/#/c/28914/>
>> that made
>> > things a good bit faster if I remember correctly.
>> Might be worth
>> doing
>> > something similar for the l3-agent.
>> >
>> > Best,
>> >
>> > Aaron
>> >
>> >
>> > On Mon, Mar 10, 2014 at 5:07 PM, Joe Gordon <
>> joe.gordon0 at gmail.com <mailto:joe.gordon0 at gmail.com>
>> <mailto:joe.gordon0 at gmail.com
>> <mailto:joe.gordon0 at gmail.com>> > wrote:
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Mar 10, 2014 at 3:57 PM, Joe Gordon <
>> joe.gordon0 at gmail.com <mailto:joe.gordon0 at gmail.com>
>> <mailto:joe.gordon0 at gmail.com
>> <mailto:joe.gordon0 at gmail.com>> > wrote:
>> >
>> >
>> >
>> > I looked into the python to C options and haven't
>> found anything
>> promising
>> > yet.
>> >
>> >
>> > I tried Cython, and RPython, on a trivial hello
>> world app, but
>> git similar
>> > startup times to standard python.
>> >
>> > The one thing that did work was adding a '-S' when
>> starting
>> python.
>> >
>> > -S Disable the import of the module site and the
>> site-dependent
>> manipulations
>> > of sys.path that it entails.
>> >
>> > Using 'python -S' didn't appear to help in devstack
>> >
>> > #!/usr/bin/python -S
>> > # PBR Generated from u'console_scripts'
>> >
>> > import sys
>> > import site
>> >
>> site.addsitedir('/mnt/stack/__oslo.rootwrap/oslo/rootwrap')
>> >
>> >
>> >
>> >
>> >
>> >
>> > I am not sure if we can do that for rootwrap.
>> >
>> >
>> > jogo at dev:~/tmp/pypy-2.2.1-src$ time ./tmp-c
>> > hello world
>> >
>> > real 0m0.021s
>> > user 0m0.000s
>> > sys 0m0.020s
>> > jogo at dev:~/tmp/pypy-2.2.1-src$ time ./tmp-c
>> > hello world
>> >
>> > real 0m0.021s
>> > user 0m0.000s
>> > sys 0m0.020s
>> > jogo at dev:~/tmp/pypy-2.2.1-src$ time python -S
>> ./tmp.py
>> > hello world
>> >
>> > real 0m0.010s
>> > user 0m0.000s
>> > sys 0m0.008s
>> >
>> > jogo at dev:~/tmp/pypy-2.2.1-src$ time python -S
>> ./tmp.py
>> > hello world
>> >
>> > real 0m0.010s
>> > user 0m0.000s
>> > sys 0m0.008s
>> >
>> >
>> >
>> > On Mon, Mar 10, 2014 at 3:26 PM, Miguel Angel Ajo
>> Pelayo <
>> > mangelajo at redhat.com <mailto:mangelajo at redhat.com>
>> <mailto:mangelajo at redhat.com <mailto:mangelajo at redhat.com>>
>> > wrote:
>> >
>> >
>> > Hi Carl, thank you, good idea.
>> >
>> > I started reviewing it, but I will do it more
>> carefully tomorrow
>> morning.
>> >
>> >
>> >
>> > ----- Original Message -----
>> > > All,
>> > >
>> > > I was writing down a summary of all of this and
>> decided to just
>> do it
>> > > on an etherpad. Will you help me capture the big
>> picture
>> there? I'd
>> > > like to come up with some actions this week to try
>> to address
>> at least
>> > > part of the problem before Icehouse releases.
>> > >
>> > >
>>
>> https://etherpad.openstack.__org/p/neutron-agent-exec-__performance
>>
>> <https://etherpad.openstack.org/p/neutron-agent-exec-performance>
>> > >
>> > > Carl
>> > >
>> > > On Mon, Mar 10, 2014 at 5:26 AM, Miguel Angel Ajo <
>> majopela at redhat.com <mailto:majopela at redhat.com>
>> <mailto:majopela at redhat.com <mailto:majopela at redhat.com>> >
>> > > wrote:
>> > > > Hi Yuri & Stephen, thanks a lot for the
>> clarification.
>> > > >
>> > > > I'm not familiar with unix domain sockets at low
>> level, but ,
>> I wonder
>> > > > if authentication could be achieved just with
>> permissions
>> (only users in
>> > > > group "neutron" or group "rootwrap" accessing
>> this service.
>> > > >
>> > > > I find it an interesting alternative, to the
>> other proposed
>> solutions,
>> > > > but
>> > > > there are some challenges associated with this
>> solution,
>> which could make
>> > > > it
>> > > > more complicated:
>> > > >
>> > > > 1) Access control, file system permission based
>> or token
>> based,
>> > > >
>> > > > 2) stdout/stderr/return encapsulation/forwarding
>> to the
>> caller,
>> > > > if we have a simple/fast RPC mechanism we can
>> use, it's a
>> matter
>> > > > of serializing a dictionary.
>> > > >
>> > > > 3) client side implementation for 1 + 2.
>> > > >
>> > > > 4) It would need to accept new domain socket
>> connections in
>> green threads
>> > > > to
>> > > > avoid spawning a new process to handle a new
>> connection.
>> > > >
>> > > > The advantages:
>> > > > * we wouldn't need to break the only-python-rule.
>> > > > * we don't need to rewrite/translate rootwrap.
>> > > >
>> > > > The disadvantages:
>> > > > * it needs changes on the client side (neutron +
>> other
>> projects),
>> > > >
>> > > >
>> > > > Cheers,
>> > > > Miguel Ángel.
>> > > >
>> > > >
>> > > >
>> > > > On 03/08/2014 07:09 AM, Yuriy Taraday wrote:
>> > > >>
>> > > >> On Fri, Mar 7, 2014 at 5:41 PM, Stephen Gran
>> > > >> < stephen.gran at theguardian.com
>> <mailto:stephen.gran at theguardian.com>
>> <mailto:stephen.gran at __theguardian.com
>> <mailto:stephen.gran at theguardian.com>> <mailto:
>> stephen.gran at theguardian.com
>> <mailto:stephen.gran at theguardian.com>
>> <mailto:stephen.gran at __theguardian.com
>> <mailto:stephen.gran at theguardian.com>> >>
>> > > >> wrote:
>> > > >>
>> > > >> Hi,
>> > > >>
>> > > >> Given that Yuriy says explicitly 'unix socket',
>> I dont
>> think he
>> > > >> means 'MQ' when he says 'RPC'. I think he just
>> means a daemon
>> > > >> listening on a unix socket for execution
>> requests. This
>> seems like
>> > > >> a reasonably sensible idea to me.
>> > > >>
>> > > >>
>> > > >> Yes, you're right.
>> > > >>
>> > > >> On 07/03/14 12:52, Miguel Angel Ajo wrote:
>> > > >>
>> > > >>
>> > > >> I thought of this option, but didn't consider
>> it, as It's
>> somehow
>> > > >> risky to expose an RPC end executing
>> priviledged (even
>> filtered)
>> > > >> commands.
>> > > >>
>> > > >>
>> > > >> subprocess module have some means to do RPC
>> securely over
>> UNIX sockets.
>> > > >> I does this by passing some token along with
>> messages. It
>> should be
>> > > >> secure because with UNIX sockets we don't need
>> anything
>> stronger since
>> > > >> MITM attacks are not possible.
>> > > >>
>> > > >> If I'm not wrong, once you have credentials for
>> messaging,
>> you can
>> > > >> send messages to any end, even filtered, I
>> somehow see
>> this as a
>> > > >> higher
>> > > >> risk option.
>> > > >>
>> > > >>
>> > > >> As Stephen noted, I'm not talking about using
>> MQ for RPC.
>> Just some
>> > > >> local UNIX socket with very simple RPC over it.
>> > > >>
>> > > >> And btw, if we add RPC in the middle, it's
>> possible that all
>> those
>> > > >> system call delays increase, or don't decrease
>> all it'll be
>> > > >> desirable.
>> > > >>
>> > > >>
>> > > >> Every call to rootwrap would require the
>> following.
>> > > >>
>> > > >> Client side:
>> > > >> - new client socket;
>> > > >> - one message sent;
>> > > >> - one message received.
>> > > >>
>> > > >> Server side:
>> > > >> - accepting new connection;
>> > > >> - one message received;
>> > > >> - one fork-exec;
>> > > >> - one message sent.
>> > > >>
>> > > >> This looks like way simpler than passing
>> through sudo and
>> rootwrap that
>> > > >> requires three exec's and whole lot of
>> configuration files
>> opened and
>> > > >> parsed.
>> > > >>
>> > > >> --
>> > > >>
>> > > >> Kind regards, Yuriy.
>> > > >>
>> > > >>
>> > > >>
>> _________________________________________________
>> > > >> OpenStack-dev mailing list
>> > > >> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> <mailto:OpenStack-dev at lists.__openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>>
>> > > >>
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> > > >>
>> > > >
>> > > > _________________________________________________
>> > > > OpenStack-dev mailing list
>> > > > OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> <mailto:OpenStack-dev at lists.__openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>>
>> > > >
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> > >
>> > > _________________________________________________
>> > > OpenStack-dev mailing list
>> > > OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> <mailto:OpenStack-dev at lists.__openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>>
>> > >
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> > >
>> >
>> > _________________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> <mailto:OpenStack-dev at lists.__openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>>
>> >
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> >
>> >
>> >
>> > _________________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> <mailto:OpenStack-dev at lists.__openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>>
>> >
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> >
>> >
>> >
>> > _________________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> <mailto:OpenStack-dev at lists.__openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>>
>> >
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> >
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> <mailto:OpenStack-dev at lists.__openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>>
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>>
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>> _________________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.__org
>> <mailto:OpenStack-dev at lists.openstack.org>
>>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list