Open Stack

I was thinking that we could document the information about sudo and
iproute2 patches with the upcoming release.  How would I go about
doing this?  Is there any section in our documentation about OS level
tweaks or requirements such as these that could present this
information as part of the release?

Carl

On Wed, Mar 5, 2014 at 9:58 AM, Rick Jones <rick.jones2 at hp.com> wrote:
> On 03/05/2014 06:42 AM, Miguel Angel Ajo wrote:
>>
>>
>>      Hello,
>>
>>      Recently, I found a serious issue about network-nodes startup time,
>> neutron-rootwrap eats a lot of cpu cycles, much more than the processes
>> it's wrapping itself.
>>
>>      On a database with 1 public network, 192 private networks, 192
>> routers, and 192 nano VMs, with OVS plugin:
>>
>>
>> Network node setup time (rootwrap): 24 minutes
>> Network node setup time (sudo):     10 minutes
>
>
> I've not been looking at rootwrap, but have been looking at sudo and ip.
> (Using some scripts which create "fake routers" so I could look without any
> of this icky OpenStack stuff in the way :) ) The Ubuntu 12.04 versions of
> each at least will enumerate all the interfaces on the system, even though
> they don't need to.
>
> There was already an upstream change to 'ip' that eliminates the unnecessary
> enumeration.  In the last few weeks an enhancement went into the upstream
> sudo that allows one to configure sudo to not do the same thing.   Down in
> the low(ish) three figures of interfaces it may not be a Big Deal (tm) but
> as one starts to go beyond that...
>
> commit f0124b0f0aa0e5b9288114eb8e6ff9b4f8c33ec8
> Author: Stephen Hemminger <stephen at networkplumber.org>
> Date:   Thu Mar 28 15:17:47 2013 -0700
>
>     ip: remove unnecessary ll_init_map
>
>     Don't call ll_init_map on modify operations
>     Saves significant overhead with 1000's of devices.
>
> http://www.sudo.ws/pipermail/sudo-workers/2014-January/000826.html
>
> Whether your environment already has the 'ip' change I don't know, but odd
> are probably pretty good it doesn't have the sudo enhancement.
>
>
>>     That's the time since you reboot a network node, until all namespaces
>> and services are restored.
>
>
> So, that includes the time for the system to go down and reboot, not just
> the time it takes to rebuild once rebuilding starts?
>
>
>>     If you see appendix "1", this extra 14min overhead, matches with the
>> fact that rootwrap needs 0.3s to start, and launch a system command
>> (once filtered).
>>
>>      14minutes =  840 s.
>>      (840s. / 192 resources)/0.3s ~= 15 operations /
>> resource(qdhcp+qrouter) (iptables, ovs port creation & tagging, starting
>> child processes, etc..)
>>
>>     The overhead comes from python startup time + rootwrap loading.
>
>
> How much of the time is python startup time?  I assume that would be all the
> "find this lib, find that lib" stuff one sees in a system call trace?  I saw
> a boatload of that at one point but didn't quite feel like wading into that
> at the time.
>
>
>>     I suppose that rootwrap was designed for lower amount of system
>> calls (nova?).
>
>
> And/or a smaller environment perhaps.
>
>
>>     And, I understand what rootwrap provides, a level of filtering that
>> sudo cannot offer. But it raises some question:
>>
>> 1) It's actually someone using rootwrap in production?
>>
>> 2) What alternatives can we think about to improve this situation.
>>
>>     0) already being done: coalescing system calls. But I'm unsure
>> that's enough. (if we coalesce 15 calls to 3 on this system we get:
>> 192*3*0.3/60 ~=3 minutes overhead on a 10min operation).
>
>
> It may not be sufficient, but it is (IMO) certainly necessary.  It will make
> any work that minimizes or eliminates the overhead of rootwrap look that
> much better.
>
>
>>     a) Rewriting rules into sudo (to the extent that it's possible), and
>> live with that.
>>     b) How secure is neutron about command injection to that point? How
>> much is user input filtered on the API calls?
>>     c) Even if "b" is ok , I suppose that if the DB gets compromised,
>> that could lead to command injection.
>>
>>     d) Re-writing rootwrap into C (it's 600 python LOCs now).
>>
>>     e) Doing the command filtering at neutron-side, as a library and
>> live with sudo with simple filtering. (we kill the python/rootwrap
>> startup overhead).
>>
>> 3) I also find 10 minutes a long time to setup 192 networks/basic tenant
>> structures, I wonder if that time could be reduced by conversion
>> of system process calls into system library calls (I know we don't have
>> libraries for iproute, iptables?, and many other things... but it's a
>> problem that's probably worth looking at.)
>
>
> Certainly going back and forth creating short-lived processes is at least
> anti-social and perhaps ever so slightly upsetting to the process scheduler.
> Particularly "at scale."  The/a problem is though that the Linux networking
> folks have been somewhat reticent about creating libraries (at least any
> that they would end-up supporting) because they have a concern it will
> lock-in interfaces and reduce their freedom of movement.
>
> happy benchmarking,
>
> rick jones
> the fastest procedure call is the one you never make
>
>
>>
>> Best,
>> Miguel Ángel Ajo.
>>
>>
>> Appendix:
>>
>> [1] Analyzing overhead:
>>
>> [root at rhos4-neutron2 ~]# echo "int main() { return 0; }" > test.c
>> [root at rhos4-neutron2 ~]# gcc test.c -o test
>> [root at rhos4-neutron2 ~]# time test      # to time process invocation on
>> this machine
>>
>> real    0m0.000s
>> user    0m0.000s
>> sys    0m0.000s
>>
>>
>> [root at rhos4-neutron2 ~]# time sudo bash -c 'exit 0'
>>
>> real    0m0.032s
>> user    0m0.010s
>> sys    0m0.019s
>>
>>
>> [root at rhos4-neutron2 ~]# time python -c'import sys;sys.exit(0)'
>>
>> real    0m0.057s
>> user    0m0.016s
>> sys    0m0.011s
>>
>> [root at rhos4-neutron2 ~]# time neutron-rootwrap --help
>> /usr/bin/neutron-rootwrap: No command specified
>>
>> real    0m0.309s
>> user    0m0.128s
>> sys    0m0.037s
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev