Open Stack

Hi,

This is the follow-up of the discussion we started yesterday on rootwrap
usage at the cross-project meeting.

A bit of history to stage this story first. OpenStack nodes sometimes
need to run things with elevated privileges. Nova started out as calling
sudo shell commands to execute those, which basically allowed it to run
anything as root. Distributions didn't really like that, so packagers
started to ship sudoers files that restricted the commands Nova could
run with sudo. It created a maintenance nightmare, where the code and
the packaging were routinely out of sync. Rootwrap was designed so that
we would maintain the allowed commands in the code itself, which
facilitated maintenance. It would also allow more complex rules than
sudoers allowed, like the ability to only kill a certain executable
processes.

Rootwrap was then adopted by Cinder and Neutron, and moved to oslo to
avoid code duplication. There were still two long-standing problems with
it, though.

The first one is performance -- each call would spawn a Python
interpreter which would then call the system command. This was fine when
there were just a few calls here and there, not so much when it's called
a hundred times in a row. During the Juno cycle, a daemon mode was added
to solve this issue. It is significantly faster than running sudo
directly (the often-suggested alternative). Projects still have to start
adopting it though. Neutron and Cinder have started work to do that in Kilo.

The second problem is the quality of the filter definitions. Rootwrap is
a framework to enable isolation. It's only as good as the filters each
project defines. Most of them rely on CommandFilters that do not check
any argument, instead of using more powerful filters (which are arguably
more painful to maintain). Developers routinely add filter definitions
that basically remove any isolation that might have been there, like
allowing blank dd, tee, chown or chmod.

Basically, some nodes run such a variety of commands as root that
maintaining proper isolation is a non-trivial amount of work: that's the
case for Nova's compute nodes and Cinder's volume nodes. I would argue
that for all other types of nodes, rootwrap is still an appropriate way
to provide isolation. But for nodes which need to make a very wide
variety of root calls, we pretend to isolate, but the filter definitions
are so full of holes that we actually don't.

What solutions do we have ?

(1) we could get our act together and audit and fix those filter
definitions. Remove superfluous usage of root rights, make use of
advanced filters for where we actually need them. We have been preaching
for that at many many design summits. This is a lot of work though...
There were such efforts in the past, but they were never completed for
some types of nodes. Worse, the bad filter definitions kept coming back,
since developers take shortcuts, reviewers may not have sufficient
security awareness to detect crappy filter definitions, and I don't
think we can design a gate test that would have such awareness.

(2) bite the bullet and accept that some types of nodes actually need
root rights for so many different things, they should just run as root
anyway. I know a few distributions which won't be very pleased by such a
prospect, but that would be a more honest approach (rather than claiming
we provide efficient isolation when we really don't). An added benefit
is that we could replace a number of shell calls by Python code, which
would simplify the code and increase performance.

(3) intermediary solution where we would run as the nova user but run
sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
That would leave it up to distros to choose between a blanket sudoer or
maintain their own filtering rules. I think it's a bit hypocritical
though (pretend the distros could filter if they wanted it, when we
dropped the towel on doing that ourselves). I'm also not convinced it's
more secure than solution 2, and it prevents from reducing the number of
shell-outs, which I think is a worthy idea.

In all cases I would not drop the baby with the bath water, and keep
rootwrap for all the cases where root rights are needed on a very
specific set of commands (like neutron, or nova's api-metadata). The
daemon mode should address the performance issue for the projects making
a lot of calls.

-- 
Thierry Carrez (ttx)