Open Stack

On Fri, 24 Jun 2016 at 00:40 Sean Dague <sean at dague.net> wrote:

> On 06/23/2016 10:07 AM, Sean McGinnis wrote:
> > On Thu, Jun 23, 2016 at 12:19:34AM +0000, Angus Lees wrote:
> >> So how does rootwrap fit into the "theory of upgrade"?
> >>
> >> The doc talks about deprecating config, but is silent on when new
> required
> >> config (rootwrap filters) should be installed.  By virtue of the way the
> >> grenade code works (install N from scratch, attempt to run code from
> N+1),
> >> we effectively have a policy that any new configs are installed at some
> >> nebulous time *after* the N+1 code is deployed.  In particular, this
> means
> >> a new rootwrap filter needs to be merged a whole release before that
> >> rootwrap entry can be used - and anything else is treated as an
> "exception"
> >> (see for example the nova from-* scripts which have basically updated
> >> rootwrap for each release).
> >>
> >> --
> >>
> >> Stepping back, I feel like an "expand-contract" style upgrade process
> >> involving rootwrap should look something like
> >> 0. Update rootwrap to be the union of N and N+1 rootwrap filters,
> >> 1. Rolling update from N to N+1 code,
> >> 2. Remove N-only rootwrap entries.
> >>
> >> We could make that a bit easier for deployers by having a sensible
> >> deprecation policy that requires our own rootwrap filters for each
> release
> >> are already the union of this release and the last (which indeed is
> already
> >> our policy aiui), and then it would just look like:
> >> 0. Install rootwrap filters from N+1
> >> 1. Rolling update to N+1 code
> >
> > I think effectively this is what we've ended up with in the past.
> >
> > We've had this issue for some time. There have been several releases
> > where either Cinder drivers or os-brick changes have needed to add
> > rootwrap changes. Theoretically we _should_ have hit these problems long
> > ago.
> >
> > I think the only reason it hasn't come up before is that these changes
> > are usually for vendor storage backends. So they never got hit in
> > grenade tests since those use LVM. We have third party CI, but grenade
> > tests are not a part of that.
> >
> > The switch to privsep now has really highlighted this gap. I think we
> > need to make this implied constraint clear and have it documented. To
> > upgrade we will need to make sure the rootwrap filters are in place
> > _before_ we perform any upgrades.
>
> Are we going to have to do this for every service individually as it
> moves to privsep? Or is there a way we can do it common once, take the
> hit, and everyone moves forward?
>
> For instance, can we get oslo.rootwrap to make an exception, in code,
> for privsep-helper? Thereby not having to touch a config file in etc to
> roll forward.
>

Obviously, if we had nothing already in place then: no, there would be
nothing that our python code could do that would suddenly allow it to run
commands as root without some additional (sudo or similar) config by the
sysadmin (good!).

None of these are great, but:

Possibility 1:  Backdoor rootwrap

However if we assume rootwrap already exists then we _could_ rollout a new
version of oslo.rootwrap that contains a backdoor that allows
privsep-helper to be run as root for any context, without the need to
install a new rootwrap filter.

Disclaimers:

- It wouldn't work for virtualenvs, because the "privsep-helper" executable
won't be in sudo's usual PATH.

- Retro-fitting something like that to rootwrap feels like it's skirting
close to some sort of ethical contract we've made with admins regarding
rootwrap's featureset.  Not saying we shouldn't do it, just that we should
think about how an operator is going to feel about that.


Possibility 2: Wider rootwrap filter

In the past, I've been proposing rootwrap filters that match only specific
privsep "privileged contexts" by name.  On further reflection, if we're
assuming the existing python modules installed into root's python path are
already trustworthy (and we _are_ assuming that), then it might also be
reasonable to trust *any* privsep entrypoint declared within that module
path.  This gives a larger attack surface to think about (particularly if
python libraries including privsep decorators were installed for some
reason other than providing privsep entry points), but there's no reason
why this is _necessarily_ an issue.

This allows us to get to a single rootwrap filter per-project (or rather,
"per-rootwrap") since projects use separate rootwrap config directories -
so we would still have to do a thing once per project.


Possibility 3: Skip rootwrap, use just sudo

sudoers isn't very expressive - but we could install a new rootwrap-like
wrapper into sudoers once system-wide, which includes some sort of logic to
start privsep-helpers.  This could be as simple as a small shell script.
The advantage this has over rootwrap is that it would contain some sort of
system-wide config, rather than per-project.

Downsides

- Would still need to be installed once system-wide.

- Would need to be configured per-virtualenv, since otherwise we have no
way to know which virtualenvs should be given root powers.


Possibility 4: Run as root initially

Another option would be to follow the usual Unix daemon model: Start the
process with all required privileges, and avoid sudo/rootwrap entirely.

In this version, we take a once-off hit to tell everyone to start running
their OpenStack agents as root (probably from init/systemd), and right at
the top of main() we fork() the privsep-helper and then drop to a regular
uid.  No sudo or rootwrap ever (although the unprivileged code could
continue to use it while we clean up all the legacy code).

A glorious future, but still a big per-project deployment change that has
to be managed somehow.


...

About here I run out of ideas...  Any other suggestions/comments?

Note that all the above is about privsep specifically, whereas my original
mail related to any config that didn't have a suitable in-code default
(rootwrap filters being just an obvious example of that).  Regardless of
how the privsep-specific discussion goes, I feel like we should still talk
about when any such "required" config changes need to be deployed.

 - Gus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160624/f7b2052d/attachment.html>