[neutron][nova] Rootwrap daemon and privsep
Hey, If I believe both the nova and neutron documentation: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.use... https://docs.openstack.org/neutron/latest/configuration/neutron.html#agent.r... At scale, we are supposed to enable the rootwrap-daemon option. I have issues enabling that, but before going further on my issue, I'd like to understand the difference with privsep daemon. Is privsep a new daemon which is supposed to replace the rootwrap one? Is privsep being launch after rootwrap? IS privsep enabled by default, so I should not care about rootwrap at all? I'd like to understand more about that. Everything I try to find on popular search engine seems outdated, so if someone could give me a hand on finding the good page to understand that, I'l love it :p Cheers, -- Arnaud Morin
Adding [large scale SIG], sorry! -- Arnaud Morin On 29.11.19 - 15:53, Arnaud Morin wrote:
Hey,
If I believe both the nova and neutron documentation: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.use... https://docs.openstack.org/neutron/latest/configuration/neutron.html#agent.r...
At scale, we are supposed to enable the rootwrap-daemon option. I have issues enabling that, but before going further on my issue, I'd like to understand the difference with privsep daemon.
Is privsep a new daemon which is supposed to replace the rootwrap one? Is privsep being launch after rootwrap? IS privsep enabled by default, so I should not care about rootwrap at all?
I'd like to understand more about that. Everything I try to find on popular search engine seems outdated, so if someone could give me a hand on finding the good page to understand that, I'l love it :p
Cheers,
-- Arnaud Morin
Arnaud Morin wrote:
[...] I'd like to understand the difference with privsep daemon.
Is privsep a new daemon which is supposed to replace the rootwrap one? Is privsep being launch after rootwrap? IS privsep enabled by default, so I should not care about rootwrap at all?
I can help with that, since I originally created rootwrap. Rootwrap is a privilege escalation control mechanism. It serves as a way to filter what the service user on the machine can execute as root via sudo. The idea is that sudoers files do not provide enough granularity, so instead of trying to describe what is allowed and what is not in sudoers file, we basically allow calling "sudo rootwrap command" and let rootwrap figure it out. Rootwrap reads a number of (root-owned) configuration files and decides to allow calling the command or not. There are two issues with this mechanism. The first is that the performance is not great, as first you run a python executable (rootwrap), which in turn spawns another process if the command is allowed. If you do that for hundreds of "ip" calls as you set up networking in neutron, this can add up pretty fast. The second issue is that rootwrap is only as secure as its configuration. If for example you configure rootwrap to allow the 'nova' user to run the "chmod" command, well that's basically the same as allowing it run anything as root. You have to use advanced filters to further refine what it can actually do based on command parameter analysis, and there is only so much you can control that way. Rootwrap-daemon is a way to partially help with the first issue. Rather than calling a new rootwrap Python process every time a command needs to be called, you maintain a long-running rootwrap process that will process all requests. It significantly improves performance, but it adds inter-process communication complexity (never great in a security system). And it does nothing to address the second issue. Privsep is the "right" way of addressing both issues. Rather than having the code try to call shell commands as root, privsep allows the code to call Python functions as root. This solves the performance issue, as you don't have the overhead of a separate Python process + shell process every time you want to change the ownership of a file, you can just call a Python function that will call os.chown() and get ear to syscall efficiency. It also solves the granularity issue, by allowing to call a function that will only do what you want to do, rather than have to find a way to filter parameters so that the command you call cannot be abused to do other things. The main issue with privsep is that it requires changing the code. You have to set it up in every project (it's now done for most), but then every place the service calls utils.execute(command, run_as_root=True) needs to be changed to call a privileged Python function instead. The second issue with privsep is that it still needs root to start. The way this is usually done is by using rootwrap itself to bootstrap privsep... which can be confusing. There are obviously other ways to start the process as root, but since most of those services still make use of rootwrap anyway, that is what continues to be used for the initial bootstrapping. Ideally services would be completely transitioned to privsep, and we would discontinue rootwrap. Hoping this helps, -- Thierry Carrez (ttx)
Hello, Thanks Thierry, it helps a lot, things look clearer now. Still, one point is still confused in my mind. Now that we are supposed to use privsep in most cases, services (nova, neutron and others) are supposed to start some sort of privsep daemon (something called privsep-helper if I believe what I see running on my infra). With what you said, I suspect that this privsep-helper is bootstrapped through a rootwrap execution. First question is: is the daemon mode of rootwrap still needed? Maybe not if the whole nova code is using the privsep mechanism now? Maybe it is if some part of the code has not yet moved on privsep? Moreover, if I take nova for exemple, I can see two privsep daemon running on my compute node: - one with context nova.privsep.sys_admin_pctxt - second with context vif_plug_linux_bridge.privsep.vif_plug Do you know why is it like that? Thanks, -- Arnaud Morin On 29.11.19 - 18:17, Thierry Carrez wrote:
Arnaud Morin wrote:
[...] I'd like to understand the difference with privsep daemon.
Is privsep a new daemon which is supposed to replace the rootwrap one? Is privsep being launch after rootwrap? IS privsep enabled by default, so I should not care about rootwrap at all?
I can help with that, since I originally created rootwrap.
Rootwrap is a privilege escalation control mechanism. It serves as a way to filter what the service user on the machine can execute as root via sudo. The idea is that sudoers files do not provide enough granularity, so instead of trying to describe what is allowed and what is not in sudoers file, we basically allow calling "sudo rootwrap command" and let rootwrap figure it out. Rootwrap reads a number of (root-owned) configuration files and decides to allow calling the command or not.
There are two issues with this mechanism. The first is that the performance is not great, as first you run a python executable (rootwrap), which in turn spawns another process if the command is allowed. If you do that for hundreds of "ip" calls as you set up networking in neutron, this can add up pretty fast.
The second issue is that rootwrap is only as secure as its configuration. If for example you configure rootwrap to allow the 'nova' user to run the "chmod" command, well that's basically the same as allowing it run anything as root. You have to use advanced filters to further refine what it can actually do based on command parameter analysis, and there is only so much you can control that way.
Rootwrap-daemon is a way to partially help with the first issue. Rather than calling a new rootwrap Python process every time a command needs to be called, you maintain a long-running rootwrap process that will process all requests. It significantly improves performance, but it adds inter-process communication complexity (never great in a security system). And it does nothing to address the second issue.
Privsep is the "right" way of addressing both issues. Rather than having the code try to call shell commands as root, privsep allows the code to call Python functions as root. This solves the performance issue, as you don't have the overhead of a separate Python process + shell process every time you want to change the ownership of a file, you can just call a Python function that will call os.chown() and get ear to syscall efficiency. It also solves the granularity issue, by allowing to call a function that will only do what you want to do, rather than have to find a way to filter parameters so that the command you call cannot be abused to do other things.
The main issue with privsep is that it requires changing the code. You have to set it up in every project (it's now done for most), but then every place the service calls utils.execute(command, run_as_root=True) needs to be changed to call a privileged Python function instead.
The second issue with privsep is that it still needs root to start. The way this is usually done is by using rootwrap itself to bootstrap privsep... which can be confusing. There are obviously other ways to start the process as root, but since most of those services still make use of rootwrap anyway, that is what continues to be used for the initial bootstrapping.
Ideally services would be completely transitioned to privsep, and we would discontinue rootwrap.
Hoping this helps,
-- Thierry Carrez (ttx)
Arnaud Morin wrote:
[...] Still, one point is still confused in my mind.
Now that we are supposed to use privsep in most cases, services (nova, neutron and others) are supposed to start some sort of privsep daemon (something called privsep-helper if I believe what I see running on my infra). With what you said, I suspect that this privsep-helper is bootstrapped through a rootwrap execution.
Yes. It obviously depends on packaging and how they start services, but by default this is how privsep-helper is started.
First question is: is the daemon mode of rootwrap still needed? Maybe not if the whole nova code is using the privsep mechanism now? Maybe it is if some part of the code has not yet moved on privsep?
I'll defer to nova experts, but yes, it's a trade-off that depends on how much has already been migrated, and how often the remaining rootwrap commands are called. Looking at nova compute node it only has a couple of rootwrap filters left[1], but maybe the performance loss of dropping daemon mode there is acceptable. [1] https://opendev.org/openstack/nova/src/branch/master/etc/nova/rootwrap.d/com...
Moreover, if I take nova for exemple, I can see two privsep daemon running on my compute node: - one with context nova.privsep.sys_admin_pctxt - second with context vif_plug_linux_bridge.privsep.vif_plug
Do you know why is it like that?
Privsep is finer-grained than rootwrap. Rather than just being root or non-root, you can require specific capabilities. See [2] for an example of such context definition. Privsep-helper starts as root and drops to the required capabilities to run with the minimum-needed privileges. That requires separate executables, one for each context. Specific libraries (like os_brick) or plugins can also come with their own privsep context. [2] https://opendev.org/openstack/nova/src/branch/master/nova/privsep/__init__.p... Reading the initial spec[3] can give you extra context on how the change was driven. The current implementation has not strayed far away from that initial concept: [3] https://specs.openstack.org/openstack/oslo-specs/specs/liberty/privsep.html -- Thierry Carrez (ttx)
On 12/2/2019 5:19 AM, Thierry Carrez wrote:
I'll defer to nova experts, but yes, it's a trade-off that depends on how much has already been migrated, and how often the remaining rootwrap commands are called. Looking at nova compute node it only has a couple of rootwrap filters left[1], but maybe the performance loss of dropping daemon mode there is acceptable.
[1] https://opendev.org/openstack/nova/src/branch/master/etc/nova/rootwrap.d/com...
I want to say mikal converted everything native to nova from rootwrap to privsep and that was completed in Train: https://docs.openstack.org/releasenotes/nova/train.html#security-issues "The transition from rootwrap (or sudo) to privsep has been completed for nova. The only case where rootwrap is still used is to start privsep helpers. All other rootwrap configurations for nova may now be removed." Looking at what's in the compute.filters file looks like it's all stuff for os-brick, but I though os-brick was fully using privsep natively as well? Maybe it's just a matter of someone working on this TODO: https://opendev.org/openstack/nova/src/branch/master/etc/nova/rootwrap.d/com... -- Thanks, Matt
Matt Riedemann wrote:
[...] I want to say mikal converted everything native to nova from rootwrap to privsep and that was completed in Train:
https://docs.openstack.org/releasenotes/nova/train.html#security-issues
"The transition from rootwrap (or sudo) to privsep has been completed for nova. The only case where rootwrap is still used is to start privsep helpers. All other rootwrap configurations for nova may now be removed."
Looking at what's in the compute.filters file looks like it's all stuff for os-brick, but I though os-brick was fully using privsep natively as well? Maybe it's just a matter of someone working on this TODO:
https://opendev.org/openstack/nova/src/branch/master/etc/nova/rootwrap.d/com...
That's great news! I'll have a deeper look and propose changes if appropriate. Cheers, -- Thierry Carrez (ttx)
Hey, Thanks for the explanations! Le mar. 3 déc. 2019 à 10:43, Thierry Carrez <thierry@openstack.org> a écrit :
Matt Riedemann wrote:
[...] I want to say mikal converted everything native to nova from rootwrap to privsep and that was completed in Train:
https://docs.openstack.org/releasenotes/nova/train.html#security-issues
"The transition from rootwrap (or sudo) to privsep has been completed for nova. The only case where rootwrap is still used is to start privsep helpers. All other rootwrap configurations for nova may now be removed."
Looking at what's in the compute.filters file looks like it's all stuff for os-brick, but I though os-brick was fully using privsep natively as well? Maybe it's just a matter of someone working on this TODO:
https://opendev.org/openstack/nova/src/branch/master/etc/nova/rootwrap.d/com...
That's great news! I'll have a deeper look and propose changes if appropriate.
Cheers,
-- Thierry Carrez (ttx)
participants (3)
-
Arnaud Morin
-
Matt Riedemann
-
Thierry Carrez