On Thu, 2020-03-19 at 16:04 +0000, Sean Mooney wrote:

On Thu, 2020-03-19 at 16:50 +0100, Iury Gregory wrote:

...

Hi Rodolfo,

Thanks for raising this topic. I like the idea and I can work in the ironic part.

Em qui., 19 de mar. de 2020 às 16:47, Rodolfo Alonso <ralonsoh@redhat.com> escreveu:

...

Hello all:

With this mail I would like to propose the goal to move away from oslo.rootwrap and migrate to oslo.privsep. The last one offers a superior security model, faster and more secure. During the last cycles and since privsep was released, the Community has encouraged the usage of privsep and the deprecation of any existing code still using rootwrap. i had a rather long rant about why privsep is only more secure if you use it correctly i would say that today most project that use privsep are not as secure as you think they are or as secure as you would like them to be.

i would say that the privsep guide also is not entirely correct as it advocate groping privileged function in a singel module and gives an example of a context that has two many capabilities. we have the unfortunate situation that in nova and neuton a singel privsep context is used for all privadge function meaning it has the some of all posible capablites that any code path can require. that is an anti pattern. also we group privaldage funcition in a <project?>/privsep/... module which encurages function reuse which encurages widing the contract to make them more resuable which again is an anti patteren. this lead to function like nova.privsep.path.writefile(path, mode, data). which litally allows you to write to any file on the system with any mod and put any data. that breaks all the rules. first it has a wide contract by takeing both the path and mode. second its not got _privaladged in the fucntion name so we have to have a hacking check to prevent you doing "from nova.privsep import fs; ... fs.writefile(...)" without realising its privaldaged. thrid its grouped in privsep module with encurage it to be used form may part of the code base require int the wide contract, and finally becasue nova only has one privsep context capabilities=[capabilities.CAP_CHOWN, capabilities.CAP_DAC_OVERRIDE, capabilities.CAP_DAC_READ_SEARCH, capabilities.CAP_FOWNER, capabilities.CAP_NET_ADMIN, capabilities.CAP_SYS_ADMIN], instead of just the file capablies it also has cap_net_admin and cap_sys_admin so it can mess with sysfs and so other thing that would normaly be prevented if it only had the capablites required for the specalised fucntion it callse writefile. for exampel create a vm conosle log.... this has come up in the past on the mainlist and at at least two ptgs http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004358.htm... so in parallel to this project that already use privsep might need to have a second goal of use privsep properly. p.s. yes this is the less ranty version fo this mail :) i do think this is a good goal.

...

...

For any developer willing to collaborate, there are plenty of code examples, as I’ll provide later, implementing and using privsep for new methods and migrations.

If this goal is approved, I'll open a Story ( https://storyboard.openstack.org/) and any developer will be able to add a task for each patch or set of them related. This would be the tracker for this common effort.

PROJECTS TO MIGRATE. -------------------- Projects that are still using rootwrap: http://codesearch.openstack.org/?q=rootwrap&i=nope&files=.*.py&repos= neutron os-brick designate cinder ironic-inspector neutron-vpnaas nova

nova does not use rootwarp anymore. so it came up on irc that nova still has one use of rootwrap which is for os-bricks.

os-brick itself actully does nto use rootwap directly as of newton https://github.com/openstack/os-brick/commit/dbf77fba1061cb4e93b3db5f8117d6c... but instead rely on the calling agent (nova or cinder) to pass in a root_helper string with is passed to oslo_concurrency.processutils.execute in effect this is only used to spawn the privsep deamon process. out side of test code os-brick only usage of root_helper is when set via nova or cinder and all prviladge ecalation happens by calling a single function in os_brick.rootwap which does not use rootwap unless root_helper is set to use rootwap. so execute in there rootwrap module https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f2... only elevets to root via calling execute_root which gains privladges via privsep https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f2... so while os-brick shoudl definetly be looked at to make the usage of privsep actully more secure so that it does not simple pass eveythin to a single function that forad args to process utils, they dont actully need to do anyting to drop rootwrap. instead os_brick need to work on the second potential goal of use privsep correctly. i personally think the use of rootwap in nova for os-bircks provide almost no security benifit and would suggest we should remove it. once os-brick passes the command to execute it will be dispatch to the privesep process via the unix socket which will spawn a shell using oslo_concurrancy.process_utils.execute with CAP_SYS_ADMIN privaldges and no root wrap filtering will be applied at that point.

...

...

solum glance_store ironic zun magnum manila networking-bagpipe sahara ceilometer cinderlib freezer ironic-lib monasca-agent tacker tripleo-common

USAGE DOCUMENTATION ABOUT PRIVSEP. ---------------------------------- How to create a privsep context, assign privileges and use it as a decorator: https://docs.openstack.org/oslo.privsep/latest/user/index.html

HOW TO MIGRATE FROM ROOTWRAP TO PRIVSEP. ---------------------------------------- From the same link provided previously, in the section “Converting from rootwrap to privsep”:

https://docs.openstack.org/oslo.privsep/latest/user/index.html#converting-fr...

oslo.privsep provides a class, PrivContext, that can be used to create a decorator function. The instance created is a context of execution and has defined a list of capabilities, matching the Linux capabilities. The privsep context decorator should contain the minimum needed capabilities to execute the decorated function.

For example:

default = priv_context.PrivContext( __name__, cfg_section='privsep', pypath=__name__ + '.default', capabilities=[caps.CAP_SYS_ADMIN, caps.CAP_NET_ADMIN, caps.CAP_DAC_OVERRIDE, caps.CAP_DAC_READ_SEARCH, caps.CAP_SYS_PTRACE], ) actully i didnt comment on this before but ^ is a terable example. we should use several smllaer context and not one big one with multple capablities.

...

...

The function “entrypoint” of this instance can be used as a decorator for another function:

@privileged.default.entrypoint def delete_interface(ifname, namespace, **kwargs): _run_iproute_link("del", ifname, namespace, **kwargs)

As commented in the given link, a straight 1:1 filter:function replacement generally results in functions that are still too broad for good security. It is better to replace each chmod rootwrap call with a narrow privsep function that will limit it to specific files.

yep and that also means that you shoudl avoid groupign them in a <project>/privspec/<submodule> folder and instead put the spealised funtion int he module that uses it to avoid the tempation to make overly broad itnerfaces.

...

...

MIGRATION EXAMPLES. ------------------- Nova:

https://review.opendev.org/#/q/project:openstack/nova+branch:master+topic:my... Neutron:

https://review.opendev.org/#/q/status:merged+project:openstack/neutron+branc... os-vif <https://review.opendev.org/#/q/status:merged+project:openstack/neutron+branch:master+topic:bug/1492714os-vif>: https://review.opendev.org/#/c/287725/

as i alluded too above you can look at nova os-vif and neuton for inpseration but may dont follow everythin they do blindly. os-vif is proaly closest to what i woudl think is correct usage but we have not been fully consitent with idea that all privileged function should have _privileged in the name. we do however use multiple privsep contexts, one per plugin, we only need CAP_NET_ADMIN so we dont have multiple context

that one is form neutron and should really be 4 context one for cap_sys_admin one for cap_net_admin one for file access ( caps.CAP_DAC_OVERRIDE, caps.CAP_DAC_READ_SEARCH,) and one for ptrace so if people are converting project please done creat large context like the one above crate small ones that only have the caps a specific function reuqires. the function should also be sufixed with _privileged so that when its called its imediatly obvious this function has elevated privldatees e.g. delete_interface_privileged not delete_interface per plugin for different capablities. os-vif also does not use rootwap to launch the privesep deamon which is what nova is doing with os-bricks. os-vif does that iself and you can alter it via config opitn in the nova.conf which we do in our fucntional tests by setting the helper_command flag in the os_vif_privileged section dynamically https://github.com/openstack/os-vif/blob/d5b61d10655b3b7a9d32378b5f7bcdb0647... but you could change this in the nova.conf if you wanted to inject the use of rootwap or any other pviladge escalation mechanisum to futer restict the privsep deamon. this makes use of the fact that every time you creat a privsep context you can specify a config secition for it to read privsep parmater form such as https://github.com/openstack/os-vif/blob/master/os_vif/tests/functional/priv... whe we defien the os_vif_privileged we we can set any of the privsep libs standard config itmes such as helper_command https://github.com/openstack/oslo.privsep/blob/e896ed39c4b2d8492bb91283d99c6... this is how nova and os-bicks and other project should configure and spawan privsep deamons instead of using rootwap to do that. os-vif isnt perfect either but it is easy to fool your self an think privsep will do more fo you then it is if you blindly follow the example form the privsep usage docs without thinking through each step and what shoudl be the responsibility of libs or clients and where your configuration shoudl reside. e.g. nova/cinder should not be respocnisble for securign os-brick

...

...

Thank you and regards.

On Thu, 2020-03-19 at 18:24 -0500, Ben Nemec wrote:

On 3/19/20 5:22 PM, Sean Mooney wrote:

...

On Thu, 2020-03-19 at 16:04 +0000, Sean Mooney wrote:

...

On Thu, 2020-03-19 at 16:50 +0100, Iury Gregory wrote:

...

Hi Rodolfo,

Thanks for raising this topic. I like the idea and I can work in the ironic part.

Em qui., 19 de mar. de 2020 às 16:47, Rodolfo Alonso <ralonsoh@redhat.com> escreveu:

...

Hello all:

With this mail I would like to propose the goal to move away from oslo.rootwrap and migrate to oslo.privsep. The last one offers a superior security model, faster and more secure. During the last cycles and since privsep was released, the Community has encouraged the usage of privsep and the deprecation of any existing code still using rootwrap.

i had a rather long rant about why privsep is only more secure if you use it correctly i would say that today most project that use privsep are not as secure as you think they are or as secure as you would like them to be.

i would say that the privsep guide also is not entirely correct as it advocate groping privileged function in a singel module and gives an example of a context that has two many capabilities.

we have the unfortunate situation that in nova and neuton a singel privsep context is used for all privadge function meaning it has the some of all posible capablites that any code path can require. that is an anti pattern. also we group privaldage funcition in a <project?>/privsep/... module which encurages function reuse which encurages widing the contract to make them more resuable which again is an anti patteren.

Wasn't there a reason for them to all be in one module though? Like a hacking check to verify that privileged functions were used safely?

You're right about widening the contract for reuse being bad, but it seems to me that could happen wherever the code lives. A lot of the problems with the Nova implementation are that it is (or at least was) a straight port of the rootwrap calls, so in some ways it just exposed the insecurity of the rootwrap design too.

...

this lead to function like nova.privsep.path.writefile(path, mode, data). which litally allows you to write to any file on the system with any mod and put any data. that breaks all the rules. first it has a wide contract by takeing both the path and mode. second its not got _privaladged in the fucntion name so we have to have a hacking check to prevent you doing "from nova.privsep import fs; ... fs.writefile(...)" without realising its privaldaged.

Ah, this is what I was thinking of.

I guess you either use a naming convention for all privileged functions or put them in a single/few modules that you can run hacking checks on. I've meant to see if we could pull the Nova hacking check into privsep so other projects could use it.

Either approach has its benefits and drawbacks. We could probably mention both in the privsep user docs and just say that projects should be consistent in which they use.

actully i think what you were thinking of was the prefix https://github.com/openstack/nova/blob/master/nova/privsep/__init__.py#L22 which is used in the entrypoint decorator https://github.com/openstack/oslo.privsep/blob/e896ed39c4b2d8492bb91283d99c6... to ensure that the decorate can not be used out side of the module or sub module where the context is defiend. so in the nova case we do sys_admin_pctxt = priv_context.PrivContext( 'nova', cfg_section='nova_sys_admin', pypath=__name__ + '.sys_admin_pctxt', capabilities=[capabilities.CAP_CHOWN, capabilities.CAP_DAC_OVERRIDE, capabilities.CAP_DAC_READ_SEARCH, capabilities.CAP_FOWNER, capabilities.CAP_NET_ADMIN, capabilities.CAP_SYS_ADMIN], ) and teh first parmater 'nova' ensure that any function that use the decorator must be part of nova e.g. this will not work --- ~/my_project/my_module.py--- from nova import privsep @privsep.sys_admin_pctxt.entrypoint def my_func(): print("hi") in nova we technically could use 'nova.privsep' to prevent any other moduel in nova form containg privladged funcitons based on that context. in the os-vif ovs plugin we do vif_plug = priv_context.PrivContext( "vif_plug_ovs", cfg_section="vif_plug_ovs_privileged", pypath=__name__ + ".vif_plug", capabilities=[c.CAP_NET_ADMIN], ) so in the linxu bridge plugin you could not incorrectly use the ovs context because they are not in the same submodule. line lenght/typing is the main drawback for the suffix/nameing convention but yes both approchs work i obviously have a bias to the one i prefer but since i have not gone through and change all of os-vif to follow the suffix approch i dont think its worth needless code change just to conform to it.

...

thrid its grouped in privsep module with encurage it to be used form may part of the code base require int the wide contract, and finally becasue nova only has one privsep context capabilities=[capabilities.CAP_CHOWN, capabilities.CAP_DAC_OVERRIDE, capabilities.CAP_DAC_READ_SEARCH, capabilities.CAP_FOWNER, capabilities.CAP_NET_ADMIN, capabilities.CAP_SYS_ADMIN],

instead of just the file capablies it also has cap_net_admin and cap_sys_admin so it can mess with sysfs and so other thing that would normaly be prevented if it only had the capablites required for the specalised fucntion it callse writefile. for exampel create a vm conosle log....

this has come up in the past on the mainlist and at at least two ptgs http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004358.htm...

Yeah, I think it has been acknowledged that the Nova use of privsep is not ideal. I do want to note that the use of general-purpose privileged functions like writefile is specifically discouraged in multiple places in the privsep docs. That was a lesson learned from the initial Nova migration.

I believe privsep already supports multiple contexts, so maybe the docs need to be expanded to include that as a best practice.

ya you can. os-vif is always using two contexts. we have 1 context per plugin an one for the root of the libary. strictly speakign you dont need to do that but we wanted to use the in tree plugins as a template for creating your own and in general each plug should have its own security context but normally you would only have muliple contexts if you they had differnet capabliies. since each context mean an addtional unix socket and demon you don twant to have hundres of them but nova should proably have 3 looking at the set of capablites ie uses. to use multiple context after they are defiend you just need to use the correct decorator so its really easy to work with.