[Manila] Manila user overwriting existing Ceph users
Hello everyone, I'm currently testing manila with CephFS and I stumbled upon a behavior where manila is able to overwrite existing Ceph users. In my testing setup glance, nova, cinder and manila share the same Ceph cluster. However they have different users. In this situation when you create a share and allow acces via "manila access-allow cephshare1 cephx test" If the user "test" is already used to access some pools on the cluster, let's say cinder-volume or glance-images it will be overwritten with the permissions for the share. Which will break any resources that was using it. I've recheck the configuration files multiple times to see if I could set some properties to avoid this but I didn't find any. By quickly looking at the code here : https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/... A check is done but only for the manila user. I'm on Rocky version but this part doesn't seems to have changed since. That lead me to some questions : - Does manila must have his own dedicated Ceph cluster ? - Is there any workaroud to this ? Other than putting some gibberish names for services users ? - Is it possible to lock some users in the Ceph cluster to prevent this behavior ? If someone has some clues on this, thanks in advance. Jahson.B
On 11/12/20 10:24 AM, Babel Jahson wrote:
Hello everyone,
I'm currently testing manila with CephFS and I stumbled upon a behavior where manila is able to overwrite existing Ceph users. In my testing setup glance, nova, cinder and manila share the same Ceph cluster. However they have different users. In this situation when you create a share and allow acces via "manila access-allow cephshare1 cephx test" If the user "test" is already used to access some pools on the cluster, let's say cinder-volume or glance-images it will be overwritten with the permissions for the share. Which will break any resources that was using it. I've recheck the configuration files multiple times to see if I could set some properties to avoid this but I didn't find any. By quickly looking at the code here : https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/... A check is done but only for the manila user. I'm on Rocky version but this part doesn't seems to have changed since.
That lead me to some questions : - Does manila must have his own dedicated Ceph cluster ? - Is there any workaroud to this ? Other than putting some gibberish names for services users ? - Is it possible to lock some users in the Ceph cluster to prevent this behavior ?
hi Jahnson, I am adding a few folks who can probably help us better but I also wanted to ask a question to understand better the use case the cephx user which cinder/glance/nova use has specific permissions to operate on their pools and this is configured in their respective config, not something you have access from the actual openstack guests; are you saying that "access-allow" is overwriting the cephx caps which were set for the cephx user which, for example, cinder is configured to use? in that case maybe better would be for the manila workflow to add/remove caps to existing users instead of overwriting the caps? is that be what you expected to happen? -- Giulio Fidente GPG KEY: 08D733BA
Hi Giulio, Thank you for your response.
the cephx user which cinder/glance/nova use has specific permissions to operate on their pools and this is configured in their respective config, not something you have access from the actual openstack guests; are you saying that "access-allow" is overwriting the cephx caps which were set for the cephx user which, for example, cinder is configured to use? Yes a cinder user can be overwritten in the Ceph config cluster by the command "access-allow" to a share. Basically it goes from something like this : [client.cindertest] key = <cindertestkey> caps mon = "profile rbd" caps osd = "profile rbd pool=some-pool, profile rbd pool=some-pool
in that case maybe better would be for the manila workflow to add/remove caps to existing users instead of overwriting the caps? is that be what you expected to happen? Not really, I mean it's a possibility but is it safe to just add those caps to an existing user ? Won't that interfere with something else ? A way to prevent the creation of a user like "cindertest" seems a better solution to me but I maybe wrong. It's behavior manila already has. If a user have been created with manila for a share in a project and you ask for that user in another
To something like that : [client.cindertest] key = <cindertestkey> caps mds = "allow rw path=/volumes/_nogroup/<some_id>" caps mon = "allow r" caps osd = "allow rw pool=<cephfs_data> namespace=fsvolumens_<some_id>" Which can be problematic. project in openstack it wouldn't let you used it. Jahson On 12/11/2020 11:56, Giulio Fidente wrote:
On 11/12/20 10:24 AM, Babel Jahson wrote:
Hello everyone,
I'm currently testing manila with CephFS and I stumbled upon a behavior where manila is able to overwrite existing Ceph users. In my testing setup glance, nova, cinder and manila share the same Ceph cluster. However they have different users. In this situation when you create a share and allow acces via "manila access-allow cephshare1 cephx test" If the user "test" is already used to access some pools on the cluster, let's say cinder-volume or glance-images it will be overwritten with the permissions for the share. Which will break any resources that was using it. I've recheck the configuration files multiple times to see if I could set some properties to avoid this but I didn't find any. By quickly looking at the code here : https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/... A check is done but only for the manila user. I'm on Rocky version but this part doesn't seems to have changed since.
That lead me to some questions : - Does manila must have his own dedicated Ceph cluster ? - Is there any workaroud to this ? Other than putting some gibberish names for services users ? - Is it possible to lock some users in the Ceph cluster to prevent this behavior ? hi Jahnson, I am adding a few folks who can probably help us better but I also wanted to ask a question to understand better the use case
the cephx user which cinder/glance/nova use has specific permissions to operate on their pools and this is configured in their respective config, not something you have access from the actual openstack guests; are you saying that "access-allow" is overwriting the cephx caps which were set for the cephx user which, for example, cinder is configured to use?
in that case maybe better would be for the manila workflow to add/remove caps to existing users instead of overwriting the caps? is that be what you expected to happen?
On 12/11/20 13:32 +0100, Babel Jahson wrote:
Hi Giulio,
Thank you for your response.
the cephx user which cinder/glance/nova use has specific permissions to operate on their pools and this is configured in their respective config, not something you have access from the actual openstack guests; are you saying that "access-allow" is overwriting the cephx caps which were set for the cephx user which, for example, cinder is configured to use? Yes a cinder user can be overwritten in the Ceph config cluster by the command "access-allow" to a share. Basically it goes from something like this : [client.cindertest] key = <cindertestkey> caps mon = "profile rbd" caps osd = "profile rbd pool=some-pool, profile rbd pool=some-pool
To something like that : [client.cindertest] key = <cindertestkey> caps mds = "allow rw path=/volumes/_nogroup/<some_id>" caps mon = "allow r" caps osd = "allow rw pool=<cephfs_data> namespace=fsvolumens_<some_id>"
Which can be problematic.
in that case maybe better would be for the manila workflow to add/remove caps to existing users instead of overwriting the caps? is that be what you expected to happen? Not really, I mean it's a possibility but is it safe to just add those caps to an existing user ? Won't that interfere with something else ? A way to prevent the creation of a user like "cindertest" seems a better solution to me but I maybe wrong. It's behavior manila already has. If a user have been created with manila for a share in a project and you ask for that user in another project in openstack it wouldn't let you used it.
Jahson
On 12/11/2020 11:56, Giulio Fidente wrote:
On 11/12/20 10:24 AM, Babel Jahson wrote:
Hello everyone,
I'm currently testing manila with CephFS and I stumbled upon a behavior where manila is able to overwrite existing Ceph users. In my testing setup glance, nova, cinder and manila share the same Ceph cluster. However they have different users. In this situation when you create a share and allow acces via "manila access-allow cephshare1 cephx test" If the user "test" is already used to access some pools on the cluster, let's say cinder-volume or glance-images it will be overwritten with the permissions for the share. Which will break any resources that was using it. I've recheck the configuration files multiple times to see if I could set some properties to avoid this but I didn't find any. By quickly looking at the code here : https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/... A check is done but only for the manila user. I'm on Rocky version but this part doesn't seems to have changed since.
That lead me to some questions : - Does manila must have his own dedicated Ceph cluster ? - Is there any workaroud to this ? Other than putting some gibberish names for services users ? - Is it possible to lock some users in the Ceph cluster to prevent this behavior ? hi Jahnson, I am adding a few folks who can probably help us better but I also wanted to ask a question to understand better the use case
the cephx user which cinder/glance/nova use has specific permissions to operate on their pools and this is configured in their respective config, not something you have access from the actual openstack guests; are you saying that "access-allow" is overwriting the cephx caps which were set for the cephx user which, for example, cinder is configured to use?
in that case maybe better would be for the manila workflow to add/remove caps to existing users instead of overwriting the caps? is that be what you expected to happen?
Babel, Thanks for this report. Would you be so kind as to open a Launchpad bug against Manila for this issue? Manila, Nova, Cinder, etc. all use the same Ceph cluster but use different pools in that cluster. It does seem that we need the CephFS driver (or perhaps something in the Ceph cluster) to mark the service users as such and prevent changes to them by Manila. -- Tom Barron
Hi Tom, No problem, I'll create a Launchpadd issue for this. Jahson On 12/11/2020 14:30, Tom Barron wrote:
On 12/11/20 13:32 +0100, Babel Jahson wrote:
Hi Giulio,
Thank you for your response.
the cephx user which cinder/glance/nova use has specific permissions to operate on their pools and this is configured in their respective config, not something you have access from the actual openstack guests; are you saying that "access-allow" is overwriting the cephx caps which were set for the cephx user which, for example, cinder is configured to use? Yes a cinder user can be overwritten in the Ceph config cluster by the command "access-allow" to a share. Basically it goes from something like this : [client.cindertest] key = <cindertestkey> caps mon = "profile rbd" caps osd = "profile rbd pool=some-pool, profile rbd pool=some-pool
To something like that : [client.cindertest] key = <cindertestkey> caps mds = "allow rw path=/volumes/_nogroup/<some_id>" caps mon = "allow r" caps osd = "allow rw pool=<cephfs_data> namespace=fsvolumens_<some_id>"
Which can be problematic.
in that case maybe better would be for the manila workflow to add/remove caps to existing users instead of overwriting the caps? is that be what you expected to happen? Not really, I mean it's a possibility but is it safe to just add those caps to an existing user ? Won't that interfere with something else ? A way to prevent the creation of a user like "cindertest" seems a better solution to me but I maybe wrong. It's behavior manila already has. If a user have been created with manila for a share in a project and you ask for that user in another project in openstack it wouldn't let you used it.
Jahson
On 12/11/2020 11:56, Giulio Fidente wrote:
On 11/12/20 10:24 AM, Babel Jahson wrote:
Hello everyone,
I'm currently testing manila with CephFS and I stumbled upon a behavior where manila is able to overwrite existing Ceph users. In my testing setup glance, nova, cinder and manila share the same Ceph cluster. However they have different users. In this situation when you create a share and allow acces via "manila access-allow cephshare1 cephx test" If the user "test" is already used to access some pools on the cluster, let's say cinder-volume or glance-images it will be overwritten with the permissions for the share. Which will break any resources that was using it. I've recheck the configuration files multiple times to see if I could set some properties to avoid this but I didn't find any. By quickly looking at the code here : https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/...
A check is done but only for the manila user. I'm on Rocky version but this part doesn't seems to have changed since.
That lead me to some questions : - Does manila must have his own dedicated Ceph cluster ? - Is there any workaroud to this ? Other than putting some gibberish names for services users ? - Is it possible to lock some users in the Ceph cluster to prevent this behavior ? hi Jahnson, I am adding a few folks who can probably help us better but I also wanted to ask a question to understand better the use case
the cephx user which cinder/glance/nova use has specific permissions to operate on their pools and this is configured in their respective config, not something you have access from the actual openstack guests; are you saying that "access-allow" is overwriting the cephx caps which were set for the cephx user which, for example, cinder is configured to use?
in that case maybe better would be for the manila workflow to add/remove caps to existing users instead of overwriting the caps? is that be what you expected to happen?
Babel,
Thanks for this report. Would you be so kind as to open a Launchpad bug against Manila for this issue?
Manila, Nova, Cinder, etc. all use the same Ceph cluster but use different pools in that cluster. It does seem that we need the CephFS driver (or perhaps something in the Ceph cluster) to mark the service users as such and prevent changes to them by Manila.
-- Tom Barron
participants (3)
-
Babel Jahson
-
Giulio Fidente
-
Tom Barron