[nova][keystone] What happens to key pairs after user is deleted
Hi all, From the API perspective it is possible to delete user without deleting its key pairs. Practice showed, however, that keypairs of deleted user still exist and can be queried by API knowing id of the deleted user (at least in devstack and 1 other public cloud). I know it may be tricky if there is still VM provisioned with the key, but deleting user logically means nobody has access to the private key anyway. And since key pairs belong to users and not to projects it is not possible to clean them up in the project cleanup either. Actually from the API pov there is no reasonable way to ever find those (without knowing ID of the deleted user which is logically not known anymore). If there is no cleanup this can in the mid term cause trashing the database (records are small, but still), especially when using “dynamic” users to perform some actions. So far I haven’t tried to grep through code basis of Nova to check what is happening, neither tried to check behavior over time, and decided first to ask here whether somebody knows what should be generally happening here, is it a bug or feature? Thanks, Artem
On Fri, 2022-10-14 at 12:23 +0200, Artem Goncharov wrote:
Hi all,
From the API perspective it is possible to delete user without deleting its key pairs.
from a nova api perspective you have violated the precondition if you do not remove all resouces owned by the user in nova before you delete the user in keystone. so you are conflaiting two things. it is possibel to do in keystone apit but its not valid to do that as you have not met the precondtion of cleaing up the resocue in other project. so no we donot support deleteign the keyparis after the fact like that.
Practice showed, however, that keypairs of deleted user still exist and can be queried by API knowing id of the deleted user (at least in devstack and 1 other public cloud). I know it may be tricky if there is still VM provisioned with the key, but deleting user logically means nobody has access to the private key anyway.
correct noone shoudl have access to the key but again you are not allowed to delete the user before you remoave any resouces used by it you can delete the keypari wihotut deleteing it form the vms that were created with it. deleting the keypair has never implied removing the keypair form ths authrised keys in the vm so no assumitions shoudl be made that removing the keypair alther who can log into the vm. that is just not part of what the nova api.
And since key pairs belong to users and not to projects it is not possible to clean them up in the project cleanup either.
Actually from the API pov there is no reasonable way to ever find those (without knowing ID of the deleted user which is logically not known anymore).
If there is no cleanup this can in the mid term cause trashing the database (records are small, but still), especially when using “dynamic” users to perform some actions. ya so if we wanted to suprot automatic clean up of user resouce likek keyparis we woudl need a new user-deleted exeternal event in the nova api and nova could delete the key pair and any other user (not project) owned api resoruse but i think the key pari is the only example we have today. teh vms are owned by the project not the user.
So far I haven’t tried to grep through code basis of Nova to check what is happening, neither tried to check behavior over time, and decided first to ask here whether somebody knows what should be generally happening here, is it a bug or feature?
this is not a bug its user error. nova and all other openstack servces to my knoladge require that you clean up the reosuce used by users or proejct are cleaned up before you remove a user/project form keystone. so by violatign that requirement you can nolonger interact with apis that depedn in the delete entitiy and that is expect. it woudl be a large cross proejct effort to chagne that. alternitivly keystoen could prevent the user/project form being deleteed if there are resuouce used by that user/project in other service btu tthat woudl also be a cross project effort. for this specific issue we could add a new Admin only api to allow the deletion fo user keypairs btu that woudl be a new feature.
Thanks, Artem
Thanks for answer
On 14. Oct 2022, at 13:54, Sean Mooney <smooney@redhat.com> wrote:
On Fri, 2022-10-14 at 12:23 +0200, Artem Goncharov wrote:
Hi all,
From the API perspective it is possible to delete user without deleting its key pairs.
from a nova api perspective you have violated the precondition if you do not remove all resouces owned by the user in nova before you delete the user in keystone. so you are conflaiting two things. it is possibel to do in keystone apit but its not valid to do that as you have not met the precondtion of cleaing up the resocue in other project. so no we donot support deleteign the keyparis after the fact like that.
Practice showed, however, that keypairs of deleted user still exist and can be queried by API knowing id of the deleted user (at least in devstack and 1 other public cloud). I know it may be tricky if there is still VM provisioned with the key, but deleting user logically means nobody has access to the private key anyway.
correct noone shoudl have access to the key but again you are not allowed to delete the user before you remoave any resouces used by it you can delete the keypari wihotut deleteing it form the vms that were created with it. deleting the keypair has never implied removing the keypair form ths authrised keys in the vm so no assumitions shoudl be made that removing the keypair alther who can log into the vm. that is just not part of what the nova api.
And since key pairs belong to users and not to projects it is not possible to clean them up in the project cleanup either.
Actually from the API pov there is no reasonable way to ever find those (without knowing ID of the deleted user which is logically not known anymore).
If there is no cleanup this can in the mid term cause trashing the database (records are small, but still), especially when using “dynamic” users to perform some actions. ya so if we wanted to suprot automatic clean up of user resouce likek keyparis we woudl need a new user-deleted exeternal event in the nova api and nova could delete the key pair and any other user (not project) owned api resoruse but i think the key pari is the only example we have today. teh vms are owned by the project not the user.
Right, this is only valid for key pairs. That is precisely the reason why project cleanup is not dealing with that today.
So far I haven’t tried to grep through code basis of Nova to check what is happening, neither tried to check behavior over time, and decided first to ask here whether somebody knows what should be generally happening here, is it a bug or feature?
this is not a bug its user error.
I disagree here. There is nothing that blocks you from doing so. User error is i.e. to try to delete something what is still used. Here the user was never said to delete all of the resources, cause same statement is valid to deletion or not deletion of the VM created by the user. You should not be able to delete project if there are resources remaining, and you should not be able to drop user if key pairs exist? And the biggest issue is that once user recognised he did an “error” - he is not able to fix it. My personal opinion is that it is a bug to let user to make error. What is the user supposed to do after he recognised he deleted project without deleting resources first? As admin you have chance to catch resources not belonging to any existing project, but not with key pairs. And how big are the chances you can still do this on a big cloud?
nova and all other openstack servces to my knoladge require that you clean up the reosuce used by users or proejct are cleaned up before you remove a user/project form keystone. so by violatign that requirement you can nolonger interact with apis that depedn in the delete entitiy and that is expect. it woudl be a large cross proejct effort to chagne that.
alternitivly keystoen could prevent the user/project form being deleteed if there are resuouce used by that user/project in other service btu tthat woudl also be a cross project effort.
This feels logical, but most likely not easy to achieve, because otherwise Keystone need to query every service asking whether deletion of this user should be blocked or not. Keystone sending announcement to the services that certain user/project//domain was deleted so that service makes decision what to do with that is easier to achieve, but blocking is really the only way to make user experience correct and avoid creating a mess in a first place.
for this specific issue we could add a new Admin only api to allow the deletion fo user keypairs btu that woudl be a new feature.
From the user perspective I would prefer extending list key pairs api with something like “all users”. Having info like that customer based cleanup can determine all KPs owned by deleted users and drop them. For that, however, some form of tracking to which domain user was belonging (maybe instead of "all_users" add param "user_domain_id") need to be also done. I would like to prevent that only super-duper admin of the cloud can do such cleanup, otherwise in a big public cloud it will become a mess.
Thanks, Artem
On Fri, 2022-10-14 at 15:06 +0200, Artem Goncharov wrote:
Thanks for answer
On 14. Oct 2022, at 13:54, Sean Mooney <smooney@redhat.com> wrote:
On Fri, 2022-10-14 at 12:23 +0200, Artem Goncharov wrote:
Hi all,
From the API perspective it is possible to delete user without deleting its key pairs.
from a nova api perspective you have violated the precondition if you do not remove all resouces owned by the user in nova before you delete the user in keystone. so you are conflaiting two things. it is possibel to do in keystone apit but its not valid to do that as you have not met the precondtion of cleaing up the resocue in other project. so no we donot support deleteign the keyparis after the fact like that.
Practice showed, however, that keypairs of deleted user still exist and can be queried by API knowing id of the deleted user (at least in devstack and 1 other public cloud). I know it may be tricky if there is still VM provisioned with the key, but deleting user logically means nobody has access to the private key anyway.
correct noone shoudl have access to the key but again you are not allowed to delete the user before you remoave any resouces used by it you can delete the keypari wihotut deleteing it form the vms that were created with it. deleting the keypair has never implied removing the keypair form ths authrised keys in the vm so no assumitions shoudl be made that removing the keypair alther who can log into the vm. that is just not part of what the nova api.
And since key pairs belong to users and not to projects it is not possible to clean them up in the project cleanup either.
Actually from the API pov there is no reasonable way to ever find those (without knowing ID of the deleted user which is logically not known anymore).
If there is no cleanup this can in the mid term cause trashing the database (records are small, but still), especially when using “dynamic” users to perform some actions. ya so if we wanted to suprot automatic clean up of user resouce likek keyparis we woudl need a new user-deleted exeternal event in the nova api and nova could delete the key pair and any other user (not project) owned api resoruse but i think the key pari is the only example we have today. teh vms are owned by the project not the user.
Right, this is only valid for key pairs. That is precisely the reason why project cleanup is not dealing with that today.
So far I haven’t tried to grep through code basis of Nova to check what is happening, neither tried to check behavior over time, and decided first to ask here whether somebody knows what should be generally happening here, is it a bug or feature?
this is not a bug its user error.
I disagree here. There is nothing that blocks you from doing so. User error is i.e. to try to delete something what is still used. Here the user was never said to delete all of the resources, cause same statement is valid to deletion or not deletion of the VM created by the user. You should not be able to delete project if there are resources remaining, and you should not be able to drop user if key pairs exist? And the biggest issue is that once user recognised he did an “error” - he is not able to fix it.
My personal opinion is that it is a bug to let user to make error. What is the user supposed to do after he recognised he deleted project without deleting resources first? As admin you have chance to catch resources not belonging to any existing project, but not with key pairs. And how big are the chances you can still do this on a big cloud?
As a general observation, one of the many downsides of service-specific quotas (as opposed to keystone-managed "limits") is that if/when a project is deleted in keystone, quotas related to that porject can be left hanging around in various services. It's up to the admin to go and clean these up manually, either ahead of time (using the clients) or after the fact (probably via DB, since OSC at least will fail to find the project when looking it up). It sounds like key pairs suffer from a similar issue. Yay, microservices. Stephen
nova and all other openstack servces to my knoladge require that you clean up the reosuce used by users or proejct are cleaned up before you remove a user/project form keystone. so by violatign that requirement you can nolonger interact with apis that depedn in the delete entitiy and that is expect. it woudl be a large cross proejct effort to chagne that.
alternitivly keystoen could prevent the user/project form being deleteed if there are resuouce used by that user/project in other service btu tthat woudl also be a cross project effort.
This feels logical, but most likely not easy to achieve, because otherwise Keystone need to query every service asking whether deletion of this user should be blocked or not. Keystone sending announcement to the services that certain user/project//domain was deleted so that service makes decision what to do with that is easier to achieve, but blocking is really the only way to make user experience correct and avoid creating a mess in a first place.
for this specific issue we could add a new Admin only api to allow the deletion fo user keypairs btu that woudl be a new feature.
From the user perspective I would prefer extending list key pairs api with something like “all users”. Having info like that customer based cleanup can determine all KPs owned by deleted users and drop them. For that, however, some form of tracking to which domain user was belonging (maybe instead of "all_users" add param "user_domain_id") need to be also done. I would like to prevent that only super-duper admin of the cloud can do such cleanup, otherwise in a big public cloud it will become a mess.
Thanks, Artem
participants (3)
-
Artem Goncharov
-
Sean Mooney
-
Stephen Finucane