Open Stack

On 2/25/2016 5:31 AM, Takashi Natsume wrote:
> Hi Nova and Cinder developers.
>
> As I reported in a bug report [1], nova swap volume
> (updating an attached volume) fuction does not work
> in the case of non admin users by default.
> (Volumes are stuck.)
>
> Before I was working for fixing another swap volume bug [2][3].
> But Ryan fixed it on the Cinder side [4].
> As a result, admin users can execute swap volume function,
> but it was not fixed in the case of non admin users.
> So I reported the bug report [1].
>
> In the patch[5], I tried to change the default cinder's policy
> to allow non admin users to execute migrate_volume_completion API.
> But it was rejected by the cinder project ('-2' was voted).
>
> In the patch[5], it was suggested to make the swap volume API admin only
> on the Nova side.
> But IMO, the swap volume function should be allowed to non admin users
> because attaching a volume and detaching a volume can be performed
> by non admin users.

I agree with this. DuncanT said in IRC that he didn't think non-admin 
users should be using the swap-volume API in nova because it can be 
problematic, but I'm not sure why, is there more history or detail 
there? I'd think it shouldn't be any worse than doing a detach/attach in 
quick succession (like in a CI test for example).

>
> If migrate_volume_completion is only allowed to admin users
> by default on the Cinder side, attaching a new volume and
> detaching an old volume should be performed on the Nova side
> when swapping volumes.

My understanding of the problem is as follows:

1. Admin-initiated volume migration in Cinder calls off to Nova to 
perform the swap-volume, and then Nova calls back to Cinder's 
migrate_volume_completion API. This is fine since it's an admin that 
initiated this series of operations on the Cinder side (that's by 
default, however, this is broken if the policy file for Cinder is change 
to allow non-admins to migrate volumes).

2. A non-admin swap-volume API call in Nova fails because Nova blindly 
makes the migrate_volume_completion call to Cinder which fails with a 
403 because the Cinder API policy has that as an admin action by default.

I don't know the history around when the swap-volume API was added to 
Nova, was it specifically for this volume migration scenario in Cinder? 
  Are there other use cases?  Knowing those would be good to determine 
if Nova should change it's default policy for swap-volume, although, 
again, that's only a default and can be changed per deployment so we 
probably shouldn't rely on it.

Ideally we would have implemented this like the nova/neutron server 
events callback API in Nova during vif plugging (nova does the vif plug 
on the host then waits for neutron to update it's database for the port 
status and sends an event (API call) to nova to continue booting the 
server). That server events API in nova is admin-only by default and 
neutron is configured with admin credentials for nova to use it.

Another option would be for Nova to handle a 403 response when calling 
Cinder's migrate_volume_completion API and ignore it if we don't have an 
admin context. This is pretty hacky though. It assumes that it's a 
non-admin user initiating the swap-volume operation. It wouldn't be a 
problem for the volume migration operation initiated in Cinder since by 
default that's admin-only, so nova shouldn't get a 403 when calling 
migrate_volume_completion. The trap would be if the cinder policy for 
volume migration was changed to allow non-admins, but if someone did 
that, they should also change the policy for migrate_volume_completion 
to allow non-admin too.

>
> If you have a good idea, please let me know it.
>
> [1] Cinder volumes are stuck when non admin user executes nova swap volume API
>      https://bugs.launchpad.net/cinder/+bug/1522705
>
> [2] Cinder volume stuck in swap_volume
>      https://bugs.launchpad.net/nova/+bug/1471098
>
> [3] Fix cinder volume stuck in swap_volume
>      https://review.openstack.org/#/c/207385/
>
> [4] Fix swap_volume for case without migration
>      https://review.openstack.org/#/c/247767/
>
> [5] Enable volume owners to execute migrate_volume_completion
>      https://review.openstack.org/#/c/253363/
>
> Regards,
> Takashi Natsume
> NTT Software Innovation Center
> E-mail: natsume.takashi at lab.ntt.co.jp
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

-- 

Thanks,

Matt Riedemann