[Openstack-security] [Bug 1252519] Re: Live migration failed because of file permission changed

Maurice Leeflang maurice at leeflang.net
Thu Aug 14 09:19:08 UTC 2014


I have the same problem. 
I am currently trying to isolate the cause to it. 

The first live migration of an instance works, but the second one (back to the first node) fails.
The ownership changed, but even manually as root I am not allowed to read the disk file.
Only when kvm closes the fd's on these files (when I suspend the instance, for example), the files can be read again. The files can still be read when the instance is resumed, even a live migration is possible again, then. The ownership after a resume is qemu again (not root), so I can understand why Barrow points that way.

It smells like some locking situation on the gluster side, but I am not able to pinpoint it to a configuration option or bug yet
I will do some more tests to see why (and how)  gluster is spoiling the fun. 
Please do not put the blame on some gluster bug or behaviour yet. 
The fact that the ownership of instance files on shared storage changes indicates that there is still a resize or something being done (post migration), which, IMHO, is not needed in the case of a live migration with de nova instances directory on shared storage for all compute nodes participating.

-- 
You received this bug notification because you are a member of OpenStack
Security Group, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1252519

Title:
  Live migration failed because of file permission changed

Status in OpenStack Compute (Nova):
  Triaged

Bug description:
  Openstack : Havana
  OS : CentOS 6.4
  Shared storage with GlusterFS :  /var/lib/nova/instances mounted on glusterfs shared

  
  Instance start up fine on node01.  When live migration happen, it moved to node02 but failed with the following error

  2013-11-18 16:27:37.813 9837 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: Unexpected error while running command.
  Command: env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/aa1deb40-ae1d-45e4-a37e-7b0607df372f/disk
  Exit code: 1
  Stdout: ''
  Stderr: "qemu-img: Could not open '/var/lib/nova/instances/aa1deb40-ae1d-45e4-a37e-7b0607df372f/disk'\n"
  2013-11-18 16:27:37.813 9837 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
  2013-11-18 16:27:37.813 9837 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
  2013-11-18 16:27:37.813 9837 TRACE nova.openstack.common.periodic_task     task(self, context)


  The problem is with the file ownership of "console.log" and "disk".
  Those file should be owned by user "qemu" and group "qemu" but after
  the migration, both files are owned by root

  
  drwxr-xr-x 2 nova nova       53 Nov 18 13:40 .
  drwxr-xr-x 6 nova nova      110 Nov 18 13:43 ..
  -rw-rw---- 1 root root     1546 Nov 18 13:43 console.log
  -rw-r--r-- 1 root root 12058624 Nov 18 13:42 disk
  -rw-r--r-- 1 nova nova     1569 Nov 18 13:42 libvirt.xml

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1252519/+subscriptions




More information about the Openstack-security mailing list