<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have setup two Compute nodes (using Openstack master branch) and configured libvirt for block migration (by following step #1 and #4 mentioned
<a href="http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html">
here</a>). In addition, I have also disabled apparmor for libvirtd profile and have an entry in ‘/etc/hosts’ of both the Compute hosts.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From both the Compute hosts, I am able to connect and fetch the list of running instances on the other host (using the hostname), as follows:<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:.5in">virsh # connect qemu+tcp://ubuntu-dev-001/system<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:.5in">virsh # list<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The issue is block migrating an instance between these hosts fails with the following error in the source Compute host console:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">libvir: QEMU error : operation failed: migration job: unexpectedly failed<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:318<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] MSG_ID is 8d91158236ae4de0bd8b89533f060892 from (pid=18487) multicall /opt/stack/nova/nova/rpc/amqp.py:321<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">2012-04-11 03:15:54 DEBUG nova.rpc.amqp [-] Making asynchronous cast on compute.ubuntu-dev-001... from (pid=18487) cast /opt/stack/nova/nova/rpc/amqp.py:343<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Traceback (most recent call last):<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> readers.get(fileno, noop).cb(fileno)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> result = function(*args, **kwargs)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2179, in _live_migration<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> recover_method(ctxt, instance_ref, dest, block_migration)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> self.gen.next()<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 2175, in _live_migration<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> FLAGS.live_migration_bandwidth)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 689, in migrateToURI<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">libvirtError: operation failed: migration job: unexpectedly failed<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Removing descriptor: 12<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I find the following in the qemu log file (/var/log/libvirt/qemu/instance-00000003.log ) on the destination Compute host:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">2012-04-11 04:14:25.971: starting up<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000003 -uuid f58d5f32-6d55-43fb-89ed-c33ebf72d1ed
-nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /opt/stack/nova/instances/instance-00000003/kernel -initrd
/opt/stack/nova/instances/instance-00000003/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/nova/instances/instance-00000003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-netdev tap,ifname=tapfcfa2a6c-35,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:7d:32:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/nova/instances/instance-00000003/console.log -device isa-serial,chardev=charserial0,id=serial0
-chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k en-us -vga cirrus -incoming tcp:0.0.0.0:49166 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Domain id=15 is tainted: high-privileges<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Domain id=15 is tainted: shell-scripts<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">char device redirected to /dev/pts/17<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">2012-04-11 04:14:26.406: shutting down<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">can't delete tapfcfa2a6c-35 from eth1: Operation not supported<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">SIOCSIFADDR: Permission denied<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">SIOCSIFFLAGS: Permission denied<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">SIOCSIFFLAGS: Permission denied<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">/etc/qemu-ifdown: could not launch network script
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Would be great if someone can point anything that I am missing here or any configuration changes required to resolve this issue.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Thanks & Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Unmesh Gurjar</span></b><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> | Lead Engineer |
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Vertex Software Private Ltd. |
<b>w.</b> +91.20.6604.1500 x 379 | <b>m.</b> +91.982.324.7631 | <a href="mailto:unmesh.gurjar@nttdata.com">
unmesh.gurjar@nttdata.com</a> | </span><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Follow us on Twitter@NTTDATAAmericas<b><o:p></o:p></b></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br clear="both">
______________________________________________________________________<BR>
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding<BR>
</body>
</html>