<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I was able to resolve the problem – Thanks to hint Vish provided and suggestion from Rohit (CC’ed)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Turns out there is some problem with nfs v4 – So we mounted the share using nfs version 3 (mount –o vers=3)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Now we could “chown” the files in nfs folder, and the instance creation also worked !!!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks Rohit !!!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks Vish !!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Mandar<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Vaze, Mandar
<br>
<b>Sent:</b> Wednesday, April 04, 2012 4:41 PM<br>
<b>To:</b> openstack@lists.launchpad.net<br>
<b>Cc:</b> 'vishvananda@gmail.com'<br>
<b>Subject:</b> RE: [Openstack] Instance fails to spawn when instance_path is nfs mounted<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">On the link given below vish asked if I could “chown” the files in that folder – turns out I couldn’t.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ve answered his query on the link below – but repeating the updates here for larger audience (Unfortunately, I have revived a question that is marked as
“solved” – so not many people may refer/respond to the link)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border-top:double windowtext 2.25pt;border-left:none;border-bottom:double windowtext 2.25pt;border-right:none;padding:1.0pt 0in 1.0pt 0in">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Vish :<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">> 1) is it possible to chown files on your nfs mount?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Not sure whom should I give the ownership to ? As the dir listing shows above, the files are created with correct owner/group settings. These files were created
by nova-compute process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I "touch"ed a file for testing - which was created with "mandar:mandar" - I tried changing the ownership, but I keep getting "invalid" argument - not sure if
that is what you were referring to ..<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ touch x<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ ll<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">total 16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxrwxrwx 4 root root 4096 2012-04-04 03:46 ./<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxr-xr-x 15 mandar mandar 4096 2012-04-04 03:39 ../<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxrwxr-x 2 mandar libvirtd 4096 2012-04-04 02:18 _base/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxr-xr-x 2 mandar libvirtd 4096 2012-04-04 03:41 instance-0000000f/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">-rw-rw-r-- 1 mandar mandar 0 2012-04-04 03:46 x<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ chown root x<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">chown: changing ownership of `x': Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ ll<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">total 16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxrwxrwx 4 root root 4096 2012-04-04 03:46 ./<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxr-xr-x 15 mandar mandar 4096 2012-04-04 03:39 ../<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxrwxr-x 2 mandar libvirtd 4096 2012-04-04 02:18 _base/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">drwxr-xr-x 2 mandar libvirtd 4096 2012-04-04 03:41 instance-0000000f/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">-rw-rw-r-- 1 mandar mandar 0 2012-04-04 03:46 x<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ sudo chown root x<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">chown: changing ownership of `x': Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ chgrp libvirtd x<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">chgrp: changing group of `x': Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ sudo chgrp libvirtd x<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">chgrp: changing group of `x': Invalid argument<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I also debugged the "_chown_console_log_for_instance" in "nova/virt/libvirt/connection.py"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">and with breakpoint before "if os.path.exists(console_log)" - I touched "console.log" at the appropriate path from another terminal.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Over there also I am getting the same error:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">2012-04-04 03:56:55 TRACE nova.rpc.amqp Command: sudo /usr/local/bin/nova-rootwrap chown 1000 /home/mandar/nfs_shared_instances_path/instance-00000010/console.log<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">2012-04-04 03:56:55 TRACE nova.rpc.amqp Exit code: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">2012-04-04 03:56:55 TRACE nova.rpc.amqp Stdout: ''<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">2012-04-04 03:56:55 TRACE nova.rpc.amqp Stderr: "/bin/chown: changing ownership of `/home/mandar/nfs_shared_instances_path/instance-00000010/console.log': Invalid argument\n"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So what is the fix for this ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Mandar<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:openstack-bounces+mandar.vaze=vertex.co.in@lists.launchpad.net">openstack-bounces+mandar.vaze=vertex.co.in@lists.launchpad.net</a>
<a href="mailto:[mailto:openstack-bounces+mandar.vaze=vertex.co.in@lists.launchpad.net]">
[mailto:openstack-bounces+mandar.vaze=vertex.co.in@lists.launchpad.net]</a> <b>On Behalf Of
</b>Mandar Vaze / ????? ???<br>
<b>Sent:</b> Tuesday, April 03, 2012 10:14 PM<br>
<b>To:</b> <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
<b>Subject:</b> [Openstack] Instance fails to spawn when instance_path is nfs mounted<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I saw an old question posted here :<br>
<a href="https://answers.launchpad.net/nova/+question/164689" target="_blank">https://answers.launchpad.net/nova/+question/164689</a><br>
<br>
But I am not trying live migration.<br>
<br>
I have nfs mounted instances_path - so when I try to spawn an instance I<br>
run into the above errors. Especially following :<br>
<br>
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 372, in createWithFlags<br>
40842 2012-04-03 05:42:27 TRACE nova.rpc.amqp if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)<br>
40843 2012-04-03 05:42:27 TRACE nova.rpc.amqp libvirtError: internal error Process exited while reading console log output: chardev: opening backend "file" failed<br>
<br>
But as you can see below, several files are created in this folder, so I<br>
am not sure if mine if permissions issue (Else none of the files would<br>
get created) The problem is reported when libvirt tries to write to<br>
console.log (File itself is created with correct permissions - just that<br>
this is zero byte file)<br>
<br>
mandar@ubuntu-dev-mandar:/nfs_shared_instances_path/instance-00000005$ ll<br>
total 10944<br>
drwxrwxr-x 2 mandar libvirtd 4096 2012-04-03 05:42 ./<br>
drwxrwxrwx 4 root root 4096 2012-04-03 05:42 ../<br>
-rw-rw---- 1 mandar libvirtd 0 2012-04-03 05:42 console.log<br>
-rw-r--r-- 1 mandar libvirtd 6291968 2012-04-03 05:42 disk<br>
-rw-rw-r-- 1 mandar libvirtd 4731440 2012-04-03 05:42 kernel<br>
-rw-rw-r-- 1 mandar libvirtd 1067 2012-04-03 05:42 libvirt.xml<br>
-rw-rw-r-- 1 mandar libvirtd 2254249 2012-04-03 05:42 ramdisk<br>
<br>
I'm suspecting : <a href="https://bugs.launchpad.net/ubuntu/maverick/+source/libvirt/+bug/632696" target="_blank">
https://bugs.launchpad.net/ubuntu/maverick/+source/libvirt/+bug/632696</a><br>
But I the above doesn't show itself in non-NFS setup<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Please suggest !!!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Mandar<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br clear="both">
______________________________________________________________________<BR>
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding<BR>
</body>
</html>