<div dir="ltr"><div><div>Have you migrated to Neutron at the same time as upgrading?<br><br></div>There are installer docs for metadata with Neutron; the troubleshooting process will depend on the production config you chose - e.g. namespaced network, or not etc.<br>
<br></div>Cheers,<br>Rob<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 29 October 2013 14:14, Bill Owen <span dir="ltr"><<a href="mailto:billowen@us.ibm.com" target="_blank">billowen@us.ibm.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<p><font face="sans-serif">Padraig, Robert, Chenrui,</font><br>
<font face="sans-serif">It seems to be a problem with nova metadata service.</font></p><div class="im"><br>
<br>
<tt><font>> Are you using libguestfs to do the injection? </font></tt><br>
</div><tt><font>Yes - I wasn't originally, but have installed this on my controller and compute nodes.</font></tt><div class="im"><br>
<tt><font><br>
> What's the value of the following in nova.conf?<br>
> libvirt_inject_key<br>
> libvirt_inject_partition</font></tt><br>
<br>
</div><font face="sans-serif">I have the following set - I added inject_password as well to see if any additional information could be seen.</font><br>
<font face="sans-serif">/etc/nova/nova.conf:libvirt_inject_partition = -1</font><br>
<font face="sans-serif">/etc/nova/nova.conf:libvirt_inject_key = True</font><br>
<font face="sans-serif">/etc/nova/nova.conf:libvirt_inject_password = True</font><br>
<br>
<font face="sans-serif">When I boot a Ubuntu precise image I see the following in the console log:</font>
<ul style="padding-left:18pt"><font face="sans-serif">[ 1.573692] EXT4-fs (vda1): re-mounted. Opts: (null)</font><br>
<font face="sans-serif">cloud-init start-local running: Tue, 29 Oct 2013 00:15:41 +0000. up 5.11 seconds</font><br>
<font face="sans-serif">no instance data found in start-local</font><br>
<font face="sans-serif">ci-info: lo : 1 127.0.0.1 255.0.0.0 .</font><br>
<font face="sans-serif">ci-info: eth0 : 1 10.10.100.2 255.255.255.0 fa:16:3e:a6:5a:98</font><br>
<font face="sans-serif">ci-info: route-0: 0.0.0.0 10.10.100.3 0.0.0.0 eth0 UG</font><br>
<font face="sans-serif">ci-info: route-1: 10.10.100.0 0.0.0.0 255.255.255.0 eth0 U</font><br>
<font face="sans-serif">cloud-init start running: Tue, 29 Oct 2013 00:15:41 +0000. up 5.89 seconds</font><br>
<font face="sans-serif">2013-10-29 00:16:32,646 - util.py[WARNING]: '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [50/120s]: url error [timed out]</font><br>
<font face="sans-serif">2013-10-29 00:17:23,698 - util.py[WARNING]: '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [101/120s]: url error [timed out]</font><br>
<font face="sans-serif">2013-10-29 00:17:41,717 - util.py[WARNING]: '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [119/120s]: url error [timed out]</font><br>
<font face="sans-serif">2013-10-29 00:17:42,718 - DataSourceEc2.py[CRITICAL]: giving up on md after 120 seconds</font><br>
<br>
<font face="sans-serif">no instance data found in start</font><br>
<font face="sans-serif">Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd</font></ul>
<br>
<font face="sans-serif">Here are the metadata values defined in nova.conf:</font><br>
<font face="sans-serif"># grep metadata_ /etc/nova/nova.conf</font><br>
<font face="sans-serif">metadata_host=192.167.11.5</font><br>
<font face="sans-serif">metadata_port=8775</font><br>
<font face="sans-serif">metadata_listen=0.0.0.0</font><br>
<font face="sans-serif">metadata_listen_port=8775</font><br>
<font face="sans-serif">metadata_manager=nova.api.manager.MetadataManager</font><br>
<br>
<font face="sans-serif">I am able to query the metadata server from compute / controller nodes:</font><br>
<font face="sans-serif"># curl <a href="http://192.167.11.5:8775" target="_blank">http://192.167.11.5:8775</a></font><br>
<font face="sans-serif">1.0</font><br>
<font face="sans-serif">...</font><br>
<font face="sans-serif">2008-09-01</font><br>
<font face="sans-serif">2009-04-04</font><br>
<br>
<font face="sans-serif">If I boot cirros image and try to query the metadata server using 169.264.169.254, it fails:</font><br>
<font face="sans-serif">$ curl <a href="http://169.254.169.254/" target="_blank">http://169.254.169.254/</a></font><br>
<font face="sans-serif">curl: (7) couldn't connect to host</font><br>
<br>
<font face="sans-serif">I suspect that IP tables rules that are created to redirect from 169.264.169.254 on the guest to $metadata_host:$metadata_port are not being created correctly - suggestions on how to debug this or why this might be the case? </font><br>
<br>
<font face="sans-serif">Is there documentation that helps with the setup and verification of nova api metadata service?</font><br>
<br>
<font face="sans-serif">Thanks,<br>
Bill Owen <br>
<a href="mailto:billowen@us.ibm.com" target="_blank">billowen@us.ibm.com</a><br>
Strategic Test Methods and Tools <br>
<a href="tel:520-799-4829" value="+15207994829" target="_blank">520-799-4829</a>, T/L 321-4829<br>
</font><br>
<br>
<img src="cid:1__=08BBF680DF9004F38f9e8a93df938@us.ibm.com" alt="Inactive hide details for Pádraig Brady ---10/26/2013 02:41:27 PM---On 10/26/2013 01:43 AM, Bill Owen wrote: > I've just update" border="0" height="16" width="16"><font color="#424282" face="sans-serif">Pádraig Brady ---10/26/2013 02:41:27 PM---On 10/26/2013 01:43 AM, Bill Owen wrote: > I've just updated my test environment to stable-havana.</font><br>
<br>
<font color="#5F5F5F" face="sans-serif" size="1">From: </font><font face="sans-serif" size="1">Pádraig Brady <<a href="mailto:P@draigbrady.com" target="_blank">P@draigbrady.com</a>></font><br>
<font color="#5F5F5F" face="sans-serif" size="1">To: </font><font face="sans-serif" size="1">Bill Owen/Tucson/IBM@IBMUS</font><br>
<font color="#5F5F5F" face="sans-serif" size="1">Cc: </font><font face="sans-serif" size="1"><a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a></font><br>
<font color="#5F5F5F" face="sans-serif" size="1">Date: </font><font face="sans-serif" size="1">10/26/2013 02:41 PM</font><br>
<font color="#5F5F5F" face="sans-serif" size="1">Subject: </font><font face="sans-serif" size="1">Re: [Openstack] Key Injection not working after upgrading from Grizzly to Havana</font><br>
<hr style="color:#8091a5" align="left" noshade size="2" width="100%"><div><div class="h5"><br>
<br>
<br>
<tt><font>On 10/26/2013 01:43 AM, Bill Owen wrote:<br>
> I've just updated my test environment to stable-havana.<br>
> <br>
> I have booted vm instances with Fedora and Ubuntu images with a key_name specified:<br>
> $ nova boot --key_name key <vm-name> --image <image-id> --flavor 2 test_vm<br>
> <br>
> After the image becomes active, I try to ssh to the image, but get an error message: <br>
> $ ssh -i key.pem fedora@<vm-ip-addr><br>
> Permission denied (publickey,gssapi-keyex,gssapi-with-mic).<br>
> <br>
> I tried using keys/images that worked in grizzly, as well as newly created keys and new images following the instructions in the install docs:<br>
> </font></tt><tt><font><a href="http://docs.openstack.org/trunk/install-guide/install/apt/content/nova-boot.html" target="_blank">http://docs.openstack.org/trunk/install-guide/install/apt/content/nova-boot.html</a></font></tt><tt><font><br>
> <br>
> I don't see anything about changes in this area in release notes. Any suggestions on what I might be missing or how to debug would be appreciated!<br>
> In particular, is there a way to increase debug logging so I can see when it tries to do the key injection on the new vm?<br>
> <br>
> FWIW, cirros image boots and I can ssh/login using cirros user and password.<br>
<br>
Injection is not under active development in Havana,<br>
and so theoretically nothing should have changed here.<br>
<br>
Are you using libguestfs to do the injection?<br>
What's the value of the following in nova.conf?<br>
<br>
libvirt_inject_key<br>
libvirt_inject_partition<br>
<br>
Note failure to inject a key does not cause a guest to error,<br>
only failure to inject a user specified file does at present.<br>
However at debug level, messages are printed as to why there<br>
were errors with injecting the other components. So please<br>
set debug=True in nova.conf, restart the nova-compute service,<br>
and try again, keeping an eye on /var/log/nova/compute.log<br>
<br>
thanks,<br>
Pádraig.<br>
<br>
</font></tt><br>
</div></div><p></p></div>
</blockquote></div><br><br clear="all"><br>-- <br>Robert Collins <<a href="mailto:rbtcollins@hp.com" target="_blank">rbtcollins@hp.com</a>><br>Distinguished Technologist<br>HP Converged Cloud
</div>