<html><body>
<p><font size="2" face="sans-serif">No - looking at that now.</font><br>
<font size="2" face="sans-serif">Thanks,<br>
Bill Owen <br>
</font><br>
<br>
<img width="16" height="16" src="cid:1__=08BBF680DF8DD72E8f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for Robert Collins ---10/28/2013 06:55:47 PM---Have you migrated to Neutron at the same time as upgrading"><font size="2" color="#424282" face="sans-serif">Robert Collins ---10/28/2013 06:55:47 PM---Have you migrated to Neutron at the same time as upgrading? There are installer docs for metadata wi</font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">Robert Collins <robertc@robertcollins.net></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">Bill Owen/Tucson/IBM@IBMUS</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Cc: </font><font size="1" face="sans-serif">Pádraig Brady <P@draigbrady.com>, "Chenrui (A)" <kiwik.chenrui@huawei.com>, "openstack@lists.openstack.org" <openstack@lists.openstack.org></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">10/28/2013 06:55 PM</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [Openstack] Key Injection not working after upgrading from Grizzly to Havana</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<font size="3" face="serif">Have you migrated to Neutron at the same time as upgrading?<br>
</font><br>
<font size="3" face="serif">There are installer docs for metadata with Neutron; the troubleshooting process will depend on the production config you chose - e.g. namespaced network, or not etc.<br>
</font><br>
<font size="3" face="serif">Cheers,<br>
Rob</font><br>
<font size="3" face="serif"><br>
</font><br>
<font size="3" face="serif">On 29 October 2013 14:14, Bill Owen <</font><a href="mailto:billowen@us.ibm.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>billowen@us.ibm.com</u></font></a><font size="3" face="serif">> wrote:</font>
<ul style="padding-left: 9pt"><font size="3" face="sans-serif">Padraig, Robert, Chenrui,<br>
It seems to be a problem with nova metadata service.</font>
<p><font size="3" face="serif"><br>
</font><tt><font size="3"><br>
> Are you using libguestfs to do the injection? </font></tt><br>
<tt><font size="3">Yes - I wasn't originally, but have installed this on my controller and compute nodes.</font></tt><br>
<tt><font size="3"><br>
<br>
> What's the value of the following in nova.conf?<br>
> libvirt_inject_key<br>
> libvirt_inject_partition</font></tt><font size="3" face="serif"><br>
</font><br>
<font size="3" face="sans-serif">I have the following set - I added inject_password as well to see if any additional information could be seen.<br>
/etc/nova/nova.conf:libvirt_inject_partition = -1<br>
/etc/nova/nova.conf:libvirt_inject_key = True<br>
/etc/nova/nova.conf:libvirt_inject_password = True</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
When I boot a Ubuntu precise image I see the following in the console log:</font><font size="3" face="serif"> </font>
<ul style="padding-left: 36pt"><font size="3" face="sans-serif">[ 1.573692] EXT4-fs (vda1): re-mounted. Opts: (null)<br>
cloud-init start-local running: Tue, 29 Oct 2013 00:15:41 +0000. up 5.11 seconds<br>
no instance data found in start-local<br>
ci-info: lo : 1 127.0.0.1 255.0.0.0 .<br>
ci-info: eth0 : 1 10.10.100.2 255.255.255.0 fa:16:3e:a6:5a:98<br>
ci-info: route-0: 0.0.0.0 10.10.100.3 0.0.0.0 eth0 UG<br>
ci-info: route-1: 10.10.100.0 0.0.0.0 255.255.255.0 eth0 U<br>
cloud-init start running: Tue, 29 Oct 2013 00:15:41 +0000. up 5.89 seconds<br>
2013-10-29 00:16:32,646 - util.py[WARNING]: '</font><a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/2009-04-04/meta-data/instance-id</u></font></a><font size="3" face="sans-serif">' failed [50/120s]: url error [timed out]<br>
2013-10-29 00:17:23,698 - util.py[WARNING]: '</font><a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/2009-04-04/meta-data/instance-id</u></font></a><font size="3" face="sans-serif">' failed [101/120s]: url error [timed out]<br>
2013-10-29 00:17:41,717 - util.py[WARNING]: '</font><a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/2009-04-04/meta-data/instance-id</u></font></a><font size="3" face="sans-serif">' failed [119/120s]: url error [timed out]<br>
2013-10-29 00:17:42,718 - DataSourceEc2.py[CRITICAL]: giving up on md after 120 seconds</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
no instance data found in start<br>
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd</font></ul>
<font size="3" face="sans-serif"><br>
Here are the metadata values defined in nova.conf:<br>
# grep metadata_ /etc/nova/nova.conf<br>
metadata_host=192.167.11.5<br>
metadata_port=8775<br>
metadata_listen=0.0.0.0<br>
metadata_listen_port=8775<br>
metadata_manager=nova.api.manager.MetadataManager</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
I am able to query the metadata server from compute / controller nodes:<br>
# curl </font><a href="http://192.167.11.5:8775/" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>http://192.167.11.5:8775</u></font></a><font size="3" face="sans-serif"><br>
1.0<br>
...<br>
2008-09-01<br>
2009-04-04</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
If I boot cirros image and try to query the metadata server using 169.264.169.254, it fails:<br>
$ curl </font><a href="http://169.254.169.254/" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/</u></font></a><font size="3" face="sans-serif"><br>
curl: (7) couldn't connect to host</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
I suspect that IP tables rules that are created to redirect from 169.264.169.254 on the guest to $metadata_host:$metadata_port are not being created correctly - suggestions on how to debug this or why this might be the case? </font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
Is there documentation that helps with the setup and verification of nova api metadata service?</font><font size="3" face="serif"><br>
</font><font size="3" face="sans-serif"><br>
Thanks,<br>
Bill Owen </font><font size="3" color="#0000FF" face="sans-serif"><u><br>
</u></font><a href="mailto:billowen@us.ibm.com" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>billowen@us.ibm.com</u></font></a><font size="3" face="sans-serif"><br>
Strategic Test Methods and Tools </font><font size="3" color="#0000FF" face="sans-serif"><u><br>
</u></font><a href="tel:520-799-4829" target="_blank"><font size="3" color="#0000FF" face="sans-serif"><u>520-799-4829</u></font></a><font size="3" face="sans-serif">, T/L 321-4829</font><font size="3" face="serif"><br>
<br>
<br>
</font><img src="cid:1__=08BBF680DF8DD72E8f9e8a93df938@us.ibm.com" width="16" height="16" alt="Inactive hide details for Pádraig Brady ---10/26/2013 02:41:27 PM---On 10/26/2013 01:43 AM, Bill Owen wrote: > I've just update"><font size="3" color="#424282" face="sans-serif">Pádraig Brady ---10/26/2013 02:41:27 PM---On 10/26/2013 01:43 AM, Bill Owen wrote: > I've just updated my test environment to stable-havana.</font><font size="3" face="serif"><br>
</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
From: </font><font size="1" face="sans-serif">Pádraig Brady <</font><a href="mailto:P@draigbrady.com" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>P@draigbrady.com</u></font></a><font size="1" face="sans-serif">></font><font size="1" color="#5F5F5F" face="sans-serif"><br>
To: </font><font size="1" face="sans-serif">Bill Owen/Tucson/IBM@IBMUS</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Cc: </font><a href="mailto:openstack@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" color="#5F5F5F" face="sans-serif"><br>
Date: </font><font size="1" face="sans-serif">10/26/2013 02:41 PM</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Subject: </font><font size="1" face="sans-serif">Re: [Openstack] Key Injection not working after upgrading from Grizzly to Havana</font><font size="3" face="serif"><br>
</font><hr width="100%" size="2" align="left" noshade><br>
<font size="3" face="serif"><br>
<br>
</font><tt><font size="3"><br>
On 10/26/2013 01:43 AM, Bill Owen wrote:<br>
> I've just updated my test environment to stable-havana.<br>
> <br>
> I have booted vm instances with Fedora and Ubuntu images with a key_name specified:<br>
> $ nova boot --key_name key <vm-name> --image <image-id> --flavor 2 test_vm<br>
> <br>
> After the image becomes active, I try to ssh to the image, but get an error message: <br>
> $ ssh -i key.pem fedora@<vm-ip-addr><br>
> Permission denied (publickey,gssapi-keyex,gssapi-with-mic).<br>
> <br>
> I tried using keys/images that worked in grizzly, as well as newly created keys and new images following the instructions in the install docs:<br>
> </font></tt><a href="http://docs.openstack.org/trunk/install-guide/install/apt/content/nova-boot.html" target="_blank"><tt><font size="3" color="#0000FF"><u>http://docs.openstack.org/trunk/install-guide/install/apt/content/nova-boot.html</u></font></tt></a><tt><font size="3"><br>
> <br>
> I don't see anything about changes in this area in release notes. Any suggestions on what I might be missing or how to debug would be appreciated!<br>
> In particular, is there a way to increase debug logging so I can see when it tries to do the key injection on the new vm?<br>
> <br>
> FWIW, cirros image boots and I can ssh/login using cirros user and password.<br>
<br>
Injection is not under active development in Havana,<br>
and so theoretically nothing should have changed here.<br>
<br>
Are you using libguestfs to do the injection?<br>
What's the value of the following in nova.conf?<br>
<br>
libvirt_inject_key<br>
libvirt_inject_partition<br>
<br>
Note failure to inject a key does not cause a guest to error,<br>
only failure to inject a user specified file does at present.<br>
However at debug level, messages are printed as to why there<br>
were errors with injecting the other components. So please<br>
set debug=True in nova.conf, restart the nova-compute service,<br>
and try again, keeping an eye on /var/log/nova/compute.log<br>
<br>
thanks,<br>
Pádraig.<br>
</font></tt><font size="3" face="serif"><br>
</font></ul>
<br>
<font size="3" face="serif"><br>
<br>
<br>
-- <br>
Robert Collins <</font><a href="mailto:rbtcollins@hp.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>rbtcollins@hp.com</u></font></a><font size="3" face="serif">><br>
Distinguished Technologist<br>
HP Converged Cloud </font>
<p></body></html>