<html><body bgcolor="#FFFFFF">
<p><font size="2" face="sans-serif">Hi,</font><br>
<font size="2" face="sans-serif">I am using nova networking for now.</font><br>
<br>
<font size="2" face="sans-serif">I've done some more investigation using the cirros vm. From cirros, I am able to ping other compute nodes, as well as outside world (</font><font size="3" face="serif">curl </font><a href="http://www.fedoraproject.org/"><font size="3" color="#0000FF" face="serif"><u>http://www.fedoraproject.org</u></font></a><font size="2" face="sans-serif"> succeeds).</font><br>
<br>
<font size="2" face="sans-serif">But, I am not able to ping the controller node (which is also node specified by metadata_host in nova.conf).</font><br>
<br>
<font size="2" face="sans-serif">I discovered that one of the iptables rules seems to be the problem - it is creating the following rules for nova-network-xxxROUTING:</font><br>
<font size="2" face="sans-serif">-A nova-network-POSTROUTING -s 10.10.100.0/24 -d 192.167.11.5/32 -j ACCEPT </font><br>
<font size="2" face="sans-serif">-A nova-network-POSTROUTING -s 10.10.100.0/24 -d 10.10.100.0/24 -m conntrack ! --ctstate DNAT -j ACCEPT </font><br>
<font size="2" face="sans-serif">-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.167.11.5:8775 </font><br>
<br>
<font size="2" face="sans-serif">The last rule correctly redirects metadata (169.254.169.254) requests from the guest to the metadata server at 192.167.11.5:8775</font><br>
<br>
<font size="2" face="sans-serif">The first rule is confusing - if I temporarily remove it from iptables, I am able to access the metadata server/controller node from the cirros guest, and can send requests to the metadata server by it's IP and by using the "magic ip":</font><br>
<br>
<font size="2" face="sans-serif">$ curl <a href="http://192.167.11.5:8775">http://192.167.11.5:8775</a></font><br>
<font size="2" face="sans-serif">1.0</font><br>
<font size="2" face="sans-serif">2007-01-19</font><br>
<font size="2" face="sans-serif">2007-03-01</font><br>
<font size="2" face="sans-serif">2007-08-29</font><br>
<font size="2" face="sans-serif">2007-10-10</font><br>
<font size="2" face="sans-serif">2007-12-15</font><br>
<font size="2" face="sans-serif">2008-02-01</font><br>
<font size="2" face="sans-serif">2008-09-01</font><br>
<font size="2" face="sans-serif">2009-04-04</font><br>
<font size="2" face="sans-serif">latest$ </font><br>
<font size="2" face="sans-serif">$ </font><br>
<font size="2" face="sans-serif">$ curl <a href="http://169.254.169.254">http://169.254.169.254</a></font><br>
<font size="2" face="sans-serif">1.0</font><br>
<font size="2" face="sans-serif">2007-01-19</font><br>
<font size="2" face="sans-serif">2007-03-01</font><br>
<font size="2" face="sans-serif">2007-08-29</font><br>
<font size="2" face="sans-serif">2007-10-10</font><br>
<font size="2" face="sans-serif">2007-12-15</font><br>
<font size="2" face="sans-serif">2008-02-01</font><br>
<font size="2" face="sans-serif">2008-09-01</font><br>
<font size="2" face="sans-serif">2009-04-04</font><br>
<font size="2" face="sans-serif">latest$ </font><br>
<br>
<font size="2" face="sans-serif">I see that the iptables rule was added to nova.network.linux_net for bug 968453:</font><br>
<font size="2" face="sans-serif"><a href="https://bugs.launchpad.net/nova/+bug/968453">https://bugs.launchpad.net/nova/+bug/968453</a></font><br>
<br>
<font size="2" face="sans-serif">I don't see why this is needed in the general case - the dmz_cidr rule just below in the code seems like it should be sufficient. </font><br>
<br>
<font size="2" face="sans-serif">As a temporary work around, I am running nova-api service on the hypervisor nodes, let metadata_host default to m_ip, and things work correctly. </font><br>
<br>
<font size="2" face="sans-serif">But I am curious what is the correct config setting to make this work.</font><br>
<br>
<font size="2" face="sans-serif">Thanks</font><br>
<font size="2" face="sans-serif">Bill Owen <br>
billowen@us.ibm.com<br>
520-799-4829, T/L 321-4829<br>
</font><br>
<br>
<img width="16" height="16" src="cid:1__=08BBF687DFF598FE8f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for Steven Dake ---10/30/2013 07:03:35 AM---On 10/28/2013 06:14 PM, Bill Owen wrote: >"><font size="2" color="#424282" face="sans-serif">Steven Dake ---10/30/2013 07:03:35 AM---On 10/28/2013 06:14 PM, Bill Owen wrote: ></font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">Steven Dake <sdake@redhat.com></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">Bill Owen/Tucson/IBM@IBMUS, Pádraig Brady <P@draigbrady.com>, Robert Collins <robertc@robertcollins.net>, "Chenrui (A)" <kiwik.chenrui@huawei.com>, </font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Cc: </font><font size="1" face="sans-serif">openstack@lists.openstack.org</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">10/30/2013 07:03 AM</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [Openstack] Key Injection not working after upgrading from Grizzly to Havana</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<font size="3" face="serif">On 10/28/2013 06:14 PM, Bill Owen wrote:</font>
<ul style="padding-left: 36pt"><br>
<font size="2" face="sans-serif">Padraig, Robert, Chenrui,<br>
It seems to be a problem with nova metadata service.</font><font size="3" face="serif"><br>
</font><tt><font size="2"><br>
> Are you using libguestfs to do the injection? <br>
Yes - I wasn't originally, but have installed this on my controller and compute nodes.<br>
<br>
> What's the value of the following in nova.conf?<br>
> libvirt_inject_key<br>
> libvirt_inject_partition</font></tt><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
I have the following set - I added inject_password as well to see if any additional information could be seen.<br>
/etc/nova/nova.conf:libvirt_inject_partition = -1<br>
/etc/nova/nova.conf:libvirt_inject_key = True<br>
/etc/nova/nova.conf:libvirt_inject_password = True</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
When I boot a Ubuntu precise image I see the following in the console log:</font><font size="3" face="serif"> </font>
<ul style="padding-left: 36pt"><br>
<font size="2" face="sans-serif">[ 1.573692] EXT4-fs (vda1): re-mounted. Opts: (null)<br>
cloud-init start-local running: Tue, 29 Oct 2013 00:15:41 +0000. up 5.11 seconds<br>
no instance data found in start-local<br>
ci-info: lo : 1 127.0.0.1 255.0.0.0 .<br>
ci-info: eth0 : 1 10.10.100.2 255.255.255.0 fa:16:3e:a6:5a:98<br>
ci-info: route-0: 0.0.0.0 10.10.100.3 0.0.0.0 eth0 UG<br>
ci-info: route-1: 10.10.100.0 0.0.0.0 255.255.255.0 eth0 U<br>
cloud-init start running: Tue, 29 Oct 2013 00:15:41 +0000. up 5.89 seconds<br>
2013-10-29 00:16:32,646 - util.py[WARNING]: '</font><a href="http://169.254.169.254/2009-04-04/meta-data/instance-id"><font size="2" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/2009-04-04/meta-data/instance-id</u></font></a><font size="2" face="sans-serif">' failed [50/120s]: url error [timed out]<br>
2013-10-29 00:17:23,698 - util.py[WARNING]: '</font><a href="http://169.254.169.254/2009-04-04/meta-data/instance-id"><font size="2" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/2009-04-04/meta-data/instance-id</u></font></a><font size="2" face="sans-serif">' failed [101/120s]: url error [timed out]<br>
2013-10-29 00:17:41,717 - util.py[WARNING]: '</font><a href="http://169.254.169.254/2009-04-04/meta-data/instance-id"><font size="2" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/2009-04-04/meta-data/instance-id</u></font></a><font size="2" face="sans-serif">' failed [119/120s]: url error [timed out]<br>
2013-10-29 00:17:42,718 - DataSourceEc2.py[CRITICAL]: giving up on md after 120 seconds</font><font size="3" face="serif"><br>
</font></ul>
</ul>
<font size="3" face="serif"><br>
These messages imply that your VM cannot access the network. One thing you can try (with your cirros image) is <br>
<br>
wget </font><a href="http://www.fedoraproject.org/"><font size="3" color="#0000FF" face="serif"><u>http://www.fedoraproject.org</u></font></a><font size="3" face="serif"><br>
<br>
This should download a file. If it doesn't, there are a couple of possibilities. One is that your VM is not getting DHCP IP addresses. If this is the case, you will need to debug that directly using the log files and wireshark. You can test this failure scenario by looking at the ip addresses (if one is assigned, dhcp is working).<br>
<br>
Another is that your "outbound" networking isn't working.<br>
<br>
You didn't mention whether your using nova or quantum. I still struggle with getting quantum to work with devstack properly - it goes from working to not working on a regular basis, so typically i just shut it off and use nova networking for the moment.<br>
<br>
Regards<br>
-steve<br>
</font>
<ul style="padding-left: 72pt"><font size="2" face="sans-serif">no instance data found in start<br>
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd</font><font size="3" face="serif"> </font></ul>
<ul style="padding-left: 36pt"><font size="2" face="sans-serif"><br>
Here are the metadata values defined in nova.conf:<br>
# grep metadata_ /etc/nova/nova.conf<br>
metadata_host=192.167.11.5<br>
metadata_port=8775<br>
metadata_listen=0.0.0.0<br>
metadata_listen_port=8775<br>
metadata_manager=nova.api.manager.MetadataManager</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
I am able to query the metadata server from compute / controller nodes:<br>
# curl </font><a href="http://192.167.11.5:8775/"><font size="2" color="#0000FF" face="sans-serif"><u>http://192.167.11.5:8775</u></font></a><font size="2" face="sans-serif"><br>
1.0<br>
...<br>
2008-09-01<br>
2009-04-04</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
If I boot cirros image and try to query the metadata server using 169.264.169.254, it fails:<br>
$ curl </font><a href="http://169.254.169.254/"><font size="2" color="#0000FF" face="sans-serif"><u>http://169.254.169.254/</u></font></a><font size="2" face="sans-serif"><br>
curl: (7) couldn't connect to host</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
I suspect that IP tables rules that are created to redirect from 169.264.169.254 on the guest to $metadata_host:$metadata_port are not being created correctly - suggestions on how to debug this or why this might be the case? </font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
Is there documentation that helps with the setup and verification of nova api metadata service?</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
Thanks,<br>
Bill Owen </font><font size="2" color="#0000FF" face="sans-serif"><u><br>
</u></font><a href="mailto:billowen@us.ibm.com"><font size="2" color="#0000FF" face="sans-serif"><u>billowen@us.ibm.com</u></font></a><font size="2" face="sans-serif"><br>
Strategic Test Methods and Tools <br>
520-799-4829, T/L 321-4829</font><font size="3" face="serif"><br>
<br>
<br>
</font><img src="cid:1__=08BBF687DFF598FE8f9e8a93df938@us.ibm.com" width="16" height="16" alt="Inactive
hide details for Pádraig Brady ---10/26/2013 02:41:27 PM---On
10/26/2013 01:43 AM, Bill Owen wrote: > I've just update"><font size="2" color="#424282" face="sans-serif">Pádraig Brady ---10/26/2013 02:41:27 PM---On 10/26/2013 01:43 AM, Bill Owen wrote: > I've just updated my test environment to stable-havana.</font><font size="3" face="serif"><br>
</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
From: </font><font size="1" face="sans-serif">Pádraig Brady </font><a href="mailto:P@draigbrady.com"><font size="1" color="#0000FF" face="sans-serif"><u><P@draigbrady.com></u></font></a><font size="1" color="#5F5F5F" face="sans-serif"><br>
To: </font><font size="1" face="sans-serif">Bill Owen/Tucson/IBM@IBMUS</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Cc: </font><a href="mailto:openstack@lists.openstack.org"><font size="1" color="#0000FF" face="sans-serif"><u>openstack@lists.openstack.org</u></font></a><font size="1" color="#5F5F5F" face="sans-serif"><br>
Date: </font><font size="1" face="sans-serif">10/26/2013 02:41 PM</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Subject: </font><font size="1" face="sans-serif">Re: [Openstack] Key Injection not working after upgrading from Grizzly to Havana</font><font size="3" face="serif"><br>
</font><hr width="100%" size="2" align="left" noshade><font size="3" face="serif"><br>
<br>
</font><tt><font size="2"><br>
On 10/26/2013 01:43 AM, Bill Owen wrote:<br>
> I've just updated my test environment to stable-havana.<br>
> <br>
> I have booted vm instances with Fedora and Ubuntu images with a key_name specified:<br>
> $ nova boot --key_name key <vm-name> --image <image-id> --flavor 2 test_vm<br>
> <br>
> After the image becomes active, I try to ssh to the image, but get an error message: <br>
> $ ssh -i key.pem fedora@<vm-ip-addr><br>
> Permission denied (publickey,gssapi-keyex,gssapi-with-mic).<br>
> <br>
> I tried using keys/images that worked in grizzly, as well as newly created keys and new images following the instructions in the install docs:<br>
> </font></tt><a href="http://docs.openstack.org/trunk/install-guide/install/apt/content/nova-boot.html"><tt><font size="2" color="#0000FF"><u>http://docs.openstack.org/trunk/install-guide/install/apt/content/nova-boot.html</u></font></tt></a><tt><font size="2"><br>
> <br>
> I don't see anything about changes in this area in release notes. Any suggestions on what I might be missing or how to debug would be appreciated!<br>
> In particular, is there a way to increase debug logging so I can see when it tries to do the key injection on the new vm?<br>
> <br>
> FWIW, cirros image boots and I can ssh/login using cirros user and password.<br>
<br>
Injection is not under active development in Havana,<br>
and so theoretically nothing should have changed here.<br>
<br>
Are you using libguestfs to do the injection?<br>
What's the value of the following in nova.conf?<br>
<br>
libvirt_inject_key<br>
libvirt_inject_partition<br>
<br>
Note failure to inject a key does not cause a guest to error,<br>
only failure to inject a user specified file does at present.<br>
However at debug level, messages are printed as to why there<br>
were errors with injecting the other components. So please<br>
set debug=True in nova.conf, restart the nova-compute service,<br>
and try again, keeping an eye on /var/log/nova/compute.log<br>
<br>
thanks,<br>
Pádraig.<br>
</font></tt><font size="3" face="serif"><br>
<br>
<br>
</font><br>
<tt><font size="3">_______________________________________________<br>
Mailing list: </font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><tt><font size="3" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></tt></a><tt><font size="3"><br>
Post to : </font></tt><a href="mailto:openstack@lists.openstack.org"><tt><font size="3" color="#0000FF"><u>openstack@lists.openstack.org</u></font></tt></a><tt><font size="3"><br>
Unsubscribe : </font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"><tt><font size="3" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</u></font></tt></a><tt><font size="3"><br>
</font></tt></ul>
<br>
</body></html>