[Openstack] Key Injection not working after upgrading from Grizzly to Havana
Bill Owen
billowen at us.ibm.com
Wed Oct 30 19:19:06 UTC 2013
Hi,
I am using nova networking for now.
I've done some more investigation using the cirros vm. From cirros, I am
able to ping other compute nodes, as well as outside world (curl
http://www.fedoraproject.org succeeds).
But, I am not able to ping the controller node (which is also node
specified by metadata_host in nova.conf).
I discovered that one of the iptables rules seems to be the problem - it is
creating the following rules for nova-network-xxxROUTING:
-A nova-network-POSTROUTING -s 10.10.100.0/24 -d 192.167.11.5/32 -j ACCEPT
-A nova-network-POSTROUTING -s 10.10.100.0/24 -d 10.10.100.0/24 -m
conntrack ! --ctstate DNAT -j ACCEPT
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80
-j DNAT --to-destination 192.167.11.5:8775
The last rule correctly redirects metadata (169.254.169.254) requests from
the guest to the metadata server at 192.167.11.5:8775
The first rule is confusing - if I temporarily remove it from iptables, I
am able to access the metadata server/controller node from the cirros
guest, and can send requests to the metadata server by it's IP and by using
the "magic ip":
$ curl http://192.167.11.5:8775
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
latest$
$
$ curl http://169.254.169.254
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
latest$
I see that the iptables rule was added to nova.network.linux_net for bug
968453:
https://bugs.launchpad.net/nova/+bug/968453
I don't see why this is needed in the general case - the dmz_cidr rule just
below in the code seems like it should be sufficient.
As a temporary work around, I am running nova-api service on the hypervisor
nodes, let metadata_host default to m_ip, and things work correctly.
But I am curious what is the correct config setting to make this work.
Thanks
Bill Owen
billowen at us.ibm.com
520-799-4829, T/L 321-4829
From: Steven Dake <sdake at redhat.com>
To: Bill Owen/Tucson/IBM at IBMUS, Pádraig Brady <P at draigbrady.com>,
Robert Collins <robertc at robertcollins.net>, "Chenrui (A)"
<kiwik.chenrui at huawei.com>,
Cc: openstack at lists.openstack.org
Date: 10/30/2013 07:03 AM
Subject: Re: [Openstack] Key Injection not working after upgrading from
Grizzly to Havana
On 10/28/2013 06:14 PM, Bill Owen wrote:
Padraig, Robert, Chenrui,
It seems to be a problem with nova metadata service.
> Are you using libguestfs to do the injection?
Yes - I wasn't originally, but have installed this on my controller
and compute nodes.
> What's the value of the following in nova.conf?
> libvirt_inject_key
> libvirt_inject_partition
I have the following set - I added inject_password as well to see if
any additional information could be seen.
/etc/nova/nova.conf:libvirt_inject_partition = -1
/etc/nova/nova.conf:libvirt_inject_key = True
/etc/nova/nova.conf:libvirt_inject_password = True
When I boot a Ubuntu precise image I see the following in the console
log:
[ 1.573692] EXT4-fs (vda1): re-mounted. Opts: (null)
cloud-init start-local running: Tue, 29 Oct 2013 00:15:41
+0000. up 5.11 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 10.10.100.2 255.255.255.0
fa:16:3e:a6:5a:98
ci-info: route-0: 0.0.0.0 10.10.100.3 0.0.0.0
eth0 UG
ci-info: route-1: 10.10.100.0 0.0.0.0 255.255.255.0
eth0 U
cloud-init start running: Tue, 29 Oct 2013 00:15:41 +0000. up
5.89 seconds
2013-10-29 00:16:32,646 - util.py[WARNING]: '
http://169.254.169.254/2009-04-04/meta-data/instance-id' failed
[50/120s]: url error [timed out]
2013-10-29 00:17:23,698 - util.py[WARNING]: '
http://169.254.169.254/2009-04-04/meta-data/instance-id' failed
[101/120s]: url error [timed out]
2013-10-29 00:17:41,717 - util.py[WARNING]: '
http://169.254.169.254/2009-04-04/meta-data/instance-id' failed
[119/120s]: url error [timed out]
2013-10-29 00:17:42,718 - DataSourceEc2.py[CRITICAL]: giving up
on md after 120 seconds
These messages imply that your VM cannot access the network. One thing you
can try (with your cirros image) is
wget http://www.fedoraproject.org
This should download a file. If it doesn't, there are a couple of
possibilities. One is that your VM is not getting DHCP IP addresses. If
this is the case, you will need to debug that directly using the log files
and wireshark. You can test this failure scenario by looking at the ip
addresses (if one is assigned, dhcp is working).
Another is that your "outbound" networking isn't working.
You didn't mention whether your using nova or quantum. I still struggle
with getting quantum to work with devstack properly - it goes from working
to not working on a regular basis, so typically i just shut it off and use
nova networking for the moment.
Regards
-steve
no instance data found in start
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Here are the metadata values defined in nova.conf:
# grep metadata_ /etc/nova/nova.conf
metadata_host=192.167.11.5
metadata_port=8775
metadata_listen=0.0.0.0
metadata_listen_port=8775
metadata_manager=nova.api.manager.MetadataManager
I am able to query the metadata server from compute / controller
nodes:
# curl http://192.167.11.5:8775
1.0
...
2008-09-01
2009-04-04
If I boot cirros image and try to query the metadata server using
169.264.169.254, it fails:
$ curl http://169.254.169.254/
curl: (7) couldn't connect to host
I suspect that IP tables rules that are created to redirect from
169.264.169.254 on the guest to $metadata_host:$metadata_port are not
being created correctly - suggestions on how to debug this or why
this might be the case?
Is there documentation that helps with the setup and verification of
nova api metadata service?
Thanks,
Bill Owen
billowen at us.ibm.com
Strategic Test Methods and Tools
520-799-4829, T/L 321-4829
Inactive
hide details for Pádraig Brady ---10/26/2013 02:41:27
PM---On
10/26/2013 01:43 AM, Bill Owen wrote: > I've just
updatePádraig Brady ---10/26/2013 02:41:27 PM---On 10/26/2013 01:43
AM, Bill Owen wrote: > I've just updated my test environment to
stable-havana.
From: Pádraig Brady <P at draigbrady.com>
To: Bill Owen/Tucson/IBM at IBMUS
Cc: openstack at lists.openstack.org
Date: 10/26/2013 02:41 PM
Subject: Re: [Openstack] Key Injection not working after upgrading
from Grizzly to Havana
On 10/26/2013 01:43 AM, Bill Owen wrote:
> I've just updated my test environment to stable-havana.
>
> I have booted vm instances with Fedora and Ubuntu images with a
key_name specified:
> $ nova boot --key_name key <vm-name> --image <image-id> --flavor
2 test_vm
>
> After the image becomes active, I try to ssh to the image, but get
an error message:
> $ ssh -i key.pem fedora@<vm-ip-addr>
> Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
>
> I tried using keys/images that worked in grizzly, as well as newly
created keys and new images following the instructions in the install
docs:
>
http://docs.openstack.org/trunk/install-guide/install/apt/content/nova-boot.html
>
> I don't see anything about changes in this area in release notes.
Any suggestions on what I might be missing or how to debug would be
appreciated!
> In particular, is there a way to increase debug logging so I can
see when it tries to do the key injection on the new vm?
>
> FWIW, cirros image boots and I can ssh/login using cirros user and
password.
Injection is not under active development in Havana,
and so theoretically nothing should have changed here.
Are you using libguestfs to do the injection?
What's the value of the following in nova.conf?
libvirt_inject_key
libvirt_inject_partition
Note failure to inject a key does not cause a guest to error,
only failure to inject a user specified file does at present.
However at debug level, messages are printed as to why there
were errors with injecting the other components. So please
set debug=True in nova.conf, restart the nova-compute service,
and try again, keeping an eye on /var/log/nova/compute.log
thanks,
Pádraig.
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131030/f09c9a68/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131030/f09c9a68/attachment.gif>
More information about the Openstack
mailing list
