[Openstack] [ironic]Failed to boot ironic instance
Li, Chen
chen.li at intel.com
Mon Jun 30 01:06:51 UTC 2014
Updates:
The final issue is caused by bug https://bugs.launchpad.net/diskimage-builder/+bug/1311987
I can now successfully boot an instance under all in one devstack environment.
Thanks for all your helps.
-chen
From: Li, Chen
Sent: Saturday, June 28, 2014 11:36 AM
To: 'Chris K'
Cc: openstack at lists.openstack.org
Subject: RE: [Openstack] [ironic]Failed to boot ironic instance
Updates:
Get help from adam_g in IRC.
Tested the tftp server from the physical node using command “busybox tftp -g -r pxelinux.0 -l /tmp/pxelinux.0 192.168.50.1”.
The command can work, that means tftp server is working.
So we assume it is a network issue.
Looks like traffic from the tenant neutron network is having trouble reaching "outside" to the hosts ip.
Actually, I haven’t enable l3 service in neutron, so I reinstalled devstack with L3 service enabled.
After the reinstallation, the ovs bridge br-ex has IP 172.24.4.1 while the public network devstack created has subnet 172.24.4.0/24.
br-ex Link encap:Ethernet HWaddr 66:70:89:aa:08:46
inet addr:172.24.4.1 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:68790 errors:0 dropped:0 overruns:0 frame:0
TX packets:68772 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3165047 (3.1 MB) TX bytes:99970878 (99.9 MB)
So, I edit IP for tftp_server in /etc/ironic/ironic.conf:
tftp_server = 172.24.4.1
And restart ironic-api and ironic-conductor services.
Then, I re-run command:
nova boot --flavor baremetal --image cirros-0.3.2-x86_64-uec --nic net-id=93a8b67d-7a48-4304-97d1-b91c2e493fd1 test1
And again, using VNCviewer to check how the instance works.
It can reach tftp server now I think, but still, it hangs there:
Output of the console, in case you can’t see the picture view:
http://paste.openstack.org/show/85104/
[cid:image001.png at 01CF9442.A0D575D0]
Thanks.
-chen
From: Li, Chen
Sent: Saturday, June 28, 2014 8:48 AM
To: 'Chris K'
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: RE: [Openstack] [ironic]Failed to boot ironic instance
Hello, Chris
Thanks for the kindly reply.
I’m fresh new to ironic and PXE and so on, my first time try to install it.
Hope the following information is helpful.
This is my log for ironic-conductor log:
http://paste.openstack.org/show/85099/
I didn’t find any errors there.
For tftp port, not very sure, I googled, it should be udp port 69, right ??
sudo netstat -an |grep udp |grep 69
udp 0 0 0.0.0.0:69 0.0.0.0:*
sudo iptables -L -n |grep 69
ACCEPT udp -- 0.0.0.0/0 192.168.50.1 udp dpt:69
I couldn’t find logs for tftp: ☹
pwd
/var/log
ls
alternatives.log boot dist-upgrade dpkg.log installer libvirt mysql news syslog upstart
apt boot.log dmesg faillog kern.log mail.err mysql.err openvswitch udev wtmp
auth.log btmp dmesg.0 fsck lastlog mail.log mysql.log qpid ufw.log
Which directory is tftp directory ?
Based on contexts in ironic.conf:
cat /etc/ironic/ironic.conf |grep tftp
tftp_master_path = /opt/stack/data/ironic/tftpboot/master_images
tftp_root = /opt/stack/data/ironic/tftpboot
tftp_server = 192.168.50.1
# Directory where master tftp images are stored on disk.
#tftp_master_path=/tftpboot/master_images
[tftp]
# Options defined in ironic.common.tftp
# IP address of Ironic compute node's tftp server. (string
#tftp_server=$my_ip
# Ironic compute node's tftp root path. (string value)
#tftp_root=/tftpboot
Here is my output for these directories :
ll /opt/stack/data/ironic/tftpboot/master_images
total 103020
drwxr-xr-x 2 stack stack 4096 Jun 27 20:08 ./
drwxr-xr-x 5 stack libvirtd 4096 Jun 28 08:19 ../
-rw-r--r-- 2 stack stack 3723817 Jun 27 20:08 23244369-febf-48d9-acb0-e9d58807469b
-rw-r--r-- 2 stack stack 4969360 Jun 27 20:08 239aacb7-398a-4281-8ab1-ac42811eef3f
-rw-r--r-- 2 stack stack 5792544 Jun 27 20:08 ac21a5c7-4ec4-4a85-a778-d0e56ba0a075
-rw-r--r-- 2 stack stack 90987013 Jun 27 20:08 cfa1d908-a002-4303-91e0-b7d8182ae151
ll /opt/stack/data/ironic/tftpboot
total 56
drwxr-xr-x 5 stack libvirtd 4096 Jun 28 08:19 ./
drwxr-xr-x 5 stack root 4096 Jun 27 20:05 ../
drwxr-xr-x 2 stack stack 4096 Jun 28 08:18 8f485798-4dbd-4eb2-9202-cf1f5a1aebda/
-rwxr-xr-x 1 stack stack 96 Jun 27 20:05 map-file*
drwxr-xr-x 2 stack stack 4096 Jun 27 20:08 master_images/
-rwxr-xr-x 1 stack stack 26720 Jun 27 19:58 pxelinux.0*
drwxr-xr-x 2 stack stack 4096 Jun 28 08:18 pxelinux.cfg/
-rw-r--r-- 1 stack stack 32 Jun 28 08:19 token-8f485798-4dbd-4eb2-9202-cf1f5a1aebda
ll /tftpboot/master_images
ls: cannot access /tftpboot/master_images: No such file or directory
ll /tftpboot/
total 36
drwxrwxrwx 2 root root 4096 Jun 26 14:49 ./
drwxr-xr-x 23 root root 4096 Jun 26 14:48 ../
-rw-r--r-- 1 stack stack 26720 Jun 26 14:49 pxelinux.0
ll /opt/stack/data/ironic/tftpboot/8f485798-4dbd-4eb2-9202-cf1f5a1aebda
total 103024
drwxr-xr-x 2 stack stack 4096 Jun 28 08:18 ./
drwxr-xr-x 5 stack libvirtd 4096 Jun 28 08:19 ../
-rw-r--r-- 1 stack stack 763 Jun 28 08:18 config
-rw-r--r-- 2 stack stack 5792544 Jun 27 20:08 deploy_kernel
-rw-r--r-- 2 stack stack 90987013 Jun 27 20:08 deploy_ramdisk
-rw-r--r-- 2 stack stack 4969360 Jun 27 20:08 kernel
-rw-r--r-- 2 stack stack 3723817 Jun 27 20:08 ramdisk
ironic node-list
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| uuid | instance_uuid | power_state | provision_state | maintenance |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| 568f5dce-565a-4fd6-8d8e-00f7f776a7ec | None | power off | None | False |
| ebf8c6cc-8477-4f9e-8cd0-fb0b418f214a | None | power off | None | False |
| 8f485798-4dbd-4eb2-9202-cf1f5a1aebda | ad56e8dc-20c4-4f55-bf0c-cbacf4ca01af | power on | wait call-back | False |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| 641962be-f8a8-4c3f-8ff1-ba3bb87cf77b | cirros-0.3.2-x86_64-uec | ami | ami | 25165824 | active |
| 239aacb7-398a-4281-8ab1-ac42811eef3f | cirros-0.3.2-x86_64-uec-kernel | aki | aki | 4969360 | active |
| 23244369-febf-48d9-acb0-e9d58807469b | cirros-0.3.2-x86_64-uec-ramdisk | ari | ari | 3723817 | active |
| cfa1d908-a002-4303-91e0-b7d8182ae151 | ir-deploy.initramfs | ari | ari | 90987013 | active |
| ac21a5c7-4ec4-4a85-a778-d0e56ba0a075 | ir-deploy.kernel | aki | aki | 5792544 | active |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
From: Chris K [mailto:nobodycam at gmail.com]
Sent: Friday, June 27, 2014 10:44 PM
To: Li, Chen
Subject: Re: [Openstack] [ironic]Failed to boot ironic instance
Hi Chen,
Are there any errors in the ironic-conductor log. Some things I would check are: Is tftp port is open on any firewall, look for any tftp errors in /var/log/. Also check that the pexlinux.0 file is in the tftp directory and readable by the tftpd service. By the looks of it you are getting good dhcp values back from the conductor, so your close.
Chris
On Fri, Jun 27, 2014 at 5:42 AM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
The output of the instance console (from VNCviewer, in case you couldn’t get the pic)
Booting from ROM..
iPXE (PCI 00:04:0) STARTING EXECTION …OK
iPXE initializing dvices…ok
iPXE 1.0.0+git-20131111.c3d1e78-2ubuntu1 –Open Source Network Boot Firmware –
http://ipxe.org
Features: HTTP HTTPS ISCSI DNS TFTP AoE bzImage ELF MBOOT PXE PXEXT Menu
net0: 52:54:00:8a:7f:0a using 82540em on PCI00:04.0(open)
[Link:up TX:0 TXE:0 RX:0 RXE:0]
Configuring(net0 52:54:00:8a:7f:)……….ok
net0:10.1.0.4/255.255.255.0<http://10.1.0.4/255.255.255.0> gw 10.1.0.1
Next server:192.168.50.1
Filename: pxelinux.0
tftp://192.168.50.1/pxelinux.0<http://192.168.50.1/pxelinux.0> ................ Connection time out (http://ipxe.org/4c126035)
No more network devices
No bootable device
From: Li, Chen
Sent: Friday, June 27, 2014 8:19 PM
To: Li, Chen; Kai Brennenstuhl
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: RE: [Openstack] [ironic]Failed to boot ironic instance
Hi list,
Based on the warning information:
WARNING ironic.common.neutron [-] No VIFs found for node c9caca46-64a7-44d5-b6db-554077b2473a when attempting to update Neutron DHCP BOOT options.
I changed to run command with nic paramter:
nova boot --flavor baremetal --image cirros-0.3.2-x86_64-uec --nic net-id=b957c908-39bc-4de5-b620-addfd5618680 test
Now I can get IP address by command “nova list”:
nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 6b513e8d-6f1b-41ce-b3b1-84cbc221e632 | test | BUILD | spawning | NOSTATE | private=10.1.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------+
But still, the instance stays under spawning status.
Then I using VNCviewer directly reach the instance to check status:
I get :
[cid:image002.png at 01CF9442.A0D575D0]
Looks like it failed to connect to tftp ???
What should I do next to debug this ???
Thanks.
-chen
From: Li, Chen [mailto:chen.li at intel.com]
Sent: Friday, June 27, 2014 3:14 PM
To: Kai Brennenstuhl
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] [ironic]Failed to boot ironic instance
Hi Kai,
Thanks for the reply.
I have seem your questions.
And, yes, I think we have met almost the same issue.
I think the dnsmasq process of neutron did not assign an ip to baremetal instance is because neutron-server didn’t allocated any IP.
You can see there is a parameter for dnsmasq process
--dhcp-hostsfile
dnsmasq process would only response to DHCP request matches MAC address exists in file -dhcp-hostsfile (at least this is how it works when there is no ironic involved).
I checked neutron-server and neutron database, no IP allocation happened.
So, there is no way dnsmasq process would response to any DHCP request.
If another DHCP server can work around this issue, then I guess we might try to solve the warning in ironic-conductor service log first:
WARNING ironic.common.neutron [-] No VIFs found for node c9caca46-64a7-44d5-b6db-554077b2473a when attempting to update Neutron DHCP BOOT options.
Thanks.
-chen
From: Kai Brennenstuhl [mailto:kai.brennenstuhl at gmail.com]
Sent: Friday, June 27, 2014 2:55 PM
To: Li, Chen
Subject: Re: [Openstack] [ironic]Failed to boot ironic instance
Hi Chen,
I think I had exactly the same problem like you [1]. My problem was that the dnsmasq process of neutron did not assign an ip to the nested baremetal instance. You can check with tcpdump if there are DHCP Boot Requests and if they are replied. I have not solved the issue but did a workaround. I was using Ubuntu 14.04 as controller VM. I configured the dhcp server by myself using isc-dhcp-server [2]. The booting baremetal VM received DHCP offers and everything worked fine. I did not configure PXE or tftp (this was done by ironic).
[1] https://ask.openstack.org/en/question/31253/ironic-node-stuck-in-provisioning-state-wait-call-back/
[2] http://wiki.ubuntuusers.de/ISC-DHCPD
Kai
2014-06-27 8:21 GMT+02:00 Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>>:
Hi list,
I'm trying to install/using ironic.
I have followed steps here :
http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html#deploying-ironic-with-devstack
While, after I run command "nova boot --flavor baremetal --image cirros-0.3.2-x86_64-uec test1"
My instance stays in BUILD and spawning for a really long time, and become ERROR at end.
nova list
+--------------------------------------+-------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+----------+
| bd10abaa-7cb4-492e-9afd-5027303631f1 | test1 | ERROR | - | Running | |
| c3e4fccb-dc2b-41dc-8270-2cce5e8fe404 | test2 | BUILD | spawning | NOSTATE | |
| 3dcfabba-ca32-44ad-988a-c22f7308ed7a | test3 | BUILD | spawning | NOSTATE | |
+--------------------------------------+-------+--------+------------+-------------+----------+
This is my log for compute.log:
http://paste.openstack.org/
Looks like it is waiting ironic node to become ACTIVE.
This is my ironic-conductor service log:
http://paste.openstack.org/show/85043/
No obvious error message can be find, but there is a warning:
WARNING ironic.common.neutron [-] No VIFs found for node c9caca46-64a7-44d5-b6db-554077b2473a when attempting to update Neutron DHCP BOOT options.
Is this warning caused instance can't start ??
I also check neutron-server, no IP has been allocated at all.
Anyone know why this is happened ???
Thanks.
-chen
Other commands output:
ironic node-list
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| uuid | instance_uuid | power_state | provision_state | maintenance |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| c9caca46-64a7-44d5-b6db-554077b2473a | 3dcfabba-ca32-44ad-988a-c22f7308ed7a | power on | wait call-back | False |
| 6d6c29e0-b81d-4ed9-9e2a-675b5221aba9 | bd10abaa-7cb4-492e-9afd-5027303631f1 | power on | wait call-back | False |
| 9604af4b-a61a-4157-8431-e61235de7b92 | c3e4fccb-dc2b-41dc-8270-2cce5e8fe404 | power on | wait call-back | False |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
ironic node-show c9caca46-64a7-44d5-b6db-554077b2473a
+------------------------+--------------------------------------------------------------------------+
| Property | Value |
+------------------------+--------------------------------------------------------------------------+
| instance_uuid | 3dcfabba-ca32-44ad-988a-c22f7308ed7a |
| target_power_state | None |
| properties | {u'memory_mb': u'512', u'cpu_arch': u'x86_64', u'local_gb': u'10', |
| | u'cpus': u'1'} |
| maintenance | False |
| driver_info | {u'pxe_deploy_ramdisk': u'cfdbb01b-fa26-4583-a29f-f2f0111499a5', |
| | u'ssh_port': u'22', u'ssh_username': u'stack', u'pxe_deploy_kernel': |
| | u'c5920355-e4ea-4c09-b774-98b85474e24f', u'ssh_virt_type': u'virsh', |
| | u'ssh_address': u'192.168.50.1', u'ssh_key_filename': |
| | u'/opt/stack/data/ironic/ssh_keys/ironic_key'} |
| extra | {} |
| last_error | None |
| created_at | 2014-06-27T05:47:27+00:00 |
| target_provision_state | deploy complete |
| driver | pxe_ssh |
| updated_at | 2014-06-27T06:03:20+00:00 |
| instance_info | {u'ramdisk': u'9ea0ac76-a07c-44db-a9b7-328839d0d358', u'kernel': |
| | u'cecfc044-6fd5-49af-b91c-8ac90b4e7976', u'root_gb': u'10', |
| | u'image_source': u'94ba7f9a-4de0-4b00-b07a-85c5647b11de', u'deploy_key': |
| | u'8GV962GHNSIDQYX30N0SVB94MCSAG4GE', u'swap_mb': u'0'} |
| chassis_uuid | 3183c4f1-b7d6-49a7-8214-1c850d468a7e |
| provision_state | wait call-back |
| reservation | None |
| power_state | power on |
| console_enabled | False |
| uuid | c9caca46-64a7-44d5-b6db-554077b2473a |
+------------------------+--------------------------------------------------------------------------+
nova image-list
+--------------------------------------+---------------------------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| 94ba7f9a-4de0-4b00-b07a-85c5647b11de | cirros-0.3.2-x86_64-uec | ACTIVE | |
| cecfc044-6fd5-49af-b91c-8ac90b4e7976 | cirros-0.3.2-x86_64-uec-kernel | ACTIVE | |
| 9ea0ac76-a07c-44db-a9b7-328839d0d358 | cirros-0.3.2-x86_64-uec-ramdisk | ACTIVE | |
| cfdbb01b-fa26-4583-a29f-f2f0111499a5 | ir-deploy.initramfs | ACTIVE | |
| c5920355-e4ea-4c09-b774-98b85474e24f | ir-deploy.kernel | ACTIVE | |
+--------------------------------------+---------------------------------+--------+--------+
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140630/6324a2e7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 27623 bytes
Desc: image001.png
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140630/6324a2e7/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 22893 bytes
Desc: image002.png
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140630/6324a2e7/attachment-0001.png>
More information about the Openstack
mailing list
