[Openstack] [ironic]Failed to boot ironic instance

Li, Chen chen.li at intel.com
Mon Jun 30 01:06:51 UTC 2014


Updates:

The final issue is caused by bug https://bugs.launchpad.net/diskimage-builder/+bug/1311987

I can now successfully boot  an instance under all in one devstack environment.

Thanks for all your helps.

-chen

From: Li, Chen
Sent: Saturday, June 28, 2014 11:36 AM
To: 'Chris K'
Cc: openstack at lists.openstack.org
Subject: RE: [Openstack] [ironic]Failed to boot ironic instance

Updates:

Get help from adam_g in IRC.

Tested the tftp server from the physical node  using command “busybox tftp -g -r pxelinux.0 -l /tmp/pxelinux.0 192.168.50.1”.
The command can work, that means tftp server is working.
So we assume it is a network issue.
Looks like traffic from the tenant neutron network is having trouble reaching "outside" to the hosts ip.

Actually, I  haven’t enable l3 service in neutron, so I reinstalled devstack with L3 service enabled.

After the reinstallation, the ovs bridge br-ex has IP 172.24.4.1  while the public network devstack created has subnet  172.24.4.0/24.

br-ex     Link encap:Ethernet  HWaddr 66:70:89:aa:08:46
          inet addr:172.24.4.1  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:68790 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68772 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3165047 (3.1 MB)  TX bytes:99970878 (99.9 MB)

So, I edit IP for tftp_server in /etc/ironic/ironic.conf:
tftp_server = 172.24.4.1

And restart ironic-api and ironic-conductor services.

Then, I re-run command:
nova boot --flavor baremetal --image cirros-0.3.2-x86_64-uec --nic net-id=93a8b67d-7a48-4304-97d1-b91c2e493fd1 test1

And again, using VNCviewer to check how the instance works.

It can reach tftp server now I think, but still, it hangs there:
Output of the console, in case you can’t see the picture view:
http://paste.openstack.org/show/85104/

[cid:image001.png at 01CF9442.A0D575D0]



Thanks.
-chen

From: Li, Chen
Sent: Saturday, June 28, 2014 8:48 AM
To: 'Chris K'
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: RE: [Openstack] [ironic]Failed to boot ironic instance

Hello, Chris

Thanks for the kindly reply.
I’m fresh new to ironic and PXE and so on, my first time try to install it.
Hope the following information is helpful.

This is my log for ironic-conductor log:
http://paste.openstack.org/show/85099/
I didn’t find any errors there.

For tftp port, not very sure, I googled, it should be udp port 69, right ??
sudo netstat -an |grep udp |grep 69
udp        0      0 0.0.0.0:69              0.0.0.0:*

               sudo iptables -L -n |grep 69
ACCEPT     udp  --  0.0.0.0/0            192.168.50.1         udp dpt:69

I couldn’t find logs for tftp: ☹
pwd
/var/log

ls
alternatives.log  boot      dist-upgrade  dpkg.log  installer  libvirt   mysql      news         syslog   upstart
apt               boot.log  dmesg         faillog   kern.log   mail.err  mysql.err  openvswitch  udev     wtmp
auth.log          btmp      dmesg.0       fsck      lastlog    mail.log  mysql.log  qpid         ufw.log

Which directory is  tftp directory ?

Based on contexts in  ironic.conf:
cat /etc/ironic/ironic.conf |grep tftp
tftp_master_path = /opt/stack/data/ironic/tftpboot/master_images
tftp_root = /opt/stack/data/ironic/tftpboot
tftp_server = 192.168.50.1
# Directory where master tftp images are stored on disk.
#tftp_master_path=/tftpboot/master_images
[tftp]
# Options defined in ironic.common.tftp
# IP address of Ironic compute node's tftp server. (string
#tftp_server=$my_ip
# Ironic compute node's tftp root path. (string value)
#tftp_root=/tftpboot

Here is my output for these directories :
ll /opt/stack/data/ironic/tftpboot/master_images
total 103020
drwxr-xr-x 2 stack stack        4096 Jun 27 20:08 ./
drwxr-xr-x 5 stack libvirtd     4096 Jun 28 08:19 ../
-rw-r--r-- 2 stack stack     3723817 Jun 27 20:08 23244369-febf-48d9-acb0-e9d58807469b
-rw-r--r-- 2 stack stack     4969360 Jun 27 20:08 239aacb7-398a-4281-8ab1-ac42811eef3f
-rw-r--r-- 2 stack stack     5792544 Jun 27 20:08 ac21a5c7-4ec4-4a85-a778-d0e56ba0a075
-rw-r--r-- 2 stack stack    90987013 Jun 27 20:08 cfa1d908-a002-4303-91e0-b7d8182ae151


ll /opt/stack/data/ironic/tftpboot
total 56
drwxr-xr-x 5 stack libvirtd  4096 Jun 28 08:19 ./
drwxr-xr-x 5 stack root      4096 Jun 27 20:05 ../
drwxr-xr-x 2 stack stack     4096 Jun 28 08:18 8f485798-4dbd-4eb2-9202-cf1f5a1aebda/
-rwxr-xr-x 1 stack stack       96 Jun 27 20:05 map-file*
drwxr-xr-x 2 stack stack     4096 Jun 27 20:08 master_images/
-rwxr-xr-x 1 stack stack    26720 Jun 27 19:58 pxelinux.0*
drwxr-xr-x 2 stack stack     4096 Jun 28 08:18 pxelinux.cfg/
-rw-r--r-- 1 stack stack       32 Jun 28 08:19 token-8f485798-4dbd-4eb2-9202-cf1f5a1aebda

ll /tftpboot/master_images
ls: cannot access /tftpboot/master_images: No such file or directory

ll /tftpboot/
total 36
drwxrwxrwx  2 root  root   4096 Jun 26 14:49 ./
drwxr-xr-x 23 root  root   4096 Jun 26 14:48 ../
-rw-r--r--  1 stack stack 26720 Jun 26 14:49 pxelinux.0

ll /opt/stack/data/ironic/tftpboot/8f485798-4dbd-4eb2-9202-cf1f5a1aebda
total 103024
drwxr-xr-x 2 stack stack        4096 Jun 28 08:18 ./
drwxr-xr-x 5 stack libvirtd     4096 Jun 28 08:19 ../
-rw-r--r-- 1 stack stack         763 Jun 28 08:18 config
-rw-r--r-- 2 stack stack     5792544 Jun 27 20:08 deploy_kernel
-rw-r--r-- 2 stack stack    90987013 Jun 27 20:08 deploy_ramdisk
-rw-r--r-- 2 stack stack     4969360 Jun 27 20:08 kernel
-rw-r--r-- 2 stack stack     3723817 Jun 27 20:08 ramdisk


ironic node-list
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| uuid                                 | instance_uuid                        | power_state | provision_state | maintenance |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| 568f5dce-565a-4fd6-8d8e-00f7f776a7ec | None                                 | power off   | None            | False       |
| ebf8c6cc-8477-4f9e-8cd0-fb0b418f214a | None                                 | power off   | None            | False       |
| 8f485798-4dbd-4eb2-9202-cf1f5a1aebda | ad56e8dc-20c4-4f55-bf0c-cbacf4ca01af | power on    | wait call-back  | False       |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+

glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size     | Status |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| 641962be-f8a8-4c3f-8ff1-ba3bb87cf77b | cirros-0.3.2-x86_64-uec         | ami         | ami              | 25165824 | active |
| 239aacb7-398a-4281-8ab1-ac42811eef3f | cirros-0.3.2-x86_64-uec-kernel  | aki         | aki              | 4969360  | active |
| 23244369-febf-48d9-acb0-e9d58807469b | cirros-0.3.2-x86_64-uec-ramdisk | ari         | ari              | 3723817  | active |
| cfa1d908-a002-4303-91e0-b7d8182ae151 | ir-deploy.initramfs             | ari         | ari              | 90987013 | active |
| ac21a5c7-4ec4-4a85-a778-d0e56ba0a075 | ir-deploy.kernel                | aki         | aki              | 5792544  | active |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
From: Chris K [mailto:nobodycam at gmail.com]
Sent: Friday, June 27, 2014 10:44 PM
To: Li, Chen
Subject: Re: [Openstack] [ironic]Failed to boot ironic instance

Hi Chen,

Are there any errors in the ironic-conductor log. Some things I would check are: Is tftp port is open on any firewall, look for any tftp errors in /var/log/. Also check that the pexlinux.0 file is in the tftp directory and readable by the tftpd service.  By the looks of it you are getting good dhcp values back from the conductor, so your close.

Chris

On Fri, Jun 27, 2014 at 5:42 AM, Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>> wrote:
The output of the instance console (from VNCviewer, in case you couldn’t get the pic)

Booting from ROM..
iPXE (PCI 00:04:0) STARTING EXECTION …OK
iPXE initializing dvices…ok

iPXE 1.0.0+git-20131111.c3d1e78-2ubuntu1 –Open Source Network Boot Firmware –
  http://ipxe.org
Features: HTTP HTTPS ISCSI DNS TFTP AoE bzImage ELF MBOOT PXE PXEXT Menu

net0:  52:54:00:8a:7f:0a using 82540em on PCI00:04.0(open)
  [Link:up TX:0 TXE:0 RX:0 RXE:0]
Configuring(net0 52:54:00:8a:7f:)……….ok
net0:10.1.0.4/255.255.255.0<http://10.1.0.4/255.255.255.0> gw 10.1.0.1
Next server:192.168.50.1
Filename: pxelinux.0
tftp://192.168.50.1/pxelinux.0<http://192.168.50.1/pxelinux.0> ................ Connection time out (http://ipxe.org/4c126035)
No more network devices

No bootable device

From: Li, Chen
Sent: Friday, June 27, 2014 8:19 PM
To: Li, Chen; Kai Brennenstuhl
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: RE: [Openstack] [ironic]Failed to boot ironic instance

Hi list,

Based on the warning information:
WARNING ironic.common.neutron [-] No VIFs found for node c9caca46-64a7-44d5-b6db-554077b2473a when attempting to update Neutron DHCP BOOT options.
I changed to run command with nic paramter:
nova boot --flavor baremetal --image cirros-0.3.2-x86_64-uec --nic net-id=b957c908-39bc-4de5-b620-addfd5618680 test

Now I can get IP address by command “nova list”:
nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID                                   | Name | Status | Task State | Power State | Networks         |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 6b513e8d-6f1b-41ce-b3b1-84cbc221e632 | test | BUILD  | spawning   | NOSTATE     | private=10.1.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------+

But still, the instance stays under  spawning   status.

Then I using VNCviewer directly reach the instance to check status:
I get :
       [cid:image002.png at 01CF9442.A0D575D0]

Looks like it failed to connect to tftp ???

What should I do next to debug this ???

Thanks.
-chen

From: Li, Chen [mailto:chen.li at intel.com]
Sent: Friday, June 27, 2014 3:14 PM
To: Kai Brennenstuhl
Cc: openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Subject: Re: [Openstack] [ironic]Failed to boot ironic instance

Hi Kai,

Thanks for the reply.
I have seem your questions.
And, yes, I think we have met almost the same issue.

I think the dnsmasq process of neutron did not assign an ip to baremetal instance is because neutron-server didn’t allocated any IP.
You can see there is a parameter for  dnsmasq process
--dhcp-hostsfile
dnsmasq process would only response to DHCP request matches MAC address exists in file -dhcp-hostsfile (at least this is how it works when there is no ironic involved).

I checked neutron-server and neutron database, no IP allocation happened.
So, there is no way dnsmasq process would response to any DHCP request.


If another DHCP server can work around this issue, then I guess we might try to solve the  warning in ironic-conductor service log first:
WARNING ironic.common.neutron [-] No VIFs found for node c9caca46-64a7-44d5-b6db-554077b2473a when attempting to update Neutron DHCP BOOT options.

Thanks.
-chen

From: Kai Brennenstuhl [mailto:kai.brennenstuhl at gmail.com]
Sent: Friday, June 27, 2014 2:55 PM
To: Li, Chen
Subject: Re: [Openstack] [ironic]Failed to boot ironic instance

Hi Chen,
I think I had exactly the same problem like you [1]. My problem was that the dnsmasq process of neutron did not assign an ip to the nested baremetal instance. You can check with tcpdump if there are DHCP Boot Requests and if they are replied. I have not solved the issue but did a workaround. I was using Ubuntu 14.04 as controller VM. I configured the dhcp server by myself using isc-dhcp-server [2]. The booting baremetal VM received DHCP offers and everything worked fine. I did not configure PXE or tftp (this was done by ironic).

[1] https://ask.openstack.org/en/question/31253/ironic-node-stuck-in-provisioning-state-wait-call-back/
[2] http://wiki.ubuntuusers.de/ISC-DHCPD

Kai


2014-06-27 8:21 GMT+02:00 Li, Chen <chen.li at intel.com<mailto:chen.li at intel.com>>:
Hi list,

I'm trying to install/using ironic.

I have followed steps here :
http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html#deploying-ironic-with-devstack

While, after I run command "nova boot --flavor baremetal --image cirros-0.3.2-x86_64-uec  test1"

My instance stays in BUILD and spawning for a really long time, and become ERROR at end.

nova list
+--------------------------------------+-------+--------+------------+-------------+----------+
| ID                                   | Name  | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+----------+
| bd10abaa-7cb4-492e-9afd-5027303631f1 | test1  | ERROR  | -          | Running     |          |
| c3e4fccb-dc2b-41dc-8270-2cce5e8fe404 | test2 | BUILD  | spawning   | NOSTATE     |          |
| 3dcfabba-ca32-44ad-988a-c22f7308ed7a | test3 | BUILD  | spawning   | NOSTATE     |          |
+--------------------------------------+-------+--------+------------+-------------+----------+

This is my log for compute.log:
http://paste.openstack.org/
Looks like it is waiting ironic node to become ACTIVE.

This is my ironic-conductor service log:
http://paste.openstack.org/show/85043/

No obvious error message can be find, but there is a warning:
WARNING ironic.common.neutron [-] No VIFs found for node c9caca46-64a7-44d5-b6db-554077b2473a when attempting to update Neutron DHCP BOOT options.

Is this warning caused instance can't start ??

I also check neutron-server, no IP has been allocated at all.

Anyone know why this is happened ???

Thanks.
-chen

Other commands output:
ironic node-list
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| uuid                                 | instance_uuid                        | power_state | provision_state | maintenance |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+
| c9caca46-64a7-44d5-b6db-554077b2473a | 3dcfabba-ca32-44ad-988a-c22f7308ed7a | power on    | wait call-back  | False       |
| 6d6c29e0-b81d-4ed9-9e2a-675b5221aba9 | bd10abaa-7cb4-492e-9afd-5027303631f1 | power on    | wait call-back  | False       |
| 9604af4b-a61a-4157-8431-e61235de7b92 | c3e4fccb-dc2b-41dc-8270-2cce5e8fe404 | power on    | wait call-back  | False       |
+--------------------------------------+--------------------------------------+-------------+-----------------+-------------+

ironic node-show c9caca46-64a7-44d5-b6db-554077b2473a

+------------------------+--------------------------------------------------------------------------+
| Property               | Value                                                                    |
+------------------------+--------------------------------------------------------------------------+
| instance_uuid          | 3dcfabba-ca32-44ad-988a-c22f7308ed7a                                     |
| target_power_state     | None                                                                     |
| properties             | {u'memory_mb': u'512', u'cpu_arch': u'x86_64', u'local_gb': u'10',       |
|                        | u'cpus': u'1'}                                                           |
| maintenance            | False                                                                    |
| driver_info            | {u'pxe_deploy_ramdisk': u'cfdbb01b-fa26-4583-a29f-f2f0111499a5',         |
|                        | u'ssh_port': u'22', u'ssh_username': u'stack', u'pxe_deploy_kernel':     |
|                        | u'c5920355-e4ea-4c09-b774-98b85474e24f', u'ssh_virt_type': u'virsh',     |
|                        | u'ssh_address': u'192.168.50.1', u'ssh_key_filename':                    |
|                        | u'/opt/stack/data/ironic/ssh_keys/ironic_key'}                           |
| extra                  | {}                                                                       |
| last_error             | None                                                                     |
| created_at             | 2014-06-27T05:47:27+00:00                                                |
| target_provision_state | deploy complete                                                          |
| driver                 | pxe_ssh                                                                  |
| updated_at             | 2014-06-27T06:03:20+00:00                                                |
| instance_info          | {u'ramdisk': u'9ea0ac76-a07c-44db-a9b7-328839d0d358', u'kernel':         |
|                        | u'cecfc044-6fd5-49af-b91c-8ac90b4e7976', u'root_gb': u'10',              |
|                        | u'image_source': u'94ba7f9a-4de0-4b00-b07a-85c5647b11de', u'deploy_key': |
|                        | u'8GV962GHNSIDQYX30N0SVB94MCSAG4GE', u'swap_mb': u'0'}                   |
| chassis_uuid           | 3183c4f1-b7d6-49a7-8214-1c850d468a7e                                     |
| provision_state        | wait call-back                                                           |
| reservation            | None                                                                     |
| power_state            | power on                                                                 |
| console_enabled        | False                                                                    |
| uuid                   | c9caca46-64a7-44d5-b6db-554077b2473a                                     |
+------------------------+--------------------------------------------------------------------------+


nova image-list
+--------------------------------------+---------------------------------+--------+--------+
| ID                                   | Name                            | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| 94ba7f9a-4de0-4b00-b07a-85c5647b11de | cirros-0.3.2-x86_64-uec         | ACTIVE |        |
| cecfc044-6fd5-49af-b91c-8ac90b4e7976 | cirros-0.3.2-x86_64-uec-kernel  | ACTIVE |        |
| 9ea0ac76-a07c-44db-a9b7-328839d0d358 | cirros-0.3.2-x86_64-uec-ramdisk | ACTIVE |        |
| cfdbb01b-fa26-4583-a29f-f2f0111499a5 | ir-deploy.initramfs             | ACTIVE |        |
| c5920355-e4ea-4c09-b774-98b85474e24f | ir-deploy.kernel                | ACTIVE |        |
+--------------------------------------+---------------------------------+--------+--------+

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140630/6324a2e7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 27623 bytes
Desc: image001.png
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140630/6324a2e7/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 22893 bytes
Desc: image002.png
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140630/6324a2e7/attachment-0001.png>


More information about the Openstack mailing list