Open Stack

Hello,

I am deploying an ARMv8 board using Ironic integrated with Openstack
Liberty on Ubuntu 12.04 host. I am using fake_pxe driver and doing the
power on manually. Deploy image created from disk image creator tool

Nova boot starts the deployment, deployment image is booted, ironic-agent
is able to communicate with ironic conductor. After this, scsi connection
is established, I can see this in syslog in ironic conductor node


Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.620364] scsi
host10: iSCSI Initiator over TCP/IP
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.885579] scsi
10:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.886404] scsi
10:0:0:0: Attached scsi generic sg2 type 12
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.887088] scsi
10:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.888157] sd
10:0:0:1: Attached scsi generic sg3 type 0
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.888490] sd
10:0:0:1: [sdc] 15240576 512-byte logical blocks: (7.80 GB/7.26 GiB)
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889114] sd
10:0:0:1: [sdc] Write Protect is off
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889118] sd
10:0:0:1: [sdc] Mode Sense: 69 00 10 08
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889462] sd
10:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and
FUA
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.894611]  sdc:
unknown partition table
Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.896728] sd
10:0:0:1: [sdc] Attached SCSI disk
Apr  8 16:31:35 lionfish-ocp-controller iscsid: Connection4:0 to [target:
iqn.2008-10.org.openstack:5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0, portal:
192.168.2.161,3260] through [iface: default] is operational now

On my node console, I see below messages

root at ubuntu:~# SysRq : Emergency Sync
SysRq : Power Off
reboot: Power down

I think this is triggered by scsi driver (also seen in ironic conductor
syslog) but node did not reboot due to known problem. So I powered off
manually and powered on.

Apr  8 16:31:38 lionfish-ocp-controller kernel: [177776.290404] sd
10:0:0:1: [sdc] Synchronizing SCSI cache
Apr  8 16:31:38 lionfish-ocp-controller iscsid: Connection4:0 to [target:
iqn.2008-10.org.openstack:5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0, portal:
192.168.2.161,3260] through [iface: default] is shutdown.

So far looks good. I was expecting user image download via scsi is
successful and should be booted after going down.

By then ironic conductor log says deployment failed due to neutron failing
to set DHCP BOOT option for port.


2016-04-08 16:31:14.229 31893 INFO ironic.drivers.modules.agent_base_vendor
[req-c1e50ceb-1e11-47ea-9260-e30a2f10605d - - - - -] Initial lookup for
node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 succeeded, agent is running and
waiting for commands
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron [-] Failed to
update Neutron port 7fb98457-90e6-43be-a353-f03ea1959912.
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron Traceback (most
recent call last):
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/ironic/dhcp/neutron.py", line 126, in
update_port_dhcp_opts
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
_build_client(token).update_port(port_id, port_req_body)
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102,
in with_params
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     ret =
self.function(instance, *args, **kwargs)
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 562,
in update_port
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     return
self.put(self.port_path % (port), body=body)
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 302,
in put
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
headers=headers, params=params)
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270,
in retry_request
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
headers=headers, params=params)
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 200,
in do_request
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
content_type=self.content_type())
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in
do_request
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
self.authenticate_and_fetch_endpoint_url()
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in
authenticate_and_fetch_endpoint_url
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
self.authenticate()
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in
authenticate
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
self._authenticate_keystone()
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in
_authenticate_keystone
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     raise
exceptions.Unauthorized(message=resp_body)
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron Unauthorized:
{"error": {"message": "The resource could not be found.", "code": 404,
"title": "Not Found"}}
2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
2016-04-08 16:31:38.692 31893 ERROR ironic.drivers.modules.iscsi_deploy [-]
Deploy failed for instance 8be4838b-c7be-411c-abf4-212b42dde17d. Error:
Failed to set DHCP BOOT options for any port on node
5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0.
2016-04-08 16:31:38.891 31893 INFO ironic.conductor.utils [-] Successfully
set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to power off.
2016-04-08 16:31:49.208 31893 INFO ironic.conductor.utils [-] Successfully
set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to power on.
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception for
node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0: Node failed to get image for
deploy. exception: Can not transition from state 'deploy failed' on event
'done' (no defined transition)
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor   File
"/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py",
line 339, in heartbeat
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor     self.continue_deploy(task,
**kwargs)
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor   File
"/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line
134, in wrapper
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor     return f(*args, **kwargs)
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor   File
"/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py",
line 983, in continue_deploy
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor
self.reboot_and_finish_deploy(task)
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor   File
"/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py",
line 611, in reboot_and_finish_deploy
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor     task.process_event('done')
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor   File
"/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line
342, in process_event
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor     self.fsm.process_event(event,
target_state=target_state)
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor   File
"/usr/lib/python2.7/dist-packages/ironic/common/fsm.py", line 44, in wrapper
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor     raise
excp.InvalidState(six.text_type(e))
2016-04-08 16:31:49.317 31893 ERROR
ironic.drivers.modules.agent_base_vendor InvalidState: Can not transition
from state 'deploy failed' on event 'done' (no defined transition)
2016-04-08 16:31:49.317 31893 ERROR ironic.drivers.modules.agent_base_vendor
2016-04-08 16:31:49.869 31893 INFO ironic.conductor.utils [-] Successfully
set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to power off.
2016-04-08 16:31:50.012 31893 INFO ironic.conductor.manager [-]
Successfully unprovisioned node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 with
instance 8be4838b-c7be-411c-abf4-212b42dde17d.
2016-04-08 16:31:50.621 31893 INFO ironic.conductor.manager [-] Cleaning is
disabled, node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 has been successfully
moved to AVAILABLE state.


Similar issue reported in bug - 1524779
https://bugs.launchpad.net/ironic/+bug/1524779


Can someone comment how can I overcome this scsi error?. Is there any
workaround.

Thanks!!!

-- 
Best Regards,
Senthilprabu Shanmugavel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160408/3c793937/attachment.html>