Open Stack

Hi,

With my experence, for such "2016-04-08 16:31:38.690 31893 ERROR
ironic.dhcp.neutron
[-] Failed to update Neutron port 7fb98457-90e6-43be-a353-f03ea1959912."
issue, some cases are that root cause is we did not register baremetal's
mac into ironic, so neutron can not bind mac for baremetal dhcp/pxe, and
set DHCP BOOT option for port, so run below command if it is missing:

ironic port-create -n $NODE_UUID -a $MAC_ADDRESS

Good luck!


On Fri, Apr 8, 2016 at 10:00 PM, Senthilprabu Shanmugavel <
senthilprabu.s at gmail.com> wrote:

> Hello,
>
> I am deploying an ARMv8 board using Ironic integrated with Openstack
> Liberty on Ubuntu 12.04 host. I am using fake_pxe driver and doing the
> power on manually. Deploy image created from disk image creator tool
>
> Nova boot starts the deployment, deployment image is booted, ironic-agent
> is able to communicate with ironic conductor. After this, scsi connection
> is established, I can see this in syslog in ironic conductor node
>
>
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.620364] scsi
> host10: iSCSI Initiator over TCP/IP
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.885579] scsi
> 10:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.886404] scsi
> 10:0:0:0: Attached scsi generic sg2 type 12
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.887088] scsi
> 10:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.888157] sd
> 10:0:0:1: Attached scsi generic sg3 type 0
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.888490] sd
> 10:0:0:1: [sdc] 15240576 512-byte logical blocks: (7.80 GB/7.26 GiB)
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889114] sd
> 10:0:0:1: [sdc] Write Protect is off
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889118] sd
> 10:0:0:1: [sdc] Mode Sense: 69 00 10 08
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889462] sd
> 10:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and
> FUA
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.894611]  sdc:
> unknown partition table
> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.896728] sd
> 10:0:0:1: [sdc] Attached SCSI disk
> Apr  8 16:31:35 lionfish-ocp-controller iscsid: Connection4:0 to [target:
> iqn.2008-10.org.openstack:5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0, portal:
> 192.168.2.161,3260] through [iface: default] is operational now
>
> On my node console, I see below messages
>
> root at ubuntu:~# SysRq : Emergency Sync
> SysRq : Power Off
> reboot: Power down
>
> I think this is triggered by scsi driver (also seen in ironic conductor
> syslog) but node did not reboot due to known problem. So I powered off
> manually and powered on.
>
> Apr  8 16:31:38 lionfish-ocp-controller kernel: [177776.290404] sd
> 10:0:0:1: [sdc] Synchronizing SCSI cache
> Apr  8 16:31:38 lionfish-ocp-controller iscsid: Connection4:0 to [target:
> iqn.2008-10.org.openstack:5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0, portal:
> 192.168.2.161,3260] through [iface: default] is shutdown.
>
> So far looks good. I was expecting user image download via scsi is
> successful and should be booted after going down.
>
> By then ironic conductor log says deployment failed due to neutron failing
> to set DHCP BOOT option for port.
>
>
> 2016-04-08 16:31:14.229 31893 INFO
> ironic.drivers.modules.agent_base_vendor
> [req-c1e50ceb-1e11-47ea-9260-e30a2f10605d - - - - -] Initial lookup for
> node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 succeeded, agent is running and
> waiting for commands
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron [-] Failed to
> update Neutron port 7fb98457-90e6-43be-a353-f03ea1959912.
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron Traceback (most
> recent call last):
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/ironic/dhcp/neutron.py", line 126, in
> update_port_dhcp_opts
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> _build_client(token).update_port(port_id, port_req_body)
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102,
> in with_params
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     ret =
> self.function(instance, *args, **kwargs)
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 562,
> in update_port
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     return
> self.put(self.port_path % (port), body=body)
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 302,
> in put
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> headers=headers, params=params)
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270,
> in retry_request
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> headers=headers, params=params)
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 200,
> in do_request
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> content_type=self.content_type())
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in
> do_request
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> self.authenticate_and_fetch_endpoint_url()
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in
> authenticate_and_fetch_endpoint_url
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> self.authenticate()
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in
> authenticate
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> self._authenticate_keystone()
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in
> _authenticate_keystone
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     raise
> exceptions.Unauthorized(message=resp_body)
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron Unauthorized:
> {"error": {"message": "The resource could not be found.", "code": 404,
> "title": "Not Found"}}
> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> 2016-04-08 16:31:38.692 31893 ERROR ironic.drivers.modules.iscsi_deploy
> [-] Deploy failed for instance 8be4838b-c7be-411c-abf4-212b42dde17d. Error:
> Failed to set DHCP BOOT options for any port on node
> 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0.
> 2016-04-08 16:31:38.891 31893 INFO ironic.conductor.utils [-] Successfully
> set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to power off.
> 2016-04-08 16:31:49.208 31893 INFO ironic.conductor.utils [-] Successfully
> set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to power on.
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception for
> node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0: Node failed to get image for
> deploy. exception: Can not transition from state 'deploy failed' on event
> 'done' (no defined transition)
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor   File
> "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py",
> line 339, in heartbeat
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor     self.continue_deploy(task,
> **kwargs)
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor   File
> "/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line
> 134, in wrapper
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor     return f(*args, **kwargs)
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor   File
> "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py",
> line 983, in continue_deploy
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor
> self.reboot_and_finish_deploy(task)
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor   File
> "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py",
> line 611, in reboot_and_finish_deploy
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor     task.process_event('done')
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor   File
> "/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line
> 342, in process_event
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor     self.fsm.process_event(event,
> target_state=target_state)
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor   File
> "/usr/lib/python2.7/dist-packages/ironic/common/fsm.py", line 44, in wrapper
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor     raise
> excp.InvalidState(six.text_type(e))
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor InvalidState: Can not transition
> from state 'deploy failed' on event 'done' (no defined transition)
> 2016-04-08 16:31:49.317 31893 ERROR
> ironic.drivers.modules.agent_base_vendor
> 2016-04-08 16:31:49.869 31893 INFO ironic.conductor.utils [-] Successfully
> set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to power off.
> 2016-04-08 16:31:50.012 31893 INFO ironic.conductor.manager [-]
> Successfully unprovisioned node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 with
> instance 8be4838b-c7be-411c-abf4-212b42dde17d.
> 2016-04-08 16:31:50.621 31893 INFO ironic.conductor.manager [-] Cleaning
> is disabled, node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 has been
> successfully moved to AVAILABLE state.
>
>
> Similar issue reported in bug - 1524779
> https://bugs.launchpad.net/ironic/+bug/1524779
>
>
> Can someone comment how can I overcome this scsi error?. Is there any
> workaround.
>
> Thanks!!!
>
> --
> Best Regards,
> Senthilprabu Shanmugavel
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160411/c83a6e13/attachment.html>