[openstack-dev] [Ironic] Failed to update Neutron port

Senthilprabu Shanmugavel senthilprabu.s at gmail.com
Tue Apr 19 11:25:26 UTC 2016


Hello,

Error was due to the reason Neutron client was not authenticating using V2
API.

Eventually it was resolved when I update my auth_uri to add /v2/ tag

auth_uri=http://172.19.0.3:5000/v2.0/

But this was not specified in documentation. Reference :
http://docs.openstack.org/developer/ironic/4.2.1/deploy/install-guide.html.
Even in latest 5.x version documentation.





On Mon, Apr 11, 2016 at 12:38 PM, Sam Betts (sambetts) <sambetts at cisco.com>
wrote:

> Looking at the errors here it appears that everything is configured
> correctly in Ironic (mac addresses etc) because you are getting through the
> ramdisk boot and deploy successfully.
>
> The error that is of real concern here is "Unauthorized: {"error":
> {"message": "The resource could not be found.", "code": 404, "title": "Not
> Found"}}" which seems to imply that something is wrong with the auth
> information given to Ironic and its failing to authenticate with Keystone
> as part of the neutron client. The resource could not be found error is
> thrown when it can’t find the tenant or user requested in the config file.
>
> Sam
>
>
> From: "Haomeng, Wang" <wanghaomeng at gmail.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> Date: Monday, 11 April 2016 02:45
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [Ironic] Failed to update Neutron port
>
> Hi,
>
> With my experence, for such "2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
> [-] Failed to update Neutron port 7fb98457-90e6-43be-a353-f03ea1959912."
> issue, some cases are that root cause is we did not register baremetal's
> mac into ironic, so neutron can not bind mac for baremetal dhcp/pxe, and
> set DHCP BOOT option for port, so run below command if it is missing:
>
> ironic port-create -n $NODE_UUID -a $MAC_ADDRESS
>
> Good luck!
>
>
> On Fri, Apr 8, 2016 at 10:00 PM, Senthilprabu Shanmugavel <
> senthilprabu.s at gmail.com> wrote:
>
>> Hello,
>>
>> I am deploying an ARMv8 board using Ironic integrated with Openstack
>> Liberty on Ubuntu 12.04 host. I am using fake_pxe driver and doing the
>> power on manually. Deploy image created from disk image creator tool
>>
>> Nova boot starts the deployment, deployment image is booted, ironic-agent
>> is able to communicate with ironic conductor. After this, scsi connection
>> is established, I can see this in syslog in ironic conductor node
>>
>>
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.620364] scsi
>> host10: iSCSI Initiator over TCP/IP
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.885579] scsi
>> 10:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.886404] scsi
>> 10:0:0:0: Attached scsi generic sg2 type 12
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.887088] scsi
>> 10:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.888157] sd
>> 10:0:0:1: Attached scsi generic sg3 type 0
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.888490] sd
>> 10:0:0:1: [sdc] 15240576 512-byte logical blocks: (7.80 GB/7.26 GiB)
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889114] sd
>> 10:0:0:1: [sdc] Write Protect is off
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889118] sd
>> 10:0:0:1: [sdc] Mode Sense: 69 00 10 08
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.889462] sd
>> 10:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and
>> FUA
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.894611]  sdc:
>> unknown partition table
>> Apr  8 16:31:35 lionfish-ocp-controller kernel: [177773.896728] sd
>> 10:0:0:1: [sdc] Attached SCSI disk
>> Apr  8 16:31:35 lionfish-ocp-controller iscsid: Connection4:0 to [target:
>> iqn.2008-10.org.openstack:5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0, portal:
>> 192.168.2.161,3260] through [iface: default] is operational now
>>
>> On my node console, I see below messages
>>
>> root at ubuntu:~# SysRq : Emergency Sync
>> SysRq : Power Off
>> reboot: Power down
>>
>> I think this is triggered by scsi driver (also seen in ironic conductor
>> syslog) but node did not reboot due to known problem. So I powered off
>> manually and powered on.
>>
>> Apr  8 16:31:38 lionfish-ocp-controller kernel: [177776.290404] sd
>> 10:0:0:1: [sdc] Synchronizing SCSI cache
>> Apr  8 16:31:38 lionfish-ocp-controller iscsid: Connection4:0 to [target:
>> iqn.2008-10.org.openstack:5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0, portal:
>> 192.168.2.161,3260] through [iface: default] is shutdown.
>>
>> So far looks good. I was expecting user image download via scsi is
>> successful and should be booted after going down.
>>
>> By then ironic conductor log says deployment failed due to neutron
>> failing to set DHCP BOOT option for port.
>>
>>
>> 2016-04-08 16:31:14.229 31893 INFO
>> ironic.drivers.modules.agent_base_vendor
>> [req-c1e50ceb-1e11-47ea-9260-e30a2f10605d - - - - -] Initial lookup for
>> node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 succeeded, agent is running and
>> waiting for commands
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron [-] Failed to
>> update Neutron port 7fb98457-90e6-43be-a353-f03ea1959912.
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron Traceback (most
>> recent call last):
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/ironic/dhcp/neutron.py", line 126, in
>> update_port_dhcp_opts
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> _build_client(token).update_port(port_id, port_req_body)
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102,
>> in with_params
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     ret =
>> self.function(instance, *args, **kwargs)
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 562,
>> in update_port
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     return
>> self.put(self.port_path % (port), body=body)
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 302,
>> in put
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> headers=headers, params=params)
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270,
>> in retry_request
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> headers=headers, params=params)
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 200,
>> in do_request
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> content_type=self.content_type())
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in
>> do_request
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> self.authenticate_and_fetch_endpoint_url()
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in
>> authenticate_and_fetch_endpoint_url
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> self.authenticate()
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in
>> authenticate
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> self._authenticate_keystone()
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron   File
>> "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in
>> _authenticate_keystone
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron     raise
>> exceptions.Unauthorized(message=resp_body)
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron Unauthorized:
>> {"error": {"message": "The resource could not be found.", "code": 404,
>> "title": "Not Found"}}
>> 2016-04-08 16:31:38.690 31893 ERROR ironic.dhcp.neutron
>> 2016-04-08 16:31:38.692 31893 ERROR ironic.drivers.modules.iscsi_deploy
>> [-] Deploy failed for instance 8be4838b-c7be-411c-abf4-212b42dde17d. Error:
>> Failed to set DHCP BOOT options for any port on node
>> 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0.
>> 2016-04-08 16:31:38.891 31893 INFO ironic.conductor.utils [-]
>> Successfully set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to
>> power off.
>> 2016-04-08 16:31:49.208 31893 INFO ironic.conductor.utils [-]
>> Successfully set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to
>> power on.
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception for
>> node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0: Node failed to get image for
>> deploy. exception: Can not transition from state 'deploy failed' on event
>> 'done' (no defined transition)
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor   File
>> "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py",
>> line 339, in heartbeat
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor     self.continue_deploy(task,
>> **kwargs)
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor   File
>> "/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line
>> 134, in wrapper
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor     return f(*args, **kwargs)
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor   File
>> "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py",
>> line 983, in continue_deploy
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor
>> self.reboot_and_finish_deploy(task)
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor   File
>> "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py",
>> line 611, in reboot_and_finish_deploy
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor     task.process_event('done')
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor   File
>> "/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line
>> 342, in process_event
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor     self.fsm.process_event(event,
>> target_state=target_state)
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor   File
>> "/usr/lib/python2.7/dist-packages/ironic/common/fsm.py", line 44, in wrapper
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor     raise
>> excp.InvalidState(six.text_type(e))
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor InvalidState: Can not transition
>> from state 'deploy failed' on event 'done' (no defined transition)
>> 2016-04-08 16:31:49.317 31893 ERROR
>> ironic.drivers.modules.agent_base_vendor
>> 2016-04-08 16:31:49.869 31893 INFO ironic.conductor.utils [-]
>> Successfully set node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 power state to
>> power off.
>> 2016-04-08 16:31:50.012 31893 INFO ironic.conductor.manager [-]
>> Successfully unprovisioned node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 with
>> instance 8be4838b-c7be-411c-abf4-212b42dde17d.
>> 2016-04-08 16:31:50.621 31893 INFO ironic.conductor.manager [-] Cleaning
>> is disabled, node 5044c5ab-a4f5-4ba1-9e3d-01a0cc9bb9e0 has been
>> successfully moved to AVAILABLE state.
>>
>>
>> Similar issue reported in bug - 1524779
>> https://bugs.launchpad.net/ironic/+bug/1524779
>>
>>
>> Can someone comment how can I overcome this scsi error?. Is there any
>> workaround.
>>
>> Thanks!!!
>>
>> --
>> Best Regards,
>> Senthilprabu Shanmugavel
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Best Regards,
Senthilprabu Shanmugavel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160419/8e5b261d/attachment.html>


More information about the OpenStack-dev mailing list