<html><head></head><body><div class="yahoo-style-wrap" style="font-family:times new roman, new york, times, serif;font-size:16px;"><div dir="ltr" data-setdir="false">Hi folks,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">We have a 300 strong ironic cluster which has been relatively stable for the most part.</div><div dir="ltr" data-setdir="false">Occasionally we get the IPA timeout errors.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div dir="ltr" data-setdir="false"><table class="ydpc887262bkbn-table ydpc887262btable" data-test-subj="docTable" style="margin: 0px 0px 20px; padding: 0px; border: 0px; font-stretch: inherit; font-size: 12px; line-height: inherit; font-family: "Open Sans", Helvetica, Arial, sans-serif; vertical-align: baseline; border-collapse: collapse; width: 995px; max-width: 100%; color: rgb(45, 45, 45);"><tbody style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; vertical-align: baseline;"><tr class="ydpc887262bdiscover-table-row" data-test-subj="docTableRow" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; vertical-align: baseline;"><td data-test-subj="docTableExpandToggleColumn" style="margin: 0px; padding: 8px; border-width: 1px 0px 0px; border-top-style: solid; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-top-color: rgb(217, 217, 217); border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: 1.42857; font-family: inherit; vertical-align: top; position: relative;"><button class="ydpc887262bdscTable__openButton" style="font-family: inherit; cursor: pointer; background-image: none; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; border-width: initial; border-style: none; border-color: initial; padding: 0px; margin: 0px; outline: none; font-size: inherit; border-radius: 0px; width: 14px; min-height: 14px;"><span class="ydpc887262bkuiIcon ydpc887262bfa-caret-right" style="margin: 0px; padding: 0px; border: 0px; font-stretch: normal; font-size: inherit; line-height: 1; font-family: FontAwesome; vertical-align: baseline; display: inline-block;"></span></button></td><td class="ydpc887262beui-textNoWrap" width="1%" data-test-subj="docTableField" style="margin: 0px; padding: 8px; border-width: 1px 0px 0px; border-top-style: solid; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-top-color: rgb(217, 217, 217); border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: 1.42857; font-family: inherit; vertical-align: top; position: relative; white-space: nowrap !important;"><span style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; vertical-align: baseline;">2019-10-20 23:33:37.994</span><span class="ydpc887262btable-cell-filter" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; vertical-align: baseline;"><button class="ydpc887262bfa ydpc887262bfa-search-plus ydpc887262bdocTableRowFilterButton" data-column="DateTime" style="font-size: 14px; line-height: 1; font-family: FontAwesome; cursor: pointer; background: none rgb(255, 255, 255); border-width: initial; border-style: none; border-color: initial; padding: 0px 3px; margin: 0px; outline: none; border-radius: 0px;"></button><button class="ydpc887262bfa ydpc887262bfa-search-minus ydpc887262bdocTableRowFilterButton" data-column="DateTime" style="font-size: 14px; line-height: 1; font-family: FontAwesome; cursor: pointer; background: none rgb(255, 255, 255); border-width: initial; border-style: none; border-color: initial; padding: 0px 3px; margin: 0px; outline: none; border-radius: 0px;"></button></span></td><td class="ydpc887262bdiscover-table-datafield ydpc887262beui-textBreakAll ydpc887262beui-textBreakWord" data-test-subj="docTableField" style="margin: 0px; padding: 8px; border-width: 1px 0px 0px; border-top-style: solid; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-top-color: rgb(217, 217, 217); border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: 1.42857; font-family: inherit; vertical-align: top; white-space: pre-wrap; position: relative;"><div class="ydpc887262btruncate-by-height" style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; vertical-align: baseline; display: inline-block; min-height: 115px;"><span style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; vertical-align: baseline;">2019-10-20 23:33:37.421 265272 ERROR nova.virt.ironic.driver [req-71933c39-2ac4-497b-8cdc-9578889f42fe cda0725e2a68445db857f56c49f77faf 53a4a4738d9f49e99fe580c1bd51f950 - default default] Error deploying instance c51368f6-b9d8-4c3c-bd63-d1f84ee26bfb on baremetal node 83dc67d4-36be-4004-8847-afd9b969de0e.: InstanceDeployFailure: Failed to provision instance c51368f6-b9d8-4c3c-bd63-d1f84ee26bfb: Timeout reached while waiting for callback for node 83dc67d4-36be-4004-8847-afd9b969de0e</span></div></td></tr></tbody></table></div>Here's my troubleshooting so far.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">DHCP: All good.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><div>Oct 22 09:49:58 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPDISCOVER(ns-601d0738-39) 6c:b3:11:4f:87:ac</div><div>Oct 22 09:49:58 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPOFFER(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:ac</div><div>Oct 22 09:50:02 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPREQUEST(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:ac</div><div>Oct 22 09:50:02 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPACK(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:ac host-10-33-23-94</div><div>Oct 22 10:19:47 sc-control03 dnsmasq-dhcp[965699]: 0 DHCPRELEASE(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:ac</div><div>Oct 22 10:21:23 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPDISCOVER(ns-601d0738-39) 6c:b3:11:4f:87:ac</div><div>Oct 22 10:21:23 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPOFFER(ns-601d0738-39) 10.33.23.187 6c:b3:11:4f:87:ac</div><div>Oct 22 10:21:26 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPREQUEST(ns-601d0738-39) 10.33.23.187 6c:b3:11:4f:87:ac</div><div>Oct 22 10:21:26 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPACK(ns-601d0738-39) 10.33.23.187 6c:b3:11:4f:87:ac host-10-33-23-</div><div>Oct 22 10:22:46 sc-control03 dnsmasq-dhcp[965699]: 3549042952 DHCPDISCOVER(ns-601d0738-39) 6c:b3:11:4f:87:ac</div></div><br></div><div dir="ltr" data-setdir="false">Ironic conductor logs: (shows ironic state proceeding to Power ON, ironic state: deploying. Then 30 minutes later timeout since IPA failed to check-in</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><div>2019-10-22 09:49:03.746 267066 DEBUG ironic.drivers.modules.deploy_utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Deploy boot mode is uefi for d465fa17-a2ea-43ff-a3a7-033cbb497774. get_boot_mode_for_deploy /usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py:798</div><div><br></div><div>2019-10-22 09:49:03.747 267066 DEBUG ironic.common.pxe_utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Building PXE config for node d465fa17-a2ea-43ff-a3a7-033cbb497774 create_pxe_config /usr/lib/python2.7/site-packages/ironic/common/pxe_utils.py:222</div><div><br></div><div>2019-10-22 09:49:03.748 267066 DEBUG ironic.drivers.modules.deploy_utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Deploy boot mode is uefi for d465fa17-a2ea-43ff-a3a7-033cbb497774. get_boot_mode_for_deploy /usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py:798</div><div><br></div><div>2019-10-22 09:49:04.516 267066 DEBUG ironic.drivers.modules.pxe [req-fabea5a4-da6c-48be-834a-49763c23b0e9 <br></div><div>1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Fetching necessary kernel and ramdisk for node d465fa17-a2ea-43ff-a3a7-033cbb497774 _cache_ramdisk_kernel /usr/lib/python2.7/site-packages/ironic/drivers/modules/pxe.py:382</div><div><br></div><div>2019-10-22 09:49:17.761 267066 INFO ironic.conductor.utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Successfully set node d465fa17-a2ea-43ff-a3a7-033cbb497774 power state to power on by rebooting.</div><div><br></div><div dir="ltr" data-setdir="false">===== 30 minutes later timeout. IPA failed to check-in</div><div dir="ltr" data-setdir="false"><br></div><div>2019-10-22 10:19:37.448 267066 INFO ironic.conductor.task_manager [req-762d8acf-0a55-4d8d-99d8-93020a14547c - - - - -] Node d465fa17-a2ea-43ff-a3a7-033cbb497774 moved to provision state "deploy failed" from state "wait call-back"; target provision state is "active"</div><div><br></div><div>2019-10-22 10:19:37.471 267066 DEBUG ironic.common.pxe_utils [req-762d8acf-0a55-4d8d-99d8-93020a14547c - - - - -] Cleaning up PXE config for node d465fa17-a2ea-43ff-a3a7-033cbb497774 clean_up_pxe_config /usr/lib/python2.7/site-packages/ironic/common/pxe_utils.py:288</div></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">TFTP logs: shows TFTP client timed out (weird). Any pointers here?</div><div dir="ltr" data-setdir="false">tftpd shows ramdisk_deployed completed. Then, it reports that the client timed out.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318194]: RRQ from 10.33.23.94 filename d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk remapped to /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk</div><div><br></div><div>Oct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 finished d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk</div><div>Oct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 timed out</div><div><br></div><div><br></div><div><br></div><div>Oct 22 09:50:03 sc-ironic04 in.tftpd[318148]: remap: input: bootx64.efi</div><div>Oct 22 09:50:03 sc-ironic04 in.tftpd[318148]: remap: rule 3: rewrite: /tftpboot/bootx64.efi</div><div>Oct 22 09:50:03 sc-ironic04 in.tftpd[318148]: remap: rule 3: exit</div><div>Oct 22 09:50:03 sc-ironic04 in.tftpd[318148]: RRQ from 10.33.23.94 filename bootx64.efi remapped to /tftpboot/bootx64.efi</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318148]: Error code 8: User aborted the transfer</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318150]: remap: input: bootx64.efi</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318150]: remap: rule 3: rewrite: /tftpboot/bootx64.efi</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318150]: remap: rule 3: exit</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318150]: RRQ from 10.33.23.94 filename bootx64.efi remapped to /tftpboot/bootx64.efi</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318150]: Client 10.33.23.94 finished bootx64.efi</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318151]: remap: input: grubx64.efi</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318151]: remap: rule 3: rewrite: /tftpboot/grubx64.efi</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318151]: remap: rule 3: exit</div><div>Oct 22 09:50:04 sc-ironic04 in.tftpd[318151]: RRQ from 10.33.23.94 filename grubx64.efi remapped to /tftpboot/grubx64.efi</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318151]: Client 10.33.23.94 finished grubx64.efi</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318152]: remap: input: /grub.cfg-01-6c-b3-11-4f-87-ac</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318152]: remap: rule 2: rewrite: /tftpboot//grub.cfg-01-6c-b3-11-4f-87-ac</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318152]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318152]: RRQ from 10.33.23.94 filename /grub.cfg-01-6c-b3-11-4f-87-ac remapped to /tftpboot//grub.cfg-01-6c-b3-11-4f-87-ac</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318152]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-01-6c-b3-11-4f-87-ac</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318152]: sending NAK (1, File not found) to 10.33.23.94</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318153]: remap: input: /grub.cfg-0A21175E</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318153]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A21175E</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318153]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318153]: RRQ from 10.33.23.94 filename /grub.cfg-0A21175E remapped to /tftpboot//grub.cfg-0A21175E</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318153]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A21175E</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318153]: sending NAK (1, File not found) to 10.33.23.94</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: remap: input: /grub.cfg-0A21175</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A21175</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: RRQ from 10.33.23.94 filename /grub.cfg-0A21175 remapped to /tftpboot//grub.cfg-0A21175</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A21175</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: sending NAK (1, File not found) to 10.33.23.94</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: remap: input: /grub.cfg-0A2117</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A2117</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: RRQ from 10.33.23.94 filename /grub.cfg-0A2117 remapped to /tftpboot//grub.cfg-0A2117</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A2117</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: sending NAK (1, File not found) to 10.33.23.94</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: remap: input: /grub.cfg-0A211</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A211</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: RRQ from 10.33.23.94 filename /grub.cfg-0A211 remapped to /tftpboot//grub.cfg-0A211</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A211</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: sending NAK (1, File not found) to 10.33.23.94</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: remap: input: /grub.cfg-0A21</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A21</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: RRQ from 10.33.23.94 filename /grub.cfg-0A21 remapped to /tftpboot//grub.cfg-0A21</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A21</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: sending NAK (1, File not found) to 10.33.23.94</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318158]: remap: input: /grub.cfg-0A2</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318158]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A2</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318158]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318178]: RRQ from 10.33.23.94 filename /EFI/centos/grub.cfg remapped to /tftpboot//EFI/centos/grub.cfg</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318178]: Client 10.33.23.94 finished /EFI/centos/grub.cfg</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318179]: remap: input: /EFI/centos/grub.cfg</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318179]: remap: rule 2: rewrite: /tftpboot//EFI/centos/grub.cfg</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318179]: remap: rule 2: exit</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318179]: RRQ from 10.33.23.94 filename /EFI/centos/grub.cfg remapped to /tftpboot//EFI/centos/grub.cfg</div><div>Oct 22 09:50:05 sc-ironic04 in.tftpd[318179]: Client 10.33.23.94 finished /EFI/centos/grub.cfg</div><div>Oct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.298 267066 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._sync_local_state' _process_scheduled /usr/lib/python2.7/site-packages/futurist/periodics.py:639</div><div>Oct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.298 267066 INFO ironic.conductor.manager [-] Executing sync_local_state</div><div>Oct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.305 267066 DEBUG ironic.common.hash_ring [-] Rebuilding cached hash rings ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:51</div><div>Oct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.466 267066 DEBUG ironic.common.hash_ring [-] Finished rebuilding hash rings, available drivers are ipmi, fake, snmp, fake_pxe, pxe_snmp, agent_ipmitool, pxe_ipmitool ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:56</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318184]: remap: input: /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318184]: remap: rule 0: rewrite: /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318184]: remap: rule 0: exit</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318184]: RRQ from 10.33.23.94 filename /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318184]: Client 10.33.23.94 finished /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318185]: remap: input: /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318185]: remap: rule 0: rewrite: /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318185]: remap: rule 0: exit</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318185]: RRQ from 10.33.23.94 filename /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318185]: Client 10.33.23.94 finished /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318186]: remap: input: /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318186]: remap: rule 0: rewrite: /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318186]: remap: rule 0: exit</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318186]: RRQ from 10.33.23.94 filename /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:10 sc-ironic04 in.tftpd[318186]: Client 10.33.23.94 finished /tftpboot/10.33.23.94.conf</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318192]: remap: input: d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernel</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318192]: remap: rule 3: rewrite: /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernel</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318192]: remap: rule 3: exit</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318192]: RRQ from 10.33.23.94 filename d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernel remapped to /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernel</div><div>Oct 22 09:50:15 sc-ironic04 ironic-conductor: 2019-10-22 09:50:15.898 267066 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_rescuewait_timeouts' _process_scheduled /usr/lib/python2.7/site-packages/futurist/periodics.py:639</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318192]: Client 10.33.23.94 finished d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernel</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318194]: remap: input: d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318194]: remap: rule 3: rewrite: /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318194]: remap: rule 3: exit</div><div>Oct 22 09:50:15 sc-ironic04 in.tftpd[318194]: RRQ from 10.33.23.94 filename d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk remapped to /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk</div><div><br></div><div><br></div><div><br></div><div>Oct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 finished d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk</div><div>Oct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 timed out</div></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">This has me stumped here. This exact failure seems to be happening 3 to 4 times a week on different nodes.</div><div dir="ltr" data-setdir="false">Any pointers appreciated.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">thanks,</div><div dir="ltr" data-setdir="false">Fred.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div></div></body></html>