<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:\65B0\7D30\660E\9AD4;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@\65B0\7D30\660E\9AD4";
panose-1:2 2 5 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ZH-TW link=blue vlink=purple style='text-justify-trim:punctuation'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:ZH-HK'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:ZH-HK'><o:p> </o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>I recently deploy Bare-metal node instance using Heat Template. However, </span><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B;mso-fareast-language:ZH-HK'>Nova failed to spawn due to</span><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'> a timeout error. When I look into the code I found that the timeout is related to Nova downloading disk image from Glance. The nova-schedule.log shows below:<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 ERROR nova.compute.manager [req-09e61b23-436f-4425-8db0-10dd1aea2e39 85bbc1abb4254761a5452654a6934b75 692e595702654930936a65d1a658cff4] [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Instance failed to spawn<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Traceback (most recent call last):<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn/ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] network_info=network_info,<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 444, in cache_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] self._cache_tftp_images(context, instance, tftp_image_info)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 335, in _cache_tftp_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] project_id=instance['project_id'],<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/utils.py", line 33, in cache_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] user_id, project_id)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 645, in fetch_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 196, in fetch_to_raw 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] image_service.download(context, image_id, dst_path=path)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 360, in download 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] for chunk in image_chunks:<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 478, in __iter__ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self.next()<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 494, in next 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self._resp.read(CHUNKSIZE)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/httplib.py", line 561, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] s = self.fp.read(amt)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/socket.py", line 380, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] data = self._sock.recv(left)<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 262, in recv 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout_exc=socket.timeout("timed out"))<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 155, in trampoline 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return hub.switch()<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return self.greenlet.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout: timed out<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>I don't know where the timeout is set. And does Nova download image from Glance in parallel or in sequential way?<o:p></o:p></span></p><p style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:10.5pt;margin-left:0cm;background:white;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B'>Thank you very much</span><span lang=EN-US style='font-size:10.5pt;font-family:"Arial","sans-serif";color:#4B4B4B;mso-fareast-language:ZH-HK'>.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:ZH-HK'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:ZH-HK'>Nora<o:p></o:p></span></p></div></body></html>