<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Andale Mono";
panose-1:2 11 5 9 0 0 0 0 0 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:DengXian;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:DengXian;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0cm;
background:black;
font-size:9.0pt;
font-family:"Andale Mono";
color:#2FFF12;}
span.s2
{mso-style-name:s2;
color:#2FB41D;}
span.s1
{mso-style-name:s1;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:DengXian;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="ZH-CN" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Buddies,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Have you ever met such issue that the nova-compute service looks hanging there and not able to boot instance?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">When creating an instance, we can only see logs like below and there is NOT any information.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span lang="EN-US">2021-04-23 02:39:00.252 1 DEBUG nova.compute.manager [req-a2ed90b6-792f-48e1-ba7f-e1e35d9537c2 373cb863547407cf3b99034b3b66395e76c137b40f905e7a61e25b1f97df4f3e 1ee7955a2eaf4c86bcc3e650f2a8e2a7 - fc86abb50a684911a30f7955d386a3ea
fc86abb50a684911a30f7955d386a3ea] [instance: e0e77edc-99b4-473e-b318-8e6f04428cda] Starting instance... _do_build_and_run_instance /usr/lib/python3.7/site-packages/nova/compute/manager.py:2202</span></span><span class="s2"><span lang="EN-US">^[</span></span><span class="s1"><span lang="EN-US">[00m</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span lang="EN-US">2021-04-23 03:03:04.927 1 DEBUG oslo_concurrency.lockutils [req-50659582-ad9b-4c17-bbb2-254d5e1141f9 373cb863547407cf3b99034b3b66395e76c137b40f905e7a61e25b1f97df4f3e 1ee7955a2eaf4c86bcc3e650f2a8e2a7 - fc86abb50a684911a30f7955d386a3ea
fc86abb50a684911a30f7955d386a3ea] Lock "a3619b50-3704-4f3e-b908-d525a41756eb" acquired by "nova.compute.manager.ComputeManager.build_and_run_instance.<locals>._locked_do_build_and_run_instance" :: waited 0.001s inner /usr/lib/python3.7/site-packages/oslo_concurrency/lockutils.py:327</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">There is NO other messages, including the periodically tasks.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If I sent a request to create another instance, I can only see another _do_build_and_run_instance log.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span lang="EN-US"><o:p> </o:p></span></span></p>
<p class="p1"><span class="s1"><span lang="EN-US">2021-04-23 03:03:05.061 1 DEBUG nova.compute.manager [req-50659582-ad9b-4c17-bbb2-254d5e1141f9 373cb863547407cf3b99034b3b66395e76c137b40f905e7a61e25b1f97df4f3e 1ee7955a2eaf4c86bcc3e650f2a8e2a7 - fc86abb50a684911a30f7955d386a3ea
fc86abb50a684911a30f7955d386a3ea] [instance: a3619b50-3704-4f3e-b908-d525a41756eb] Starting instance... _do_build_and_run_instance /usr/lib/python3.7/site-packages/nova/compute/manager.py:2202^[[00m<o:p></o:p></span></span></p>
<p class="p1"><span class="s1"><span lang="EN-US">2021-04-23 03:32:44.718 1 DEBUG oslo_concurrency.lockutils [req-c7857cb3-02ae-4c7b-92d7-b2ec178c1b13 373cb863547407cf3b99034b3b66395e76c137b40f905e7a61e25b1f97df4f3e 1ee7955a2eaf4c86bcc3e650f2a8e2a7 - fc86abb50a684911a30f7955d386a3ea
fc86abb50a684911a30f7955d386a3ea] Lock "3b07a6eb-12bb-4d19-9302-25ea3e746944" acquired by "nova.compute.manager.ComputeManager.build_and_run_instance.<locals>._locked_do_build_and_run_instance" :: waited 0.001s inner /usr/lib/python3.7/site-packages/oslo_concurrency/lockutils.py:327<o:p></o:p></span></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I am sure the compute service is still running as I can see the heart-beat time changes correctly.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">From the rabbitmq message, It looks the code goes here<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:7.5pt">nova/compute/manager.py<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:7.5pt"><o:p> </o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;background:white"><b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:navy">def
</span></b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">_build_and_run_instance(</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:#94558D">self</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">,
context, instance, image, injected_files,<br>
admin_password, requested_networks, security_groups,<br>
block_device_mapping, node, limits, filter_properties,<br>
request_spec=</span><b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:navy">None</span></b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">):<br>
<br>
image_name = image.get(</span><b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:teal">'name'</span></b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">)<br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:#94558D">self</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">._notify_about_instance_usage(context, instance,
</span><b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:teal">'create.start'</span></b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">,<br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:#660099">extra_usage_info</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">={</span><b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:teal">'image_name'</span></b><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">:
image_name})<br>
compute_utils.notify_about_instance_create(<br>
context, instance, </span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:#94558D">self</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">.host,<br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:#660099">phase</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">=fields.NotificationPhase.START,<br>
</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:#660099">bdms</span><span lang="EN-US" style="font-size:7.5pt;font-family:Menlo;color:black">=block_device_mapping)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">as I can see a message with "event_type": "compute.instance.create.start"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span lang="EN-US">INFO:root:Body: {'oslo.version': '2.0', 'oslo.message': '{"message_id": "0ef11509-7a65-46b8-ac7e-ed6482fc527d", "publisher_id": "compute.compute01", "event_type": "compute.instance.create.start", "priority":
"INFO", "payload": {"tenant_id": "1ee7955a2eaf4c86bcc3e650f2a8e2a7", "user_id": "373cb863547407cf3b99034b3b66395e76c137b40f905e7a61e25b1f97df4f3e", "instance_id": "3b07a6eb-12bb-4d19-9302-25ea3e746944", "display_name": "yingji-06", "reservation_id": "r-f8iphdcf",
"hostname": "yingji-06", "instance_type": "m1.tiny", "instance_type_id": 15, "instance_flavor_id": "4d5644e1-c561-4b6c-952c-f4dd93c87948", "architecture": null, "memory_mb": 512, "disk_gb": 1, "vcpus": 1, "root_gb": 1, "ephemeral_gb": 0, "host": null, "node":
null, "availability_zone": "nova", "cell_name": "", "created_at": "2021-04-23 03:32:39+00:00", "terminated_at": "", "deleted_at": "", "launched_at": "", "image_ref_url": "https://192.168.111.160:9292//images/303a325f-048</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This issue is not always reproducible and restarting the compute service can work around this.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Could you please give any suggestion on how to resolve this issue or how I can investigate ?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Yingji.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>