<div dir="ltr">Yes, cinder is running.<br><div><br></div><div><font face="courier new, monospace">root@h017:~$ service --status-all | grep cinder<br></font></div><div><font face="courier new, monospace">[ + ] cinder-volume</font><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 12, 2019 at 11:53 AM Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com">ignaziocassano@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Sorry ...the question was : how many compute nodes do you have ?</div><div>instead of how many compute nodes do gli have...</div><div><br></div><div><br></div><div>Anycase;</div><div>Did you configured cinder ?<br> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 12 lug 2019 alle ore 11:26 Jay See <<a href="mailto:jayachander.it@gmail.com" target="_blank">jayachander.it@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Ignazio, <br><div><br></div><div>One instance is stuck in error state not able to recover it. All other instances are running now.</div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">root@h004:~$ nova reset-state --all-tenants my-instance-1-2<br></font></div><div><font face="courier new, monospace">Reset state for server my-instance-1-2 succeeded; new state is error</font><br></div><div><br></div><div>I have several compute nodes (14). I am not sure what is gli?</div><div>Live migration is not working, i have tried it was not throwing any errors. But nothing seems to happen.</div><div>I am not completely sure, I haven't heard about gli before. (This setup is deployed by someone else).</div></div><br><div class="gmail_quote"></div><div>~Jay.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 12, 2019 at 6:12 AM Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" target="_blank">ignaziocassano@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Jay, for recovering vm state use the command nova reset-state....<div dir="auto"><br></div><div dir="auto">nova help reset-state to check the command requested parameters.</div><div dir="auto"><br></div><div dir="auto">Ad far as evacuation la concerned, how many compute nodes do gli have ? </div><div dir="auto">Instance live migration works? </div><div dir="auto">Are gli using shared cinder storage?</div><div dir="auto">Ignazio </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il Gio 11 Lug 2019 20:51 Jay See <<a href="mailto:jayachander.it@gmail.com" target="_blank">jayachander.it@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks for explanation Ignazio.<div><br></div><div>I have tried same same by trying to put the compute node on a failure (echo 'c' > /proc/sysrq-trigger ). Compute node was stuck and I was not able connect to it.</div><div>All the VMs are now in Error state.</div><div><br></div><div>Running the host-evacaute was successful on controller node, but now I am not able to use the VMs. Because they are all in error state now.</div><div><br></div><div><font face="courier new, monospace">root@h004:~$ nova host-evacuate h017<br>+--------------------------------------+-------------------+---------------+<br>| Server UUID | Evacuate Accepted | Error Message |<br>+--------------------------------------+-------------------+---------------+<br>| f3545f7d-b85e-49ee-b407-333a4c5b5ab9 | True | |<br>| 9094494b-cfa3-459b-8d51-d9aae0ea9636 | True | |<br>| abe7075b-ac22-4168-bf3d-d302ba37d80e | True | |<br>| c9919371-5f2e-4155-a01a-5f41d9c8b0e7 | True | |<br>| ffd983bb-851e-4314-9d1d-375303c278f3 | True | |<br>+--------------------------------------+-------------------+---------------+</font><br></div><div><font face="courier new, monospace"><br></font></div><div><font face="arial, sans-serif">Now I have restarted the compute node manually , now I am able to connect to the compute node but VMs are still in Error state. </font></div><div><font face="arial, sans-serif">1. Any ideas, how to recover the VMs?</font></div><div><font face="arial, sans-serif">2. Are there any other methods to evacuate, as this method seems to be not working in mitaka version.</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">~Jay.</font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 11, 2019 at 1:33 PM Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" rel="noreferrer" target="_blank">ignaziocassano@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Ok Jay,</div><div>let me to describe my environment.</div><div>I have an openstack made up of 3 controllers nodes ad several compute nodes.</div><div>The controller nodes services are controlled by pacemaker and the compute nodes services are controlled by remote pacemaker.</div><div>My hardware is Dell so I am using ipmi fencing device .</div><div>I wrote a service controlled by pacemaker: <br></div><div>this service controls if a compude node fails and for avoiding split brains if a compute node does nod respond on the management network and on storage network the stonith poweroff the node and then execute a nova host-evacuate.</div><div><br></div><div>Anycase to have a simulation before writing the service I described above you can do as follows:</div><div><br></div><div>connect on one compute node where some virtual machines are running</div><div>run the command: echo 'c' > /proc/sysrq-trigger (it stops immediately the node like in case of failure)</div><div>On a controller node run: nova host-evacuate "name of failed compute node"</div><div>Instances running on the failed compute node should be restarted on another compute node<br></div><div><br></div><div><br></div><div>Ignazio<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 11 lug 2019 alle ore 11:57 Jay See <<a href="mailto:jayachander.it@gmail.com" rel="noreferrer" target="_blank">jayachander.it@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi ,<div><br><div>I have tried on a failed compute node which is in power off state now.<div>I have tried on a running compute node, no errors. But nothing happens.</div><div>On running compute node - Disabled the compute service and tried migration also.</div><div><br></div><div>May be I might have not followed proper steps. Just wanted to know the steps you have followed. Otherwise, I was planning to manual migration also if possible.</div></div></div><div>~Jay.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 11, 2019 at 11:52 AM Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" rel="noreferrer" target="_blank">ignaziocassano@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Jay, <br></div><div>would you like to evacuate a failed compute node or evacuate a running compute node ?</div><div><br></div><div>Ignazio<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 11 lug 2019 alle ore 11:48 Jay See <<a href="mailto:jayachander.it@gmail.com" rel="noreferrer" target="_blank">jayachander.it@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Ignazio,<div><br></div><div>I am trying to evacuate the compute host on older version (mitaka).</div><div>Could please share the process you followed. I am not able to succeed with openstack live-migration fails with error message (this is known issue in older versions) and nova live-ligration - nothing happens even after initiating VM migration. It is almost 4 days.</div><div><br></div><div>~Jay.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 11, 2019 at 11:31 AM Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" rel="noreferrer" target="_blank">ignaziocassano@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I am sorry.</div><div>For simulating an host crash I used a wrong procedure.</div><div>Using "echo 'c' > /proc/sysrq-trigger" all work fine<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 11 lug 2019 alle ore 11:01 Ignazio Cassano <<a href="mailto:ignaziocassano@gmail.com" rel="noreferrer" target="_blank">ignaziocassano@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello All,</div><div>on ocata when I poweroff a node with active instance , doing a nova host-evacuate works fine</div><div>and instances are restartd on an active node.</div><div>On queens it does non evacuate instances but nova-api reports for each instance the following:</div><div><br></div><div>2019-07-11 10:19:54.745 13811 INFO nova.api.openstack.wsgi [req-daad0a7d-87ce-41bf-b096-a70fc306db5c 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: Cannot 'evacuate' instance e8485a5e-3623-4184-bcce-cafd56fa60b3 while it is in task_state powering-off</div><div><br></div><div>So it poweroff all instance on the failed node but does not start them on active nodes<br></div><div><br></div><div>What is changed ?</div><div>Ignazio</div><div><br></div><div><br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_5731346763580717613gmail-m_238756920609292433gmail-m_-6228910010257065679m_-4864518857793806283gmail-m_7375962085311809303gmail-m_-484793943738310843gmail-m_-3827110798004974146gmail-m_-5675443794367335487gmail_signature"><div dir="ltr"><span style="font-size:18pt;font-family:Webdings;color:green" lang="EN-GB"><div style="font-family:arial,helvetica,sans-serif;display:inline"></div>P</span><span style="font-size:8pt;font-family:Arial,sans-serif;color:green" lang="EN-US"> </span><b style="font-family:Calibri,sans-serif;font-size:14.4444px"><span style="font-size:8pt;color:green" lang="EN-US">SAVE PAPER – Please do not print this e-mail unless absolutely necessary.</span></b><br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_5731346763580717613gmail-m_238756920609292433gmail-m_-6228910010257065679m_-4864518857793806283gmail-m_7375962085311809303gmail-m_-484793943738310843gmail_signature"><div dir="ltr"><span style="font-size:18pt;font-family:Webdings;color:green" lang="EN-GB"><div style="font-family:arial,helvetica,sans-serif;display:inline"></div>P</span><span style="font-size:8pt;font-family:Arial,sans-serif;color:green" lang="EN-US"> </span><b style="font-family:Calibri,sans-serif;font-size:14.4444px"><span style="font-size:8pt;color:green" lang="EN-US">SAVE PAPER – Please do not print this e-mail unless absolutely necessary.</span></b><br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_5731346763580717613gmail-m_238756920609292433gmail-m_-6228910010257065679m_-4864518857793806283gmail_signature"><div dir="ltr"><span style="font-size:18pt;font-family:Webdings;color:green" lang="EN-GB"><div style="font-family:arial,helvetica,sans-serif;display:inline"></div>P</span><span style="font-size:8pt;font-family:Arial,sans-serif;color:green" lang="EN-US"> </span><b style="font-family:Calibri,sans-serif;font-size:14.4444px"><span style="font-size:8pt;color:green" lang="EN-US">SAVE PAPER – Please do not print this e-mail unless absolutely necessary.</span></b><br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_5731346763580717613gmail-m_238756920609292433gmail_signature"><div dir="ltr"><span style="font-size:18pt;font-family:Webdings;color:green" lang="EN-GB"><div style="font-family:arial,helvetica,sans-serif;display:inline"></div>P</span><span style="font-size:8pt;font-family:Arial,sans-serif;color:green" lang="EN-US"> </span><b style="font-family:Calibri,sans-serif;font-size:14.4444px"><span style="font-size:8pt;color:green" lang="EN-US">SAVE PAPER – Please do not print this e-mail unless absolutely necessary.</span></b><br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span lang="EN-GB" style="font-size:18pt;font-family:Webdings;color:green"><div style="font-family:arial,helvetica,sans-serif;display:inline"></div>P</span><span lang="EN-US" style="font-size:8pt;font-family:Arial,sans-serif;color:green"> </span><b style="font-family:Calibri,sans-serif;font-size:14.4444px"><span lang="EN-US" style="font-size:8pt;color:green">SAVE PAPER – Please do not print this e-mail unless absolutely necessary.</span></b><br></div></div>