<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Georgia;
panose-1:2 4 5 2 5 4 5 2 3 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.link-enhancr-element
{mso-style-name:link-enhancr-element;}
span.link-enhancr-view-on-domain
{mso-style-name:link-enhancr-view-on-domain;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Gabriel,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you very much for your help, it was exactly the same problem that you had.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I set the host variable on my rbd backend in the cinder config file and restarted the cinder-volume processes on my controllers.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US" style="color:#1F497D">Then I manually updated the database “cinder” to reflect the new cinder hostname “</span><span lang="EN-US">UPDATE volumes SET host=my_rbd_backend';”<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It fixed all my inconsistent volumes and images, I could deleted them again.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you again you definitely made my day.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">+++<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Romain<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Staicu Gabriel [mailto:gabriel_staicu@yahoo.com]
<br>
<b>Sent:</b> vendredi 5 juin 2015 10:31<br>
<b>To:</b> Aviolat Romain<br>
<b>Subject:</b> Re: [Openstack] VMs not terminated correctly and undeletable, with cinder and CEPH<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">Hi Romain,<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">I remember having a similar problem some time ago on havana. Some volumes were undeleteble. The issue in my case was the parameter at
line 570 from this document: <a href="http://docs.openstack.org/kilo/config-reference/content/section_cinder.conf.html" id="yui_3_16_0_1_1433492283924_3475">cinder.conf - OpenStack Configuration Reference - kilo</a><o:p></o:p></span></p>
</div>
<div style="margin-top:3.75pt;margin-bottom:3.75pt" id="enhancrCard_0">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="450" style="width:337.5pt;border-collapse:collapse">
<tbody>
<tr style="height:.75pt">
<td colspan="7" style="background:#E5E5E5;padding:0cm 0cm 0cm 0cm;height:.75pt">
<div id="yui_3_16_0_1_1433492283924_3492">
<p class="MsoNormal" style="mso-line-height-alt:0pt;background:#E5E5E5"><span style="font-size:1.0pt"> <o:p></o:p></span></p>
</div>
</td>
</tr>
<tr style="height:4.5pt" id="yui_3_16_0_1_1433492283924_3499">
<td width="1" rowspan="5" style="width:.75pt;background:#E5E5E5;padding:0cm 0cm 0cm 0cm;height:4.5pt">
<div>
<p class="MsoNormal" style="background:#E5E5E5"><span style="font-size:1.0pt"> <o:p></o:p></span></p>
</div>
</td>
<td width="14" rowspan="5" style="width:10.5pt;background:white;padding:0cm 0cm 0cm 0cm;height:4.5pt">
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:14.0pt"> <o:p></o:p></span></p>
</div>
</td>
<td colspan="2" style="background:white;padding:0cm 0cm 0cm 0cm;height:4.5pt" id="yui_3_16_0_1_1433492283924_3498">
<div id="yui_3_16_0_1_1433492283924_3497">
<p class="MsoNormal" style="mso-line-height-alt:4.5pt;background:white"><span style="font-size:6.0pt"> <o:p></o:p></span></p>
</div>
</td>
<td width="20" rowspan="5" style="width:15.0pt;background:white;padding:0cm 0cm 0cm 0cm;height:4.5pt">
<div>
<p class="MsoNormal" style="mso-line-height-alt:4.5pt;background:white"><span style="font-size:20.0pt"> <o:p></o:p></span></p>
</div>
</td>
<td width="1" rowspan="5" style="width:.75pt;background:#E5E5E5;padding:0cm 0cm 0cm 0cm;height:4.5pt">
<div>
<p class="MsoNormal" style="background:#E5E5E5"><span style="font-size:1.0pt"> <o:p></o:p></span></p>
</div>
</td>
<td style="padding:0cm 0cm 0cm 0cm;height:4.5pt"></td>
</tr>
<tr id="yui_3_16_0_1_1433492283924_3505">
<td width="100%" colspan="2" style="width:100.0%;padding:0cm 0cm 0cm 0cm" id="yui_3_16_0_1_1433492283924_3504">
<div id="yui_3_16_0_1_1433492283924_3503">
<div id="yui_3_16_0_1_1433492283924_3502">
<p class="MsoNormal" style="line-height:12.4pt;background:white;word-break:break-all">
<span style="font-family:"Georgia","serif""><a href="http://docs.openstack.org/kilo/config-reference/content/section_cinder.conf.html"><span class="link-enhancr-element"><span style="font-size:13.5pt;color:black">cinder.conf - OpenStack Configuration Reference
- kilo</span></span></a><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="line-height:15.0pt;background:white;word-break:break-all">
<span style="font-size:10.0pt;font-family:"Georgia","serif";color:#999999"> cinder.conf The cinder.conf file is installed in /etc/cinder by default.
<o:p></o:p></span></p>
</div>
</div>
</div>
</td>
<td style="padding:0cm 0cm 0cm 0cm"></td>
</tr>
<tr style="height:4.5pt">
<td colspan="2" style="background:white;padding:0cm 0cm 0cm 0cm;height:4.5pt"></td>
<td style="padding:0cm 0cm 0cm 0cm;height:4.5pt"></td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;font-family:"Arial","sans-serif""><a href="http://docs.openstack.org/kilo/config-reference/content/section_cinder.conf.html" target="_blank"><span class="link-enhancr-element"><span style="font-size:7.0pt;color:#999999">View
on </span></span><span class="link-enhancr-view-on-domain"><b><span style="font-size:7.0pt;color:#999999">docs.openstack.org</span></b></span></a><o:p></o:p></span></p>
</div>
</td>
<td width="100" style="width:75.0pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" align="right" style="text-align:right;line-height:8.25pt"><span class="link-enhancr-element"><span style="font-size:7.0pt;font-family:"Arial","sans-serif";color:#999999">Preview by Yahoo</span></span><span style="font-size:1.0pt;font-family:"Arial","sans-serif""><o:p></o:p></span></p>
</td>
<td style="padding:0cm 0cm 0cm 0cm"></td>
</tr>
<tr style="height:6.75pt">
<td colspan="2" style="background:white;padding:0cm 0cm 0cm 0cm;height:6.75pt"></td>
<td style="padding:0cm 0cm 0cm 0cm;height:6.75pt"></td>
</tr>
<tr style="height:.75pt">
<td colspan="7" style="background:#E5E5E5;padding:0cm 0cm 0cm 0cm;height:.75pt">
<div>
<p class="MsoNormal" style="mso-line-height-alt:0pt;background:#E5E5E5"><span style="font-size:1.0pt"> <o:p></o:p></span></p>
</div>
</td>
</tr>
<tr height="0">
<td width="1" style="border:none"></td>
<td width="5" style="border:none"></td>
<td width="243" style="border:none"></td>
<td width="193" style="border:none"></td>
<td width="7" style="border:none"></td>
<td width="1" style="border:none"></td>
<td width="0" style="border:none"></td>
</tr>
</tbody>
</table>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"> <o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">The idea was that when I was having the problem this parameter was set on the default value which was the host of the machine. So each
volume was bond the the cinder worker that created it.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">After I have modified it to an arbitrary string I did not see the problem with deleting volumes. Also that made the installation HA as
any cinder worked could take care of any volume in ceph cluster.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">Hope it gives you some hint for your issue.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">Have a nice day,<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_1_1433492283924_3285">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black">Gabriel<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt;background:white"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">On Friday, June 5, 2015 10:11 AM, Aviolat Romain <<a href="mailto:Romain.Aviolat@nagra.com">Romain.Aviolat@nagra.com</a>> wrote:</span><span style="font-family:"Helvetica","sans-serif";color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt;background:white"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt;background:white"><span style="font-family:"Helvetica","sans-serif";color:black">Dear Openstack community<br>
<br>
I have a strange intermittent problem with cinder when VMs with volumes attached are terminated. Sometimes the process doesn't work and the VMs are stuck into and undeletable state.<br>
<br>
Some info about my setup:<br>
*****************<br>
* Openstack Juno from ubuntu-cloud repo<br>
* Ubuntu 14.04 LTS on every machines <br>
* CEPH (giant) as storage backend on the storage nodes<br>
* 3x controllers in HA (HA-proxy, Galera, CEPH-monitors...)<br>
* RabbitMQ as messaging system<br>
*****************<br>
<br>
I put my whole setup in debug mode and here's what happened for a problematic VM:
<br>
<br>
Info about the VM and its volume:<br>
*****************<br>
Vm: suricata-bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc<br>
Volume attached: 821006f1-c655-4589-ba3e-7c6804c8e120 on /dev/vda<br>
Volume size: 500GB<br>
Running on hypervisor: cloudcompute2<br>
Libvirt instance ID: instance-000002e3<br>
*****************<br>
<br>
To begin here's the VM action log from Horizon, we can see that the user deleted the VM at 1.23 PM:<br>
*****************<br>
Request ID Action Start Time User ID Message<br>
req-22f33ed5-a231-4ea2-95fa-0022bf731079 delete June 4, 2015, 1:23 p.m. 27cf3aaa0d7942009b03eabf7f686849 Error<br>
*****************<br>
<br>
Here's the corresponding action on the controller that received the request:<br>
*****************<br>
2015-06-04 13:23:16.693 32302 DEBUG nova.compute.api [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] [instance: bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc] Going to try to terminate instance delete /usr/lib/python2.7/dist-packages/nova/compute/api.py:1802<br>
2015-06-04 13:23:16.774 32302 DEBUG nova.quota [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] Created reservations ['2221d555-1db0-4c36-9bd6-6dc815c22fc9', '3b70b444-7633-4a6b-b8b2-16128ee1469c', '6b8b2421-cfc1-4568-8bd8-430a982d977f'] reserve /usr/lib/python2.7/dist-packages/nova/quota.py:1310<br>
2015-06-04 13:23:16.799 32302 INFO nova.osapi_compute.wsgi.server [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] 172.24.1.17 "DELETE /v2/c9324c924a5049e7922882aff55c3813/servers/bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc HTTP/1.1" status: 204 len: 179 time: 0.1563590<br>
2015-06-04 13:23:16.883 32296 DEBUG nova.api.openstack.wsgi [req-903c3ea2-151a-4f8c-a48a-6991721c8c18 None] Calling method '<bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x7f869ec56790>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:937<br>
2015-06-04 13:23:16.884 32296 DEBUG nova.api.openstack.compute.servers [req-903c3ea2-151a-4f8c-a48a-6991721c8c18 None] Removing options 'project_id, limit' from query remove_invalid_options /usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py:1533<br>
*****************<br>
<br>
Then on nova-compute on the compute node hosting the VM, we receive the DELETE-instance request:<br>
*****************<br>
2015-06-04 13:23:16.847 1971 AUDIT nova.compute.manager [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] [instance: bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc] Terminating instance<br>
2015-06-04 13:23:16.854 1971 INFO nova.virt.libvirt.driver [-] [instance: bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc] Instance destroyed successfully.<br>
*****************<br>
<br>
Then the network device is removed:<br>
*****************<br>
2015-06-04 13:23:17.558 1971 DEBUG nova.network.linux_net [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] Net device removed: 'qvo20393401-b2' delete_net_dev /usr/lib/python2.7/dist-packages/nova/network/linux_net.py:1381<br>
*****************<br>
<br>
The the libvirt config file is removed:<br>
*****************<br>
2015-06-04 13:23:18.048 1971 INFO nova.virt.libvirt.driver [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] [instance: bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc] Deleting instance files /var/lib/nova/instances/bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc_del<br>
*****************<br>
<br>
The next one is strange:<br>
*****************<br>
2015-06-04 13:23:19.303 1971 DEBUG nova.virt.libvirt.driver [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] [instance: bbd3c3f5-85fb-4c90-8e17-e7f083a4a0bc] Could not determine fibre channel world wide node names get_volume_connector /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:1280<br>
2015-06-04 13:23:19.358 1971 DEBUG nova.volume.cinder [req-22f33ed5-a231-4ea2-95fa-0022bf731079 None] Cinderclient connection created using URL:
<a href="http://haproxy-api:8776/v2/c9324c924a5049e7922882aff55c3813" target="_blank">
http://haproxy-api:8776/v2/c9324c924a5049e7922882aff55c3813 </a>get_cinder_client_version /usr/lib/python2.7/dist-packages/nova/volume/cinder.py:255<br>
*****************<br>
<br>
We can see that the instance was correctly removed from the computenode:<br>
*****************<br>
[CloudCompute2:~] 1 $ sudo virsh list --all<br>
Id Name State<br>
----------------------------------------------------<br>
31 instance-0000043b running<br>
32 instance-000002d4 running<br>
42 instance-00000288 running<br>
47 instance-0000028f running<br>
48 instance-00000465 running<br>
52 instance-00000482 running<br>
*****************<br>
<br>
Then on controller2 we have some strange errors, looks like cinder sent messages but those timed-out (cinder-api.log).<br>
<br>
*****************<br>
2015-06-04 13:23:19.512 17252 INFO cinder.api.openstack.wsgi [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] POST
<a href="http://haproxy-api:8776/v2/c9324c924a5049e7922882aff55c3813/volumes/821006f1-c655-4589-ba3e-7c6804c8e120/action" target="_blank">
http://haproxy-api:8776/v2/c9324c924a5049e7922882aff55c3813/volumes/821006f1-c655-4589-ba3e-7c6804c8e120/action</a><br>
2015-06-04 13:23:19.513 17252 DEBUG cinder.api.openstack.wsgi [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] Action body: {"os-terminate_connection": {"connector": {"ip": "172.24.1.27", "host":
"cloudcompute2.ksnet.nagra.com", "initiator": "iqn.1993-08.org.debian:01:73625b6efe77"}}} get_method /usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py:1010<br>
2015-06-04 13:23:19.546 17252 INFO oslo.messaging._drivers.impl_rabbit [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] Connecting to AMQP server on cloudcontroller1-api:5672<br>
2015-06-04 13:23:19.556 17252 INFO oslo.messaging._drivers.impl_rabbit [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] Connected to AMQP server on cloudcontroller1-api:5672<br>
2015-06-04 13:23:19.567 17252 INFO oslo.messaging._drivers.impl_rabbit [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] Connecting to AMQP server on cloudcontroller0-api:5672<br>
2015-06-04 13:23:19.581 17252 INFO oslo.messaging._drivers.impl_rabbit [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] Connected to AMQP server on cloudcontroller0-api:5672<br>
2015-06-04 13:23:20.170 17247 INFO eventlet.wsgi.server [-] (17247) accepted ('172.24.1.19', 36820)<br>
2015-06-04 13:23:20.171 17247 INFO cinder.api.openstack.wsgi [-] OPTIONS <a href="http://172.24.1.19:8776/" target="_blank">
http://172.24.1.19:8776/</a><br>
2015-06-04 13:23:20.171 17247 DEBUG cinder.api.openstack.wsgi [-] Empty body provided in request get_body /usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py:789<br>
2015-06-04 13:23:20.172 17247 INFO cinder.api.openstack.wsgi [-] <a href="http://172.24.1.19:8776/" target="_blank">
http://172.24.1.19:8776/ </a>returned with HTTP 200<br>
2015-06-04 13:23:20.172 17247 INFO eventlet.wsgi.server [-] 172.24.1.19 - - [04/Jun/2015 13:23:20] "OPTIONS / HTTP/1.0" 200 281 0.001354<br>
2015-06-04 13:23:21.157 17245 INFO eventlet.wsgi.server [-] (17245) accepted ('172.24.1.17', 34260)<br>
2015-06-04 13:23:21.159 17245 INFO cinder.api.openstack.wsgi [-] OPTIONS <a href="http://172.24.1.19:8776/" target="_blank">
http://172.24.1.19:8776/</a><br>
2015-06-04 13:23:21.159 17245 DEBUG cinder.api.openstack.wsgi [-] Empty body provided in request get_body /usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py:789<br>
2015-06-04 13:23:21.160 17245 INFO cinder.api.openstack.wsgi [-] <a href="http://172.24.1.19:8776/" target="_blank">
http://172.24.1.19:8776/ </a>returned with HTTP 200<br>
2015-06-04 13:23:21.161 17245 INFO eventlet.wsgi.server [-] 172.24.1.17 - - [04/Jun/2015 13:23:21] "OPTIONS / HTTP/1.0" 200 281 0.002750<br>
2015-06-04 13:23:21.173 17251 INFO eventlet.wsgi.server [-] (17251) accepted ('172.24.1.18', 49259)<br>
2015-06-04 13:23:21.174 17251 INFO cinder.api.openstack.wsgi [-] OPTIONS <a href="http://172.24.1.19:8776/" target="_blank">
http://172.24.1.19:8776/</a><br>
2015-06-04 13:23:21.175 17251 DEBUG cinder.api.openstack.wsgi [-] Empty body provided in request get_body /usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py:789<br>
2015-06-04 13:23:21.176 17251 INFO cinder.api.openstack.wsgi [-] <a href="http://172.24.1.19:8776/" target="_blank">
http://172.24.1.19:8776/ </a>returned with HTTP 200 .....<br>
2015-06-04 13:24:19.338 17254 INFO eventlet.wsgi.server [-] 172.24.1.18 - - [04/Jun/2015 13:24:19] "OPTIONS / HTTP/1.0" 200 281 0.001299<br>
2015-06-04 13:24:19.585 17252 ERROR cinder.api.middleware.fault [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] Caught error: Timed out waiting for a reply to message ID 27d7e0cc6f444391823d5d4d1c1c85c1<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault Traceback (most recent call last):<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line 76, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return req.get_response(self.application)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault application, catch_exc_info=False)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault app_iter = application(self.environ, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return resp(environ, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return self.func(req, *args, **kwargs)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 99, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return request.get_response(self.application)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault application, catch_exc_info=False)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault app_iter = application(self.environ, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return self._app(env, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return resp(environ, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return resp(environ, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault response = self.app(environ, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return resp(environ, start_response)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return self.func(req, *args, **kwargs)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 897, in __call__<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault content_type, body, accept)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 945, in _process_stack<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 1021, in dispatch<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return method(req=request, **action_args)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/api/contrib/volume_actions.py", line 218, in _terminate_connection<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault self.volume_api.terminate_connection(context, volume, connector)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 87, in wrapped<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault return func(self, context, target_obj, *args, **kwargs)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 519, in terminate_connection<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault force)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/volume/rpcapi.py", line 170, in terminate_connection<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault connector=connector, force=force)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault retry=self.retry)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault timeout=timeout, retry=retry)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault retry=retry)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault result = self._waiter.wait(msg_id, timeout)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault reply, ending = self._poll_connection(msg_id, timeout)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault % msg_id)<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault MessagingTimeout: Timed out waiting for a reply to message ID 27d7e0cc6f444391823d5d4d1c1c85c1<br>
2015-06-04 13:24:19.585 17252 TRACE cinder.api.middleware.fault<br>
2015-06-04 13:24:19.596 17252 INFO cinder.api.middleware.fault [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -]
<a href="http://haproxy-api:8776/v2/c9324c924a5049e7922882aff55c3813/volumes/821006f1-c655-4589-ba3e-7c6804c8e120/action" target="_blank">
http://haproxy-api:8776/v2/c9324c924a5049e7922882aff55c3813/volumes/821006f1-c655-4589-ba3e-7c6804c8e120/action
</a>returned with HTTP 500<br>
2015-06-04 13:24:19.598 17252 INFO eventlet.wsgi.server [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] Traceback (most recent call last):<br>
File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 405, in handle_one_response<br>
write(''.join(towrite))<br>
File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 349, in write<br>
_writelines(towrite)<br>
File "/usr/lib/python2.7/socket.py", line 334, in writelines<br>
self.flush()<br>
File "/usr/lib/python2.7/socket.py", line 303, in flush<br>
self._sock.sendall(view[write_offset:write_offset+buffer_size])<br>
File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall<br>
tail = self.send(data, flags)<br>
File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send<br>
total_sent += fd.send(data[total_sent:], flags)<br>
error: [Errno 104] Connection reset by peer<br>
2015-06-04 13:24:19.598 17252 INFO eventlet.wsgi.server [req-ce2dcb7f-4bfd-4345-a199-165f13f7f11d 27cf3aaa0d7942009b03eabf7f686849 c9324c924a5049e7922882aff55c3813 - - -] 172.24.1.17 - - [04/Jun/2015 13:24:19] "POST /v2/c9324c924a5049e7922882aff55c3813/volumes/821006f1-c655-4589-ba3e-7c6804c8e120/action
HTTP/1.1" 500 0 60.230434<br>
2015-06-04 13:24:20.321 17254 INFO eventlet.wsgi.server [-] (17254) accepted ('172.24.1.19', 37634)<br>
2015-06-04 13:24:20.321 17254 INFO cinder.api.openstack.wsgi [-] OPTIONS <a href="http://172.24.1.19:8776/" target="_blank">
http://172.24.1.19:8776/</a><br>
*****************<br>
<br>
I checked in the CEPH cluster and the volume wasn't deleted. I also checked the state of the RabbitMQ cluster and everything is working as expected.<br>
<br>
I'm not able to reproduce the problem, it happen sometimes but I haven't found the trigger... When it happens the volume is un-deletable, it's marked as in-use by the VM and the VM is un-deletable because a volume is attached. The only way I found is to edit
the database (nove, cinder, ...) and delete the volume in the CEPH cluster manually.
<br>
<br>
You help will be very much appreciated ! <br>
<br>
Thanks,<br>
<br>
Romain<br>
<br>
Romain Aviolat<br>
Senior System Administrator - R&D and ops Infrastructure <br>
Kudelski Security - Kudelski Group <br>
rte de Genève 22-24, 1033 Cheseaux, SWITZERLAND<br>
+41 21 732 03 79<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>