<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Nick,</p>
<p>Thanks for the reply, looks like we could be hitting something
similar. We are running Ceph Jewel packages on the Glance node.</p>
<p>Thanks for the links to the bug reports.</p>
<p>Regards,<br>
</p>
<br>
<div class="moz-cite-prefix">On 31/01/17 11:35, Nick Jones wrote:<br>
</div>
<blockquote
cite="mid:CAB0HEW6mbfjB6DjR1b10tKi_+L0BMWt3_9O3UJq8M4b0E0dBkA@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Grant.
<div><br>
</div>
<div>Could be unrelated but I'm throwing it out there anyway as
we had similar 'weirdness' with Glance recently... If you're
running the Ceph Jewel or Hammer packages on your Glance node
then you might be running into this bug, in which image.stat()
calls in librbdpy fail intermittently:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="http://tracker.ceph.com/issues/17310">http://tracker.ceph.com/issues/17310</a><br>
</div>
<div><br>
</div>
<div>There's a corresponding bug report on Launchpad for the
Ubuntu Ceph package that has some more detail:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489">https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489</a><br>
</div>
<div><br>
</div>
<div>Cheers.</div>
<div><br>
</div>
<div class="gmail_extra">
<div>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>-- </div>
<div><br>
<div>-Nick</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On 31 January 2017 at 10:49, Grant
Morley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi All,</p>
<p>Not sure if anyone has come across this yet.
Yesterday we started to try to upload some images to
our stack and they were failing with the following
error when trying via command line:</p>
<p>Error finding address for
<a moz-do-not-send="true"
class="gmail-m_-8298096474797945511moz-txt-link-freetext"
href="http://10.6.0.3:9292/v2/images/e90a4626-4781-4b53-8914-85ff2129f777/file"
target="_blank">http://10.6.0.3:9292/v2/<wbr>images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777/file</a>:
[Errno 32] Broken pipe</p>
<p>In the Glance logs at the same time we see this:</p>
<p>20172017-01-31 10:22:45.005 2096 INFO
eventlet.wsgi.server [req-1837bce5-a90c-44d5-bf65-<wbr>c1e3a534c793
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.046623<br>
2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server
[-] 10.6.0.40 - - [31/Jan/2017 10:22:47] "OPTIONS
/versions HTTP/1.0" 200 785 0.001113<br>
2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server
[req-873dd7df-f8a4-4443-8795-<wbr>b81dcd54f412
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.032188-01-31 10:22:45.005 2096
INFO eventlet.wsgi.server
[req-1837bce5-a90c-44d5-bf65-<wbr>c1e3a534c793
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.046623<br>
2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server
[-] 10.6.0.40 - - [31/Jan/2017 10:22:47] "OPTIONS
/versions HTTP/1.0" 200 785 0.001113<br>
2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server
[req-873dd7df-f8a4-4443-8795-<wbr>b81dcd54f412
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.032188</p>
<p>Interestingly, when you delete the failed image you
get this in the glance logs:</p>
<p>2017-01-31 10:23:16.188 2099 INFO
eventlet.wsgi.server [req-c2191fe1-64de-4252-b4a2-<wbr>e84643dbed4c
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.2.190,10.6.0.39 - - [31/Jan/2017 10:23:16]
"DELETE /v2/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 204 208 0.08</p>
<p><br>
</p>
<p>It seems to be using V1 to try and upload and V2 to
remove? - However the image doesn't actually get
deleted because it remains in ceph:</p>
<p>rbd -p images ls | grep e90a4626-4781-4b53-8914-<wbr>85ff2129f777<br>
e90a4626-4781-4b53-8914-<wbr>85ff2129f777</p>
<p>It is almost as if, the image uploads and is then
forgotten about.</p>
<p>Our compute nodes are also ceph backed and they are
working absolutely fine, it is just Glance and ceph
has all of a sudden stopped working. Just wondered if
anyone had any ideas?</p>
<p>Regards,<span class="gmail-HOEnZb"><font
color="#888888"><br>
</font></span></p>
<span class="gmail-HOEnZb"><font color="#888888">
<p><br>
</p>
<div
class="gmail-m_-8298096474797945511moz-signature">--
<br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial,
sans-serif" color="#A8D8FF"> <span
style="font-size:11px;line-height:15px;float:left;margin-right:10px">
<img
src="cid:part5.09B96645.27D2F29E@absolutedevops.io"
style="color: rgb(168, 216, 255);
font-family: "lucida grande",
verdana, arial, sans-serif; font-size:
11px; line-height: 15px; float: left;
margin-top: 0px; margin-right: 10px;"> </span>
</font>
<div
style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
<span style="font-family:hind,"lucida
grande",verdana,arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant
Morley</span> <span
style="color:rgb(35,157,255);font-size:15px">
</span> <font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"> <span
style="font-size:15px"> </span> </font>
</div>
<div
style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
<font face="Hind, Lucida Grande, Verdana,
Arial, sans-serif"> <span
style="font-size:15px">Cloud Lead</span> </font>
</div>
<div style="font-family:hind,"lucida
grande",verdana,arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)">
<span style="color:rgb(97,97,97)">Absolute</span>
<span style="color:rgb(152,165,176)">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way,
Stevenage, Herts, SG1 2FP </div>
<div
style="font-family:hind,helvetica,arial;font-size:13px;margin-left:80px">
<a moz-do-not-send="true"
href="http://www.absolutedevops.io/"
style="color:rgb(35,157,255);font-family:hind,"lucida
grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none"
target="_blank">www.absolutedevops.io</a> <a
moz-do-not-send="true"
href="mailto:grant@absolutedevops.i"
style="color:rgb(35,157,255);font-family:hind,"lucida
grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none"
target="_blank">grant@absolutedevops.io</a>
<a moz-do-not-send="true"
style="color:rgb(35,157,255);font-family:hind,"lucida
grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px">0845
874 0580</a> </div>
</div>
</div>
</font></span></div>
<br>
______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators"
rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<span
style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">DataCentred
Limited registered in England and Wales no. 05611763</span>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<!--?xml version="1.0" encoding="UTF-8" standalone="no"?-->
<title></title>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"> <span style="font-size: 11px; line-height:
15px; float: left; margin-right: 10px;"> <img
src="cid:part11.9897E8EA.1D06C07F@absolutedevops.io"
style="color: #a8d8ff; font-family: 'Lucida Grande',
Verdana, Arial, sans-serif; font-size: 11px; line-height:
15px; float: left; margin-top: 0px; margin-right: 10px"> </span>
</font>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"> <span
style="font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif; font-size: 18px; font-weight: 200; color:
rgb(35, 157, 255);">Grant Morley</span> <span style="color:
rgb(35, 157, 255); font-size: 15px;"> </span> <font
face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span
style="font-size: 15px;"> </span> </font> </div>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 10px;"> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span
style="font-size: 15px;">Cloud Lead</span> </font> </div>
<div style="font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif; font-size: 13px; margin-left: 80px; line-height:
15px; color: rgb(153, 153, 153);"> <span style="color:
#616161">Absolute</span> <span style="color: #98a5b0">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP
</div>
<div style="font-family: Hind, helvetica, arial; font-size:
13px; margin-left: 80px;"> <a
href="http://www.absolutedevops.io/" style="color: #239dff;
font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;">www.absolutedevops.io</a> <a
href="mailto:grant@absolutedevops.i" style="color: #239dff;
font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;">grant@absolutedevops.io</a> <a
style="color: #239dff; font-family: Hind, 'Lucida Grande',
Verdana, Arial, sans-serif;line-height: 15px; margin-right:
6px;">0845 874 0580</a> </div>
</div>
</div>
</body>
</html>