<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>We're working on that. The original Ceph bug has been fixed in
Jewel and backported to Hammer. Now Glance is trying to use
image.size() instead of image.stat() to get the size. Here is the
link <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/378460/">https://review.openstack.org/#/c/378460/</a></p>
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 01/02/17 01:22, Grant Morley wrote:<br>
</div>
<blockquote
cite="mid:959aa173-52aa-7ee8-1261-8f645182690c@absolutedevops.io"
type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<p>Hi Nick,</p>
<p>Thanks for the reply, looks like we could be hitting something
similar. We are running Ceph Jewel packages on the Glance node.</p>
<p>Thanks for the links to the bug reports.</p>
<p>Regards,<br>
</p>
<br>
<div class="moz-cite-prefix">On 31/01/17 11:35, Nick Jones wrote:<br>
</div>
<blockquote
cite="mid:CAB0HEW6mbfjB6DjR1b10tKi_+L0BMWt3_9O3UJq8M4b0E0dBkA@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Grant.
<div><br>
</div>
<div>Could be unrelated but I'm throwing it out there anyway
as we had similar 'weirdness' with Glance recently... If
you're running the Ceph Jewel or Hammer packages on your
Glance node then you might be running into this bug, in
which image.stat() calls in librbdpy fail intermittently:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="http://tracker.ceph.com/issues/17310">http://tracker.ceph.com/issues/17310</a><br>
</div>
<div><br>
</div>
<div>There's a corresponding bug report on Launchpad for the
Ubuntu Ceph package that has some more detail:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489">https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489</a><br>
</div>
<div><br>
</div>
<div>Cheers.</div>
<div><br>
</div>
<div class="gmail_extra">
<div>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>-- </div>
<div><br>
<div>-Nick</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On 31 January 2017 at 10:49, Grant
Morley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi All,</p>
<p>Not sure if anyone has come across this yet.
Yesterday we started to try to upload some images to
our stack and they were failing with the following
error when trying via command line:</p>
<p>Error finding address for <a
moz-do-not-send="true"
class="gmail-m_-8298096474797945511moz-txt-link-freetext"
href="http://10.6.0.3:9292/v2/images/e90a4626-4781-4b53-8914-85ff2129f777/file"
target="_blank"><a class="moz-txt-link-freetext" href="http://10.6.0.3:9292/v2/">http://10.6.0.3:9292/v2/</a><wbr>images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777/file</a>:
[Errno 32] Broken pipe</p>
<p>In the Glance logs at the same time we see this:</p>
<p>20172017-01-31 10:22:45.005 2096 INFO
eventlet.wsgi.server [req-1837bce5-a90c-44d5-bf65-<wbr>c1e3a534c793
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45]
"HEAD /v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.046623<br>
2017-01-31 10:22:47.595 2099 INFO
eventlet.wsgi.server [-] 10.6.0.40 - - [31/Jan/2017
10:22:47] "OPTIONS /versions HTTP/1.0" 200 785
0.001113<br>
2017-01-31 10:22:47.720 2095 INFO
eventlet.wsgi.server [req-873dd7df-f8a4-4443-8795-<wbr>b81dcd54f412
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47]
"HEAD /v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.032188-01-31 10:22:45.005 2096
INFO eventlet.wsgi.server
[req-1837bce5-a90c-44d5-bf65-<wbr>c1e3a534c793
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45]
"HEAD /v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.046623<br>
2017-01-31 10:22:47.595 2099 INFO
eventlet.wsgi.server [-] 10.6.0.40 - - [31/Jan/2017
10:22:47] "OPTIONS /versions HTTP/1.0" 200 785
0.001113<br>
2017-01-31 10:22:47.720 2095 INFO
eventlet.wsgi.server [req-873dd7df-f8a4-4443-8795-<wbr>b81dcd54f412
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47]
"HEAD /v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 200 689 0.032188</p>
<p>Interestingly, when you delete the failed image you
get this in the glance logs:</p>
<p>2017-01-31 10:23:16.188 2099 INFO
eventlet.wsgi.server [req-c2191fe1-64de-4252-b4a2-<wbr>e84643dbed4c
4c91f07132454a97b21fff35402b78<wbr>25
4a6213a64312482896130efc304719<wbr>5c - - -]
10.6.2.190,10.6.0.39 - - [31/Jan/2017 10:23:16]
"DELETE /v2/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777
HTTP/1.1" 204 208 0.08</p>
<p><br>
</p>
<p>It seems to be using V1 to try and upload and V2 to
remove? - However the image doesn't actually get
deleted because it remains in ceph:</p>
<p>rbd -p images ls | grep e90a4626-4781-4b53-8914-<wbr>85ff2129f777<br>
e90a4626-4781-4b53-8914-<wbr>85ff2129f777</p>
<p>It is almost as if, the image uploads and is then
forgotten about.</p>
<p>Our compute nodes are also ceph backed and they are
working absolutely fine, it is just Glance and ceph
has all of a sudden stopped working. Just wondered
if anyone had any ideas?</p>
<p>Regards,<span class="gmail-HOEnZb"><font
color="#888888"><br>
</font></span></p>
<span class="gmail-HOEnZb"><font color="#888888">
<p><br>
</p>
<div
class="gmail-m_-8298096474797945511moz-signature">--
<br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana,
Arial, sans-serif" color="#A8D8FF"> <span
style="font-size:11px;line-height:15px;float:left;margin-right:10px">
<img
src="cid:part5.25E4B63B.5BE3F68C@catalyst.net.nz"
style="color: rgb(168, 216, 255);
font-family: "lucida grande",
verdana, arial, sans-serif; font-size:
11px; line-height: 15px; float: left;
margin-top: 0px; margin-right: 10px;"> </span>
</font>
<div
style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
<span style="font-family:hind,"lucida
grande",verdana,arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant
Morley</span> <span
style="color:rgb(35,157,255);font-size:15px">
</span> <font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"> <span
style="font-size:15px"> </span> </font>
</div>
<div
style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
<font face="Hind, Lucida Grande, Verdana,
Arial, sans-serif"> <span
style="font-size:15px">Cloud Lead</span>
</font> </div>
<div style="font-family:hind,"lucida
grande",verdana,arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)">
<span style="color:rgb(97,97,97)">Absolute</span>
<span style="color:rgb(152,165,176)">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle
Way, Stevenage, Herts, SG1 2FP </div>
<div
style="font-family:hind,helvetica,arial;font-size:13px;margin-left:80px">
<a moz-do-not-send="true"
href="http://www.absolutedevops.io/"
style="color:rgb(35,157,255);font-family:hind,"lucida
grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none"
target="_blank">www.absolutedevops.io</a>
<a moz-do-not-send="true"
href="mailto:grant@absolutedevops.i"
style="color:rgb(35,157,255);font-family:hind,"lucida
grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none"
target="_blank">grant@absolutedevops.io</a>
<a moz-do-not-send="true"
style="color:rgb(35,157,255);font-family:hind,"lucida
grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px">0845
874 0580</a> </div>
</div>
</div>
</font></span></div>
<br>
______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators"
rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<span
style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">DataCentred
Limited registered in England and Wales no. 05611763</span> </blockquote>
<br>
<div class="moz-signature">-- <br>
<!--?xml version="1.0" encoding="UTF-8" standalone="no"?-->
<title></title>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial, sans-serif"
color="#A8D8FF"> <span style="font-size: 11px; line-height:
15px; float: left; margin-right: 10px;"> <img
src="cid:part11.95EE1984.F3C2B2F0@catalyst.net.nz"
style="color: #a8d8ff; font-family: 'Lucida Grande',
Verdana, Arial, sans-serif; font-size: 11px;
line-height: 15px; float: left; margin-top: 0px;
margin-right: 10px"> </span> </font>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"> <span
style="font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif; font-size: 18px; font-weight: 200; color:
rgb(35, 157, 255);">Grant Morley</span> <span
style="color: rgb(35, 157, 255); font-size: 15px;"> </span>
<font face="Hind, Lucida Grande, Verdana, Arial, sans-serif">
<span style="font-size: 15px;"> </span> </font> </div>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 10px;"> <font
face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span
style="font-size: 15px;">Cloud Lead</span> </font> </div>
<div style="font-family: Hind, 'Lucida Grande', Verdana,
Arial, sans-serif; font-size: 13px; margin-left: 80px;
line-height: 15px; color: rgb(153, 153, 153);"> <span
style="color: #616161">Absolute</span> <span
style="color: #98a5b0">DevOps Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP </div>
<div style="font-family: Hind, helvetica, arial; font-size:
13px; margin-left: 80px;"> <a moz-do-not-send="true"
href="http://www.absolutedevops.io/" style="color:
#239dff; font-family: Hind, 'Lucida Grande', Verdana,
Arial, sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;">www.absolutedevops.io</a> <a
moz-do-not-send="true"
href="mailto:grant@absolutedevops.i" style="color:
#239dff; font-family: Hind, 'Lucida Grande', Verdana,
Arial, sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;"><a class="moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io">grant@absolutedevops.io</a></a> <a
moz-do-not-send="true" style="color: #239dff; font-family:
Hind, 'Lucida Grande', Verdana, Arial,
sans-serif;line-height: 15px; margin-right: 6px;">0845 874
0580</a> </div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-operators mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Cheers & Best regards,
FeiLong Wang (王飞龙)
--------------------------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a class="moz-txt-link-abbreviated" href="mailto:flwang@catalyst.net.nz">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
-------------------------------------------------------------------------- </pre>
</body>
</html>