<div dir="ltr">Hi Grant.<div><br></div><div>Could be unrelated but I'm throwing it out there anyway as we had similar 'weirdness' with Glance recently... If you're running the Ceph Jewel or Hammer packages on your Glance node then you might be running into this bug, in which image.stat() calls in librbdpy fail intermittently:</div><div><br></div><div><a href="http://tracker.ceph.com/issues/17310">http://tracker.ceph.com/issues/17310</a><br></div><div><br></div><div>There's a corresponding bug report on Launchpad for the Ubuntu Ceph package that has some more detail:</div><div><br></div><div><a href="https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489">https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489</a><br></div><div><br></div><div>Cheers.</div><div><br></div><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>-- </div><div><br><div>-Nick</div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On 31 January 2017 at 10:49, Grant Morley <span dir="ltr"><<a href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi All,</p>
<p>Not sure if anyone has come across this yet. Yesterday we started
to try to upload some images to our stack and they were failing
with the following error when trying via command line:</p>
<p>Error finding address for
<a class="gmail-m_-8298096474797945511moz-txt-link-freetext" href="http://10.6.0.3:9292/v2/images/e90a4626-4781-4b53-8914-85ff2129f777/file" target="_blank">http://10.6.0.3:9292/v2/<wbr>images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777/file</a>:
[Errno 32] Broken pipe</p>
<p>In the Glance logs at the same time we see this:</p>
<p>20172017-01-31 10:22:45.005 2096 INFO eventlet.wsgi.server
[req-1837bce5-a90c-44d5-bf65-<wbr>c1e3a534c793
4c91f07132454a97b21fff35402b78<wbr>25 4a6213a64312482896130efc304719<wbr>5c
- - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777 HTTP/1.1" 200 689
0.046623<br>
2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server [-]
10.6.0.40 - - [31/Jan/2017 10:22:47] "OPTIONS /versions HTTP/1.0"
200 785 0.001113<br>
2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server
[req-873dd7df-f8a4-4443-8795-<wbr>b81dcd54f412
4c91f07132454a97b21fff35402b78<wbr>25 4a6213a64312482896130efc304719<wbr>5c
- - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777 HTTP/1.1" 200 689
0.032188-01-31 10:22:45.005 2096 INFO eventlet.wsgi.server
[req-1837bce5-a90c-44d5-bf65-<wbr>c1e3a534c793
4c91f07132454a97b21fff35402b78<wbr>25 4a6213a64312482896130efc304719<wbr>5c
- - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777 HTTP/1.1" 200 689
0.046623<br>
2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server [-]
10.6.0.40 - - [31/Jan/2017 10:22:47] "OPTIONS /versions HTTP/1.0"
200 785 0.001113<br>
2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server
[req-873dd7df-f8a4-4443-8795-<wbr>b81dcd54f412
4c91f07132454a97b21fff35402b78<wbr>25 4a6213a64312482896130efc304719<wbr>5c
- - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD
/v1/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777 HTTP/1.1" 200 689
0.032188</p>
<p>Interestingly, when you delete the failed image you get this in
the glance logs:</p>
<p>2017-01-31 10:23:16.188 2099 INFO eventlet.wsgi.server
[req-c2191fe1-64de-4252-b4a2-<wbr>e84643dbed4c
4c91f07132454a97b21fff35402b78<wbr>25 4a6213a64312482896130efc304719<wbr>5c
- - -] 10.6.2.190,10.6.0.39 - - [31/Jan/2017 10:23:16] "DELETE
/v2/images/e90a4626-4781-4b53-<wbr>8914-85ff2129f777 HTTP/1.1" 204 208
0.08</p>
<p><br>
</p>
<p>It seems to be using V1 to try and upload and V2 to remove? -
However the image doesn't actually get deleted because it remains
in ceph:</p>
<p>rbd -p images ls | grep e90a4626-4781-4b53-8914-<wbr>85ff2129f777<br>
e90a4626-4781-4b53-8914-<wbr>85ff2129f777</p>
<p>It is almost as if, the image uploads and is then forgotten
about.</p>
<p>Our compute nodes are also ceph backed and they are working
absolutely fine, it is just Glance and ceph has all of a sudden
stopped working. Just wondered if anyone had any ideas?</p>
<p>Regards,<span class="gmail-HOEnZb"><font color="#888888"><br>
</font></span></p><span class="gmail-HOEnZb"><font color="#888888">
<p><br>
</p>
<div class="gmail-m_-8298096474797945511moz-signature">-- <br>
<div style="height:13px"> </div>
<div> <font face="Lucida Grande, Verdana, Arial, sans-serif" color="#A8D8FF"> <span style="font-size:11px;line-height:15px;float:left;margin-right:10px"> <img src="cid:part1.977ECD35.239C86DC@absolutedevops.io" style="color: rgb(168, 216, 255); font-family: "lucida grande", verdana, arial, sans-serif; font-size: 11px; line-height: 15px; float: left; margin-top: 0px; margin-right: 10px;"> </span>
</font>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px"> <span style="font-family:hind,"lucida grande",verdana,arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant Morley</span> <span style="color:rgb(35,157,255);font-size:15px"> </span> <font face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px"> </span> </font> </div>
<div style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px"> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span style="font-size:15px">Cloud Lead</span> </font> </div>
<div style="font-family:hind,"lucida grande",verdana,arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)"> <span style="color:rgb(97,97,97)">Absolute</span> <span style="color:rgb(152,165,176)">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP
</div>
<div style="font-family:hind,helvetica,arial;font-size:13px;margin-left:80px"> <a href="http://www.absolutedevops.io/" style="color:rgb(35,157,255);font-family:hind,"lucida grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">www.absolutedevops.io</a> <a href="mailto:grant@absolutedevops.i" style="color:rgb(35,157,255);font-family:hind,"lucida grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px;text-decoration:none" target="_blank">grant@absolutedevops.io</a> <a style="color:rgb(35,157,255);font-family:hind,"lucida grande",verdana,arial,sans-serif;line-height:15px;margin-right:6px">0845 874 0580</a> </div>
</div>
</div>
</font></span></div>
<br>______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
<br></blockquote></div><br></div></div>
<br>
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">DataCentred Limited registered in England and Wales no. 05611763</span>