<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
Thanks for your help all, glance is also implemented with ceph rbd
and both have ample amounts of space available:<br>
<br>
compute node: /dev/sda1 5.5T 890G 4.3T 17% /<br>
<br>
Ceph cluster: 16555 GB used, 23666 GB / 40221 GB avail<br>
<br>
Turns out is in an issue with the keystone tokens that are timing
out when the snapshot is taking place.<br>
<br>
I will get onto looking into that now.<br>
<br>
Thanks again for the advice and help.<br>
<br>
Grant<br>
<br>
<div class="moz-cite-prefix">On 03/06/16 11:57, Saverio Proto wrote:<br>
</div>
<blockquote
cite="mid:CAPmmg8u8gigkjOvaHOkubUgWMCERgfoJyiYR4E3ki5udts1WYQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hello,<br>
<br>
</div>
what is the state of the instance before asking the snapshot
? Is it running or paused ?<br>
<br>
</div>
Check on the hypervisor when the snapshot starts if you see
files in these folders:<br>
<br>
/var/lib/libvirt/qemu/save/<br>
/var/lib/nova/instances/<span class="">snapshots</span>/<br>
<br>
</div>
<div>How is your glance implemented ? Also with ceph rbd ?
Remember that a "nova snapshot" is a glance image.<br>
</div>
<div><br>
</div>
Saverio<br>
<br>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-06-03 12:17 GMT+02:00 Grant <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:grant@absolutedevops.io" target="_blank">grant@absolutedevops.io</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi all,<br>
<br>
I was wondering if someone could shed any light on an
issue we are seeing. We are running Kilo in our production
environment and when we are trying to create a snapshot
for a particular instance it gets stuck in a "saving"
state and doesn't actually ever save the image.<br>
<br>
We are using a ceph back-end and the user that is trying
to take a snapshot is able to take snapshots of all of
their other instances, it is just one that is failing.<br>
<br>
Error log from the nova compute host below:<br>
<br>
2016-06-02 17:13:48.594 52559 WARNING
urllib3.connectionpool
[req-8200a3b0-ad2a-406e-969e-c22762db3455
bb07f987fbae485c9e05f06fb0d422c2
a22e503869c34a92bceb66b0c1da7231 - - -] HttpConnectionPool
is full, discarding connection: 10.5.0.205<br>
2016-06-02 17:14:00.042 52559 ERROR nova.compute.manager
[req-8200a3b0-ad2a-406e-969e-c22762db3455
bb07f987fbae485c9e05f06fb0d422c2
a22e503869c34a92bceb66b0c1da7231 - - -] [instance:
70d42d14-66f6-4374-9038-4b6f840193e0] Error while trying<br>
to clean up image f9844dd5-5a92-4cd4-956d-8ad04cfc5e84<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] Traceback
(most recent call last):<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
line 405, in decorated_function<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
self.image_api.delete(context, image_id)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] File
"/usr/lib/python2.7/dist-packages/nova/image/api.py", line
141, in delete<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
return session.delete(context, image_id)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py",
line 410, in delete<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
self._client.call(context, 1, 'delete', image_id)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py",
line 218, in call<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
return getattr(client.images, method)(*args, **kwargs)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] File
"/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py",
line 255, in delete<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] resp,
body = self.client.delete(url)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py",
line 271, in delete<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
return self._request('DELETE', url, **kwargs)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py",
line 227, in _request<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] raise
exc.from_response(resp, resp.content)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
HTTPUnauthorized: <html><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
<head><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
<title>401 Unauthorized</title><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
</head><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
<body><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
<h1>401 Unauthorized</h1><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] This
server could not verify that you are authorized to access
the document you requested. Either you supplied the wrong
credential<br>
s (e.g., bad password), or your browser does not
understand how to supply the credentials required.<br
/><br /><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] <br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
</body><br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0]
</html> (HTTP 401)<br>
2016-06-02 17:14:00.042 52559 TRACE nova.compute.manager
[instance: 70d42d14-66f6-4374-9038-4b6f840193e0] <br>
2016-06-02 17:14:00.173 52559 ERROR
oslo_messaging.rpc.dispatcher
[req-8200a3b0-ad2a-406e-969e-c22762db3455
bb07f987fbae485c9e05f06fb0d422c2
a22e503869c34a92bceb66b0c1da7231 - - -] Exception during
message handling: Not authorized for imag<br>
e f9844dd5-5a92-4cd4-956d-8ad04cfc5e84.<br>
<br>
Any help will be appreciated.<br>
<br>
Regards,<span class="HOEnZb"><font color="#888888"><br>
<br>
<div>-- <br>
<div style="min-height:13px"> </div>
<div> <font color="#A8D8FF" face="Lucida Grande,
Verdana, Arial, sans-serif"> <span
style="font-size:11px;line-height:15px;float:left;margin-right:10px">
<img
src="cid:part2.02060902.05030801@absolutedevops.io"
style="color:#a8d8ff;font-family:'Lucida
Grande',Verdana,Arial,sans-serif;font-size:11px;line-height:15px;float:left;margin-top:0px;margin-right:10px">
</span> </font>
<div
style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:5px">
<span style="font-family:Hind,'Lucida
Grande',Verdana,Arial,sans-serif;font-size:18px;font-weight:200;color:rgb(35,157,255)">Grant
Morley</span> <span
style="color:rgb(35,157,255);font-size:15px">
</span> <font face="Hind, Lucida Grande,
Verdana, Arial, sans-serif"> <span
style="font-size:15px"> </span> </font> </div>
<div
style="margin-left:80px;line-height:15px;color:rgb(109,190,255);margin-bottom:10px">
<font face="Hind, Lucida Grande, Verdana, Arial,
sans-serif"> <span style="font-size:15px">Cloud
Lead</span> </font> </div>
<div style="font-family:Hind,'Lucida
Grande',Verdana,Arial,sans-serif;font-size:13px;margin-left:80px;line-height:15px;color:rgb(153,153,153)">
<span style="color:#616161">Absolute</span> <span
style="color:#98a5b0">DevOps Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way,
Stevenage, Herts, SG1 2FP </div>
<div
style="font-family:Hind,helvetica,arial;font-size:13px;margin-left:80px">
<a moz-do-not-send="true"
href="http://www.absolutedevops.io"
target="_blank">www.absolutedevops.io</a> <a
moz-do-not-send="true"
href="mailto:grant@absolutedevops.io"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io">grant@absolutedevops.io</a></a> <a
moz-do-not-send="true"
style="color:#239dff;font-family:Hind,'Lucida
Grande',Verdana,Arial,sans-serif;line-height:15px;margin-right:6px">0845
874 0580</a> </div>
</div>
</div>
</font></span></div>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<!--?xml version="1.0" encoding="UTF-8" standalone="no"?-->
<title></title>
<div style="height:13px"> </div>
<div> <font color="#A8D8FF" face="Lucida Grande, Verdana, Arial,
sans-serif"> <span style="font-size: 11px; line-height: 15px;
float: left; margin-right: 10px;"> <img
src="cid:part8.09020308.08000901@absolutedevops.io"
style="color: #a8d8ff; font-family: 'Lucida Grande',
Verdana, Arial, sans-serif; font-size: 11px; line-height:
15px; float: left; margin-top: 0px; margin-right: 10px"> </span>
</font>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 5px;"> <span
style="font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif; font-size: 18px; font-weight: 200; color:
rgb(35, 157, 255);">Grant Morley</span> <span style="color:
rgb(35, 157, 255); font-size: 15px;"> </span> <font
face="Hind, Lucida Grande, Verdana, Arial, sans-serif"> <span
style="font-size: 15px;"> </span> </font> </div>
<div style="margin-left: 80px; line-height: 15px; color:
rgb(109, 190, 255); margin-bottom: 10px;"> <font face="Hind,
Lucida Grande, Verdana, Arial, sans-serif"> <span
style="font-size: 15px;">Cloud Lead</span> </font> </div>
<div style="font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif; font-size: 13px; margin-left: 80px; line-height:
15px; color: rgb(153, 153, 153);"> <span style="color:
#616161">Absolute</span> <span style="color: #98a5b0">DevOps
Ltd</span> <br>
Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
Herts, SG1 2FP
</div>
<div style="font-family: Hind, helvetica, arial; font-size:
13px; margin-left: 80px;"> <a
href="http://www.absolutedevops.io/" style="color: #239dff;
font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;"><a class="moz-txt-link-abbreviated" href="http://www.absolutedevops.io">www.absolutedevops.io</a></a> <a
href="mailto:grant@absolutedevops.i" style="color: #239dff;
font-family: Hind, 'Lucida Grande', Verdana, Arial,
sans-serif;line-height: 15px; margin-right: 6px;
text-decoration: none;"><a class="moz-txt-link-abbreviated" href="mailto:grant@absolutedevops.io">grant@absolutedevops.io</a></a> <a
style="color: #239dff; font-family: Hind, 'Lucida Grande',
Verdana, Arial, sans-serif;line-height: 15px; margin-right:
6px;">0845 874 0580</a> </div>
</div>
</div>
</body>
</html>