<html><body>
<p><font size="2" face="sans-serif">I just verified this scenario. If image uploading is cancelled by Ctrl+C in v2, the image will be stuck at "saving" status because of DisconnectionError. Obviously, it doesn't make sense to leave the image in saving status, though the image can be activated by re-uploading image file. So if we're going to remove 'killed' status from v2 intentionally, at least the image status should be rollbacked to 'queued' status. Please let me know if I missed anything. </font><br>
<br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 ERROR glance.api.v2.image_data [a330c019-a951-431e-be9c-fac04e9ebf68 669ebf06ffa24f0db7e9bce04ccc83a1 82d65617aa3244dba88a316234c5fd4e] Failed to upload image data due to internal error</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data Traceback (most recent call last):</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/api/v2/image_data.py", line 52, in upload</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data try:</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/domain/proxy.py", line 127, in set_data</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data self.base.set_data(data, size)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/domain/proxy.py", line 127, in set_data</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data self.base.set_data(data, size)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/notifier.py", line 220, in set_data</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data self.image.set_data(data, size)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/api/policy.py", line 237, in set_data</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data return self.image.set_data(*args, **kwargs)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/quota/__init__.py", line 274, in set_data</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data self.image.set_data(data, size=size)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/store/__init__.py", line 668, in set_data</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data self.image.image_id, utils.CooperativeReader(data), size)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/store/__init__.py", line 358, in add_to_backend</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/store/__init__.py", line 336, in store_add_to_backend</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data (location, size, checksum, metadata) = store.add(image_id, data, size)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" color="#000080" face="Monospace"><u>File "/opt/stack/glance/glance/store/filesystem.py", line 280, in add</u></font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data raise exceptions.get(e.errno, e)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data DisconnectionError: The client disconnected while sending the POST/PUT body (640303104 more bytes were expected)</font><br>
<font size="2" color="#FF0000" face="Monospace">2014-01-27 14:38:50.611 18964 TRACE glance.api.v2.image_data </font><font size="2" face="sans-serif"> </font><br>
<br>
<font size="2" face="sans-serif">Thanks & Best regards,<br>
Fei Long Wang (王飞龙)<br>
---------------------------------------------------------------------<br>
Tech Lead of Nitrogen (SME team)<br>
Cloud Solutions and OpenStack Development<br>
Tel: 8610-82450513 | T/L: 905-0513 <br>
Email: flwang@cn.ibm.com<br>
China Systems & Technology Laboratory in Beijing<br>
---------------------------------------------------------------------<br>
</font><br>
<br>
<img width="16" height="16" src="cid:1__=C7BBF6FEDFB72E378f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Mark Washenberger ---01/27/2014 09:39:12 AM---On Sun, Jan 26, 2014 at 4:37 PM, David Koo <kpublicmail"><font size="2" color="#424282" face="sans-serif">Mark Washenberger ---01/27/2014 09:39:12 AM---On Sun, Jan 26, 2014 at 4:37 PM, David Koo <kpublicmail@gmail.com> wrote: ></font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">Mark Washenberger <mark.washenberger@markwash.net></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org>, </font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">01/27/2014 09:39 AM</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [openstack-dev] [Glance] Is the 'killed' state ever set in v2?</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<br>
<font size="3" face="serif"><br>
</font><br>
<font size="3" face="serif">On Sun, Jan 26, 2014 at 4:37 PM, David Koo <</font><a href="mailto:kpublicmail@gmail.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>kpublicmail@gmail.com</u></font></a><font size="3" face="serif">> wrote:</font>
<ul style="padding-left: 9pt"><font size="3" face="serif"><br>
> Perhaps there is still a bug where an image is getting stuck in 'saving' or<br>
> some other state when a PUT fails?<br>
</font><br>
<font size="3" face="serif"> Yes, that's precisely the problem.</font></ul>
<br>
<font size="3" face="serif">We should definitely fix that, thanks for pointing it out!</font><br>
<font size="3" face="serif"> </font>
<ul style="padding-left: 9pt"><font size="3" face="serif"><br>
Of course, one could argue that that if an upload fails the user<br>
should be able to continue trying until the upload succeeds! But in that<br>
case the image status should probably be reset to "queued" rather than<br>
stay at "saving".</font></ul>
<br>
<font size="3" face="serif">That's exactly my argument so I'd like to see it go back to 'queued'. Nothing except the status has substantially changed during an upload that fails due to either the client or the underlying store, so it is easy to just revert the status and leave the image in a state where the user can reattempt the upload.</font><br>
<font size="3" face="serif"> </font>
<ul style="padding-left: 9pt"><font size="3" face="serif"><br>
But this makes me a little uneasy because our<br>
consistency/concurrency handling seems a little weak at the moment (am I<br>
right?). If we were to have a more complicated state machine then we<br>
would need much stronger consistency guarantees when there are<br>
simultaneous uploads in progress (where some fail and some succeed)!</font></ul>
<br>
<font size="3" face="serif">+1 to less complicated state machines :-)</font><br>
<br>
<font size="3" face="serif">This is part of what the current work on the import task is designed to accomplish. When you use import, an image effectively has only two states, 'active' and nonexistent.</font><br>
<br>
<font size="3" face="serif"> </font>
<ul style="padding-left: 9pt"><font size="3" face="serif"><br>
Is there any work on this (concurrency/consistency) front? I<br>
remember seeing some patches related to caching of simultaneous<br>
downloads of an image file where issues related to concurrent update of<br>
image metadata were addressed but IIRC it was -1ed because it reduced<br>
concurrency.</font></ul>
<br>
<font size="3" face="serif">I might be confused now or confused when I did that review, because I thought it was reducing download concurrency rather than upload concurrency. Are you talking about </font><a href="https://review.openstack.org/#/c/46479/"><font size="3" color="#0000FF" face="serif"><u>https://review.openstack.org/#/c/46479/</u></font></a><font size="3" face="serif"> ?</font><br>
<font size="3" face="serif"> </font>
<ul style="padding-left: 9pt"><font size="3" face="serif"><br>
So do we bring back the 'killed' state or should we shoot for a more<br>
complicated/powerful state machine?</font></ul>
<br>
<font size="3" face="serif">I think we can get by with trying to simplify the state that is involved and fixing any bugs with our state management. Is there a specific problem you're seeing with the </font><br>
<font size="3" face="serif"> </font>
<ul style="padding-left: 9pt"><font size="3" face="serif"><br>
--<br>
Koo</font><br>
<font size="3" face="serif"><br>
<br>
On Sun, Jan 26, 2014 at 06:36:36AM -0800, Mark Washenberger wrote:<br>
> It does not seem very ReSTful--or very usable, for that matter--for a<br>
> resource to be permanently modified when you a PUT fails. So I don't think<br>
> we need the 'killed' status. It was purposefully left out of v2 images,<br>
> which is not just a reskin of v1.<br>
><br>
> Perhaps there is still a bug where an image is getting stuck in 'saving' or<br>
> some other state when a PUT fails?<br>
><br>
><br>
> On Sun, Jan 26, 2014 at 5:10 AM, David Koo <</font><a href="mailto:kpublicmail@gmail.com"><font size="3" color="#0000FF" face="serif"><u>kpublicmail@gmail.com</u></font></a><font size="3" face="serif">> wrote:<br>
><br>
> ><br>
> > Hi Fei,<br>
> ><br>
> > Thanks for the confirmation.<br>
> ><br>
> > > I think you're right. The 'killed' status should be set in method<br>
> > upload()<br>
> > > if there is an upload failure, see<br>
> > ><br>
> > </font><a href="https://github.com/openstack/glance/blob/master/glance/common/utils.py#L244" target="_blank"><font size="3" color="#0000FF" face="serif"><u>https://github.com/openstack/glance/blob/master/glance/common/utils.py#L244</u></font></a><font size="3" face="serif"><br>
> ><br>
> > I think you meant:<br>
> ><br>
> ><br>
> > </font><a href="https://github.com/openstack/glance/blob/master/glance/api/v1/upload_utils.py#L244" target="_blank"><font size="3" color="#0000FF" face="serif"><u>https://github.com/openstack/glance/blob/master/glance/api/v1/upload_utils.py#L244</u></font></a><font size="3" face="serif"><br>
> ><br>
> > (the safe_kill() call) right?<br>
> ><br>
> > --<br>
> > Koo<br>
> ><br>
> ><br>
> > > ------------------ Original ------------------<br>
> > > From: "David Koo"<</font><a href="mailto:kpublicmail@gmail.com"><font size="3" color="#0000FF" face="serif"><u>kpublicmail@gmail.com</u></font></a><font size="3" face="serif">>;<br>
> > > Date: Jan 26, 2014<br>
> > > To: "OpenStack Development Mailing<br>
> > > List"<</font><a href="mailto:openstack-dev@lists.openstack.org"><font size="3" color="#0000FF" face="serif"><u>openstack-dev@lists.openstack.org</u></font></a><font size="3" face="serif">>;<br>
> > > Subject: [openstack-dev] [Glance] Is the 'killed' state ever set in v2?<br>
> > ><br>
> > > Hi All,<br>
> > ><br>
> > > While trying to work on a bug I was trying to simulate some image<br>
> > > download failures and found that apparently the 'killed' state is never<br>
> > > set using v2 APIs.<br>
> > ><br>
> > > If I understand correctly, a file upload goes to<br>
> > > api.v2.image_data.ImageDataController.upload and goes all the way to<br>
> > > store.ImageProxy.set_data which proceeds to write to the backend store.<br>
> > ><br>
> > > If the backend store raises an exception it is simply propagated all the<br>
> > > way up. The notifier re-encodes the exceptions (which is the bug I was<br>
> > > looking at) but doesn't do anything about the image status.<br>
> > ><br>
> > > Nowhere does the image status seem to get set to 'killed'.<br>
> > ><br>
> > > Before I log a bug I just wanted to confirm with everybody whether or<br>
> > > not I've missed out on something.<br>
> > ><br>
> > > Thanks.<br>
> > ><br>
> > > --<br>
> > > Koo<br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > </font><a href="mailto:OpenStack-dev@lists.openstack.org"><font size="3" color="#0000FF" face="serif"><u>OpenStack-dev@lists.openstack.org</u></font></a><font size="3" face="serif"><br>
> > </font><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><font size="3" color="#0000FF" face="serif"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</u></font></a><font size="3" face="serif"><br>
> ><br>
<br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> </font><a href="mailto:OpenStack-dev@lists.openstack.org"><font size="3" color="#0000FF" face="serif"><u>OpenStack-dev@lists.openstack.org</u></font></a><font size="3" face="serif"><br>
> </font><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><font size="3" color="#0000FF" face="serif"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</u></font></a><font size="3" face="serif"><br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list</font><font size="3" color="#0000FF" face="serif"><u><br>
</u></font><a href="mailto:OpenStack-dev@lists.openstack.org"><font size="3" color="#0000FF" face="serif"><u>OpenStack-dev@lists.openstack.org</u></font></a><font size="3" color="#0000FF" face="serif"><u><br>
</u></font><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><font size="3" color="#0000FF" face="serif"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</u></font></a></ul>
<tt><font size="2">_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org<br>
</font></tt><tt><font size="2"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></font></tt><tt><font size="2"><br>
</font></tt><br>
</body></html>