<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.gmail-
{mso-style-name:gmail-;}
span.gmail-hoenzb
{mso-style-name:gmail-hoenzb;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Initial finding:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">This is something related to uwsgi socket timeout. Default value for socket-timeout is 30 in /etc/glance/glance-uwsgi.ini file.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">For testing purpose I have tried to create 10 GB image using create command:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">$ time glance --debug image-create --name dsl --file gentoo_root.img --disk-format iso --container-format bare<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">real 2m48.539s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">user 0m4.076s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">sys 0m10.012s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It has failed with “502 bad gateway” after 3 minutes. Then I have increased this timeout to 60 and restarted glance-api service and ran above command again.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">$ time glance --debug image-create --name dsl --file gentoo_root.img --disk-format iso --container-format bare<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">After this I was able to create image without any issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So another workaround is to set socket-timeout to higher value in /etc/glance/glance-uwsgi.ini file.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Abhishek Kekane<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Kekane, Abhishek [mailto:Abhishek.Kekane@nttdata.com]
<br>
<b>Sent:</b> Thursday, July 13, 2017 2:45 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Someone has reported this issue in glance recently.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Please refer,
<a href="https://bugs.launchpad.net/glance/+bug/1703856">https://bugs.launchpad.net/glance/+bug/1703856</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think it is same kind of failure which Andrey was mentioning.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As mentioned in the bug temporary solution is to enable the v1 api using the v1 parameter, --location instead of using --file. But this will not be applicable
to rally scenarios I guess.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I will spend some time to understand the failure cause.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Abhishek<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Andrey Kurilin [<a href="mailto:andr.kurilin@gmail.com">mailto:andr.kurilin@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, July 13, 2017 2:25 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Flavio,<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">2017-07-13 11:16 GMT+03:00 Flavio Percoco <<a href="mailto:flavio@redhat.com" target="_blank">flavio@redhat.com</a>>:<o:p></o:p></p>
<p class="MsoNormal"><span class="gmail-">On 13/07/17 00:56 -0700, Boris Pavlovic wrote:<o:p></o:p></span></p>
<p class="MsoNormal">Hi stackers,<br>
<br>
<br>
Unfortunately what was discussed in other thread (situation in glance is<br>
critical) happened.<br>
Glance stopped working and Rally team is forced to disable checking of it<br>
in Rally gates.<br>
<br>
P.S. Seems like this patch is casing the problems:<br>
<a href="https://github.com/openstack-dev/devstack/commit/1fa653635781cd975a1031e212b35b6c38196ba4" target="_blank">https://github.com/openstack-dev/devstack/commit/1fa653635781cd975a1031e212b35b6c38196ba4</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Hey Boris,<br>
<br>
Has this been brought up to the Glance team? Or is this email meant to do that?<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
Yes<br>
<br>
<a href="http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T07:44:13">http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T07:44:13</a><br>
<a href="http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T22:12:47">http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T22:12:47</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">FWIW, the switch to uwsgi is a community goal and not so much a Glance thing.<br>
Would you mind elaborating on what exactly is failing and how the glance team<br>
can help?<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal">I understand that switching to uwsgi is a community goal, but we didn't have any problems with Glance for years until now.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
As from IRC logs:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">> HTTPBadGateway: 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at 158.69.73.26 Port 80 (HTTP 502)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Trace: <br>
<br>
<a href="http://paste.openstack.org/show/615234/">http://paste.openstack.org/show/615234/</a>
<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Flavio<br>
<br>
--<br>
@flaper87<span style="color:#888888"><br>
<span class="gmail-hoenzb">Flavio Percoco</span><br>
</span><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Best regards,<br>
Andrey Kurilin.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
______________________________________________________________________<br>
Disclaimer: This email and any attachments are sent in strictest confidence<br>
for the sole use of the addressee and may contain legally privileged,<br>
confidential, and proprietary data. If you are not the intended recipient,<br>
please advise the sender by replying promptly to this email and then delete<br>
and destroy this email and any attachments without any further use, copying<br>
or forwarding.<o:p></o:p></p>
</div>
<br clear="both">
______________________________________________________________________<BR>
Disclaimer: This email and any attachments are sent in strictest confidence<BR>
for the sole use of the addressee and may contain legally privileged,<BR>
confidential, and proprietary data. If you are not the intended recipient,<BR>
please advise the sender by replying promptly to this email and then delete<BR>
and destroy this email and any attachments without any further use, copying<BR>
or forwarding.<BR>
</body>
</html>