<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.m7862380079793542757gmail-none
{mso-style-name:m_7862380079793542757gmail-none;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi neutrinos !<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">There is an Intermittent failure observed on gate jobs,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I saw same failure of one of my patches, It was on keystone<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Phase for twice I checked.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks and Regards !<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Manjeet Singh Bhatia<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> John Villalovos [mailto:openstack.org@sodarock.com]
<br>
<b>Sent:</b> Thursday, December 22, 2016 3:41 PM<br>
<b>To:</b> openstack-dev <openstack-dev@lists.openstack.org><br>
<b>Subject:</b> [openstack-dev] [all] Intermittent gate failures across multiple projects<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Gate jobs seem to be intermittently broken across all of OpenStack.<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Bug filed:<br>
<a href="https://bugs.launchpad.net/openstack-gate/+bug/1652186" target="_blank">https://bugs.launchpad.net/openstack-gate/+bug/1652186</a><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">They appear during the keystone phase of devstack and look like this:<br>
<br>
<a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787" target="_blank">http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787</a><o:p></o:p></p>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_607" target="_blank">2016-12-22 22:25:08.607</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.606 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created domain default<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_637" target="_blank">2016-12-22 22:25:08.637</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.637 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created project admin<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_657" target="_blank">2016-12-22 22:25:08.657</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.656 31714 DEBUG passlib.registry [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] registered 'sha512_crypt' handler: <class 'passlib.handlers.sha2_crypt.sha512_crypt'> register_crypt_handler /usr/local/lib/python2.7/dist-packages/passlib/registry.py:294<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_684" target="_blank">2016-12-22 22:25:08.684</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.684 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created user admin<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_700" target="_blank">2016-12-22 22:25:08.700</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.699 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created role admin<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_714" target="_blank">2016-12-22 22:25:08.714</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.714 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Granted admin on admin to user admin.<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_731" target="_blank">2016-12-22 22:25:08.731</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.730 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created region RegionOne<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_763" target="_blank">2016-12-22 22:25:08.763</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.763 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created admin endpoint </span><a href="http://15.184.67.84/identity_admin" target="_blank">http://15.184.67.84/identity_admin</a><span class="m7862380079793542757gmail-none"><o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_775" target="_blank">2016-12-22 22:25:08.775</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.774 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created internal endpoint </span><a href="http://15.184.67.84/identity" target="_blank">http://15.184.67.84/identity</a><span class="m7862380079793542757gmail-none"><o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_787" target="_blank">2016-12-22 22:25:08.787</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.787 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created public endpoint </span><a href="http://15.184.67.84/identity" target="_blank">http://15.184.67.84/identity</a><span class="m7862380079793542757gmail-none"><o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_793" target="_blank">2016-12-22 22:25:08.793</a><span class="m7862380079793542757gmail-none"> | 2016-12-22 22:25:08.793 31714 INFO keystone.assignment.core [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Creating the default role 9fe2ff9ee4384b1894a90878d3e92bab because it does not exist.<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_923" target="_blank">2016-12-22 22:25:08.923</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:main:1065 : create_keystone_accounts<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_927" target="_blank">2016-12-22 22:25:08.927</a><span class="m7862380079793542757gmail-none"> | + lib/keystone:create_keystone_accounts:371 : local admin_project<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_931" target="_blank">2016-12-22 22:25:08.931</a><span class="m7862380079793542757gmail-none"> | + lib/keystone:create_keystone_accounts:372 : openstack project show admin -f value -c id<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787" target="_blank">2016-12-22 22:25:10.787</a><span class="m7862380079793542757gmail-none"> | /opt/stack/new/devstack/lib/keystone: line 366: 31727 Illegal instruction openstack project show "admin" -f value -c id<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_791" target="_blank">2016-12-22 22:25:10.791</a><span class="m7862380079793542757gmail-none"> | + lib/keystone:create_keystone_accounts:372 : true<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_796" target="_blank">2016-12-22 22:25:10.796</a><span class="m7862380079793542757gmail-none"> | ++ lib/keystone:create_keystone_accounts:373 : openstack project show admin -f value -c id<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_589" target="_blank">2016-12-22 22:25:12.589</a><span class="m7862380079793542757gmail-none"> | + lib/keystone:create_keystone_accounts:373 : admin_project=<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_593" target="_blank">2016-12-22 22:25:12.593</a><span class="m7862380079793542757gmail-none"> | + lib/keystone:create_keystone_accounts:1 : exit_trap<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_596" target="_blank">2016-12-22 22:25:12.596</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:exit_trap:487 : local r=132<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_602" target="_blank">2016-12-22 22:25:12.602</a><span class="m7862380079793542757gmail-none"> | ++ ./stack.sh:exit_trap:488 : jobs -p<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_606" target="_blank">2016-12-22 22:25:12.606</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:exit_trap:488 : jobs=31048<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_610" target="_blank">2016-12-22 22:25:12.610</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:exit_trap:491 : [[ -n 31048 ]]<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_614" target="_blank">2016-12-22 22:25:12.614</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:exit_trap:491 : [[ -n /opt/stack/new/devstacklog.txt.2016-12-22-221622 ]]<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_617" target="_blank">2016-12-22 22:25:12.617</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:exit_trap:491 : [[ True == \T\r\u\e ]]<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621" target="_blank">2016-12-22 22:25:12.621</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:exit_trap:492 : echo 'exit_trap: cleaning up child processes'<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621" target="_blank">2016-12-22 22:25:12.621</a><span class="m7862380079793542757gmail-none"> | exit_trap: cleaning up child processes<o:p></o:p></span></pre>
<pre><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625" target="_blank">2016-12-22 22:25:12.625</a><span class="m7862380079793542757gmail-none"> | + ./stack.sh:exit_trap:493 : kill 31048<o:p></o:p></span></pre>
<pre style="margin-bottom:12.0pt"><a href="http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625" target="_blank">2016-12-22 22:25:12.625</a><a name="m_7862380079793542757__2016-12-22_22_25_"></a><span class="m7862380079793542757gmail-none"> | ./stack.sh: line 493: kill: (31048) - No such process</span><o:p></o:p></pre>
<pre><span class="m7862380079793542757gmail-none">The "Illegal instruction" message is seen in this log because of this test patch:</span><br><a href="https://review.openstack.org/414341" target="_blank">https://review.openstack.org/414341</a><span class="m7862380079793542757gmail-none"> </span><o:p></o:p></pre>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>