[openstack-dev] FW: [neutron][all] Intermittent gate failures across multiple projects

Bhatia, Manjeet S manjeet.s.bhatia at intel.com
Fri Dec 23 02:48:58 UTC 2016


Hi neutrinos !

There is an Intermittent failure observed on gate jobs,
I saw same failure of one of my patches, It was on keystone
Phase for twice I checked.

Thanks and Regards !
Manjeet Singh Bhatia

From: John Villalovos [mailto:openstack.org at sodarock.com]
Sent: Thursday, December 22, 2016 3:41 PM
To: openstack-dev <openstack-dev at lists.openstack.org>
Subject: [openstack-dev] [all] Intermittent gate failures across multiple projects

Gate jobs seem to be intermittently broken across all of OpenStack.

Bug filed:
https://bugs.launchpad.net/openstack-gate/+bug/1652186
They appear during the keystone phase of devstack and look like this:

http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787

2016-12-22 22:25:08.607<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_607> | 2016-12-22 22:25:08.606 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created domain default

2016-12-22 22:25:08.637<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_637> | 2016-12-22 22:25:08.637 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created project admin

2016-12-22 22:25:08.657<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_657> | 2016-12-22 22:25:08.656 31714 DEBUG passlib.registry [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] registered 'sha512_crypt' handler: <class 'passlib.handlers.sha2_crypt.sha512_crypt'> register_crypt_handler /usr/local/lib/python2.7/dist-packages/passlib/registry.py:294

2016-12-22 22:25:08.684<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_684> | 2016-12-22 22:25:08.684 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created user admin

2016-12-22 22:25:08.700<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_700> | 2016-12-22 22:25:08.699 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created role admin

2016-12-22 22:25:08.714<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_714> | 2016-12-22 22:25:08.714 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Granted admin on admin to user admin.

2016-12-22 22:25:08.731<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_731> | 2016-12-22 22:25:08.730 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created region RegionOne

2016-12-22 22:25:08.763<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_763> | 2016-12-22 22:25:08.763 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created admin endpoint http://15.184.67.84/identity_admin

2016-12-22 22:25:08.775<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_775> | 2016-12-22 22:25:08.774 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created internal endpoint http://15.184.67.84/identity

2016-12-22 22:25:08.787<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_787> | 2016-12-22 22:25:08.787 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created public endpoint http://15.184.67.84/identity

2016-12-22 22:25:08.793<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_793> | 2016-12-22 22:25:08.793 31714 INFO keystone.assignment.core [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Creating the default role 9fe2ff9ee4384b1894a90878d3e92bab because it does not exist.

2016-12-22 22:25:08.923<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_923> | + ./stack.sh:main:1065                     :   create_keystone_accounts

2016-12-22 22:25:08.927<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_927> | + lib/keystone:create_keystone_accounts:371 :   local admin_project

2016-12-22 22:25:08.931<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_931> | + lib/keystone:create_keystone_accounts:372 :   openstack project show admin -f value -c id

2016-12-22 22:25:10.787<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787> | /opt/stack/new/devstack/lib/keystone: line 366: 31727 Illegal instruction     openstack project show "admin" -f value -c id

2016-12-22 22:25:10.791<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_791> | + lib/keystone:create_keystone_accounts:372 :   true

2016-12-22 22:25:10.796<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_796> | ++ lib/keystone:create_keystone_accounts:373 :   openstack project show admin -f value -c id

2016-12-22 22:25:12.589<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_589> | + lib/keystone:create_keystone_accounts:373 :   admin_project=

2016-12-22 22:25:12.593<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_593> | + lib/keystone:create_keystone_accounts:1  :   exit_trap

2016-12-22 22:25:12.596<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_596> | + ./stack.sh:exit_trap:487                 :   local r=132

2016-12-22 22:25:12.602<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_602> | ++ ./stack.sh:exit_trap:488                 :   jobs -p

2016-12-22 22:25:12.606<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_606> | + ./stack.sh:exit_trap:488                 :   jobs=31048

2016-12-22 22:25:12.610<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_610> | + ./stack.sh:exit_trap:491                 :   [[ -n 31048 ]]

2016-12-22 22:25:12.614<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_614> | + ./stack.sh:exit_trap:491                 :   [[ -n /opt/stack/new/devstacklog.txt.2016-12-22-221622 ]]

2016-12-22 22:25:12.617<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_617> | + ./stack.sh:exit_trap:491                 :   [[ True == \T\r\u\e ]]

2016-12-22 22:25:12.621<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621> | + ./stack.sh:exit_trap:492                 :   echo 'exit_trap: cleaning up child processes'

2016-12-22 22:25:12.621<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621> | exit_trap: cleaning up child processes

2016-12-22 22:25:12.625<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625> | + ./stack.sh:exit_trap:493                 :   kill 31048

2016-12-22 22:25:12.625<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625> | ./stack.sh: line 493: kill: (31048) - No such process

The "Illegal instruction" message is seen in this log because of this test patch:
https://review.openstack.org/414341



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161223/167ebdfa/attachment.html>


More information about the OpenStack-dev mailing list