Hi neutrinos ! There is an Intermittent failure observed on gate jobs, I saw same failure of one of my patches, It was on keystone Phase for twice I checked. Thanks and Regards ! Manjeet Singh Bhatia From: John Villalovos [mailto:openstack.org at sodarock.com] Sent: Thursday, December 22, 2016 3:41 PM To: openstack-dev <openstack-dev at lists.openstack.org> Subject: [openstack-dev] [all] Intermittent gate failures across multiple projects Gate jobs seem to be intermittently broken across all of OpenStack. Bug filed: https://bugs.launchpad.net/openstack-gate/+bug/1652186 They appear during the keystone phase of devstack and look like this: http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787 2016-12-22 22:25:08.607<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_607> | 2016-12-22 22:25:08.606 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created domain default 2016-12-22 22:25:08.637<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_637> | 2016-12-22 22:25:08.637 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created project admin 2016-12-22 22:25:08.657<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_657> | 2016-12-22 22:25:08.656 31714 DEBUG passlib.registry [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] registered 'sha512_crypt' handler: <class 'passlib.handlers.sha2_crypt.sha512_crypt'> register_crypt_handler /usr/local/lib/python2.7/dist-packages/passlib/registry.py:294 2016-12-22 22:25:08.684<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_684> | 2016-12-22 22:25:08.684 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created user admin 2016-12-22 22:25:08.700<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_700> | 2016-12-22 22:25:08.699 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created role admin 2016-12-22 22:25:08.714<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_714> | 2016-12-22 22:25:08.714 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Granted admin on admin to user admin. 2016-12-22 22:25:08.731<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_731> | 2016-12-22 22:25:08.730 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created region RegionOne 2016-12-22 22:25:08.763<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_763> | 2016-12-22 22:25:08.763 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created admin endpoint http://15.184.67.84/identity_admin 2016-12-22 22:25:08.775<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_775> | 2016-12-22 22:25:08.774 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created internal endpoint http://15.184.67.84/identity 2016-12-22 22:25:08.787<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_787> | 2016-12-22 22:25:08.787 31714 INFO keystone.cmd.cli [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created public endpoint http://15.184.67.84/identity 2016-12-22 22:25:08.793<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_793> | 2016-12-22 22:25:08.793 31714 INFO keystone.assignment.core [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Creating the default role 9fe2ff9ee4384b1894a90878d3e92bab because it does not exist. 2016-12-22 22:25:08.923<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_923> | + ./stack.sh:main:1065 : create_keystone_accounts 2016-12-22 22:25:08.927<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_927> | + lib/keystone:create_keystone_accounts:371 : local admin_project 2016-12-22 22:25:08.931<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_931> | + lib/keystone:create_keystone_accounts:372 : openstack project show admin -f value -c id 2016-12-22 22:25:10.787<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787> | /opt/stack/new/devstack/lib/keystone: line 366: 31727 Illegal instruction openstack project show "admin" -f value -c id 2016-12-22 22:25:10.791<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_791> | + lib/keystone:create_keystone_accounts:372 : true 2016-12-22 22:25:10.796<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_796> | ++ lib/keystone:create_keystone_accounts:373 : openstack project show admin -f value -c id 2016-12-22 22:25:12.589<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_589> | + lib/keystone:create_keystone_accounts:373 : admin_project= 2016-12-22 22:25:12.593<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_593> | + lib/keystone:create_keystone_accounts:1 : exit_trap 2016-12-22 22:25:12.596<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_596> | + ./stack.sh:exit_trap:487 : local r=132 2016-12-22 22:25:12.602<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_602> | ++ ./stack.sh:exit_trap:488 : jobs -p 2016-12-22 22:25:12.606<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_606> | + ./stack.sh:exit_trap:488 : jobs=31048 2016-12-22 22:25:12.610<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_610> | + ./stack.sh:exit_trap:491 : [[ -n 31048 ]] 2016-12-22 22:25:12.614<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_614> | + ./stack.sh:exit_trap:491 : [[ -n /opt/stack/new/devstacklog.txt.2016-12-22-221622 ]] 2016-12-22 22:25:12.617<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_617> | + ./stack.sh:exit_trap:491 : [[ True == \T\r\u\e ]] 2016-12-22 22:25:12.621<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621> | + ./stack.sh:exit_trap:492 : echo 'exit_trap: cleaning up child processes' 2016-12-22 22:25:12.621<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621> | exit_trap: cleaning up child processes 2016-12-22 22:25:12.625<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625> | + ./stack.sh:exit_trap:493 : kill 31048 2016-12-22 22:25:12.625<http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625> | ./stack.sh: line 493: kill: (31048) - No such process The "Illegal instruction" message is seen in this log because of this test patch: https://review.openstack.org/414341 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161223/167ebdfa/attachment.html>