On 12/22/2016 5:40 PM, John Villalovos wrote: > Gate jobs seem to be intermittently broken across all of OpenStack. > > Bug filed: > https://bugs.launchpad.net/openstack-gate/+bug/1652186 > <https://bugs.launchpad.net/openstack-gate/+bug/1652186> > > They appear during the keystone phase of devstack and look like this: > > http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787> > > 2016-12-22 22:25:08.607 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_607> > | 2016-12-22 22:25:08.606 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created domain > default 2016-12-22 22:25:08.637 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_637> > | 2016-12-22 22:25:08.637 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created project > admin 2016-12-22 22:25:08.657 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_657> > | 2016-12-22 22:25:08.656 31714 DEBUG passlib.registry > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] registered > 'sha512_crypt' handler: <class > 'passlib.handlers.sha2_crypt.sha512_crypt'> register_crypt_handler > /usr/local/lib/python2.7/dist-packages/passlib/registry.py:294 > 2016-12-22 22:25:08.684 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_684> > | 2016-12-22 22:25:08.684 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created user admin > 2016-12-22 22:25:08.700 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_700> > | 2016-12-22 22:25:08.699 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created role admin > 2016-12-22 22:25:08.714 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_714> > | 2016-12-22 22:25:08.714 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Granted admin on > admin to user admin. 2016-12-22 22:25:08.731 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_731> > | 2016-12-22 22:25:08.730 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created region > RegionOne 2016-12-22 22:25:08.763 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_763> > | 2016-12-22 22:25:08.763 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created admin > endpoint http://15.184.67.84/identity_admin > <http://15.184.67.84/identity_admin> 2016-12-22 22:25:08.775 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_775> > | 2016-12-22 22:25:08.774 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created internal > endpoint http://15.184.67.84/identity 2016-12-22 22:25:08.787 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_787> > | 2016-12-22 22:25:08.787 31714 INFO keystone.cmd.cli > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created public > endpoint http://15.184.67.84/identity 2016-12-22 22:25:08.793 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_793> > | 2016-12-22 22:25:08.793 31714 INFO keystone.assignment.core > [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Creating the > default role 9fe2ff9ee4384b1894a90878d3e92bab because it does not exist. > 2016-12-22 22:25:08.923 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_923> > | + ./stack.sh:main:1065 : create_keystone_accounts 2016-12-22 > 22:25:08.927 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_927> > | + lib/keystone:create_keystone_accounts:371 : local admin_project > 2016-12-22 22:25:08.931 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_931> > | + lib/keystone:create_keystone_accounts:372 : openstack project show > admin -f value -c id 2016-12-22 22:25:10.787 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787> > | /opt/stack/new/devstack/lib/keystone: line 366: 31727 Illegal > instruction openstack project show "admin" -f value -c id 2016-12-22 > 22:25:10.791 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_791> > | + lib/keystone:create_keystone_accounts:372 : true 2016-12-22 > 22:25:10.796 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_796> > | ++ lib/keystone:create_keystone_accounts:373 : openstack project show > admin -f value -c id 2016-12-22 22:25:12.589 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_589> > | + lib/keystone:create_keystone_accounts:373 : admin_project= > 2016-12-22 22:25:12.593 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_593> > | + lib/keystone:create_keystone_accounts:1 : exit_trap 2016-12-22 > 22:25:12.596 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_596> > | + ./stack.sh:exit_trap:487 : local r=132 2016-12-22 22:25:12.602 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_602> > | ++ ./stack.sh:exit_trap:488 : jobs -p 2016-12-22 22:25:12.606 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_606> > | + ./stack.sh:exit_trap:488 : jobs=31048 2016-12-22 22:25:12.610 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_610> > | + ./stack.sh:exit_trap:491 : [[ -n 31048 ]] 2016-12-22 22:25:12.614 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_614> > | + ./stack.sh:exit_trap:491 : [[ -n > /opt/stack/new/devstacklog.txt.2016-12-22-221622 ]] 2016-12-22 > 22:25:12.617 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_617> > | + ./stack.sh:exit_trap:491 : [[ True == \T\r\u\e ]] 2016-12-22 > 22:25:12.621 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621> > | + ./stack.sh:exit_trap:492 : echo 'exit_trap: cleaning up child > processes' 2016-12-22 22:25:12.621 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621> > | exit_trap: cleaning up child processes 2016-12-22 22:25:12.625 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625> > | + ./stack.sh:exit_trap:493 : kill 31048 2016-12-22 22:25:12.625 > <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625> > | ./stack.sh: line 493: kill: (31048) - No such process > > The "Illegal instruction" message is seen in this log because of this > test patch: > https://review.openstack.org/414341 <https://review.openstack.org/414341> > > > > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > It's also unit test jobs: http://logs.openstack.org/05/378105/14/check/gate-cinder-python34-db/01d4149/console.html#_2016-12-22_22_18_35_037491 -- Thanks, Matt Riedemann