[openstack-dev] [all] Intermittent gate failures across multiple projects

Matt Riedemann mriedem at linux.vnet.ibm.com
Fri Dec 23 00:02:33 UTC 2016


On 12/22/2016 5:40 PM, John Villalovos wrote:
> Gate jobs seem to be intermittently broken across all of OpenStack.
>
> Bug filed:
> https://bugs.launchpad.net/openstack-gate/+bug/1652186
> <https://bugs.launchpad.net/openstack-gate/+bug/1652186>
>
> They appear during the keystone phase of devstack and look like this:
>
> http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787>
>
> 2016-12-22 22:25:08.607
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_607>
> | 2016-12-22 22:25:08.606 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created domain
> default 2016-12-22 22:25:08.637
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_637>
> | 2016-12-22 22:25:08.637 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created project
> admin 2016-12-22 22:25:08.657
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_657>
> | 2016-12-22 22:25:08.656 31714 DEBUG passlib.registry
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] registered
> 'sha512_crypt' handler: <class
> 'passlib.handlers.sha2_crypt.sha512_crypt'> register_crypt_handler
> /usr/local/lib/python2.7/dist-packages/passlib/registry.py:294
> 2016-12-22 22:25:08.684
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_684>
> | 2016-12-22 22:25:08.684 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created user admin
> 2016-12-22 22:25:08.700
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_700>
> | 2016-12-22 22:25:08.699 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created role admin
> 2016-12-22 22:25:08.714
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_714>
> | 2016-12-22 22:25:08.714 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Granted admin on
> admin to user admin. 2016-12-22 22:25:08.731
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_731>
> | 2016-12-22 22:25:08.730 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created region
> RegionOne 2016-12-22 22:25:08.763
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_763>
> | 2016-12-22 22:25:08.763 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created admin
> endpoint http://15.184.67.84/identity_admin
> <http://15.184.67.84/identity_admin> 2016-12-22 22:25:08.775
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_775>
> | 2016-12-22 22:25:08.774 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created internal
> endpoint http://15.184.67.84/identity 2016-12-22 22:25:08.787
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_787>
> | 2016-12-22 22:25:08.787 31714 INFO keystone.cmd.cli
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Created public
> endpoint http://15.184.67.84/identity 2016-12-22 22:25:08.793
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_793>
> | 2016-12-22 22:25:08.793 31714 INFO keystone.assignment.core
> [req-f336127d-0628-4b9d-bcc7-a87863c3127e - - - - -] Creating the
> default role 9fe2ff9ee4384b1894a90878d3e92bab because it does not exist.
> 2016-12-22 22:25:08.923
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_923>
> | + ./stack.sh:main:1065 : create_keystone_accounts 2016-12-22
> 22:25:08.927
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_927>
> | + lib/keystone:create_keystone_accounts:371 : local admin_project
> 2016-12-22 22:25:08.931
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_08_931>
> | + lib/keystone:create_keystone_accounts:372 : openstack project show
> admin -f value -c id 2016-12-22 22:25:10.787
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_787>
> | /opt/stack/new/devstack/lib/keystone: line 366: 31727 Illegal
> instruction openstack project show "admin" -f value -c id 2016-12-22
> 22:25:10.791
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_791>
> | + lib/keystone:create_keystone_accounts:372 : true 2016-12-22
> 22:25:10.796
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_10_796>
> | ++ lib/keystone:create_keystone_accounts:373 : openstack project show
> admin -f value -c id 2016-12-22 22:25:12.589
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_589>
> | + lib/keystone:create_keystone_accounts:373 : admin_project=
> 2016-12-22 22:25:12.593
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_593>
> | + lib/keystone:create_keystone_accounts:1 : exit_trap 2016-12-22
> 22:25:12.596
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_596>
> | + ./stack.sh:exit_trap:487 : local r=132 2016-12-22 22:25:12.602
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_602>
> | ++ ./stack.sh:exit_trap:488 : jobs -p 2016-12-22 22:25:12.606
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_606>
> | + ./stack.sh:exit_trap:488 : jobs=31048 2016-12-22 22:25:12.610
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_610>
> | + ./stack.sh:exit_trap:491 : [[ -n 31048 ]] 2016-12-22 22:25:12.614
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_614>
> | + ./stack.sh:exit_trap:491 : [[ -n
> /opt/stack/new/devstacklog.txt.2016-12-22-221622 ]] 2016-12-22
> 22:25:12.617
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_617>
> | + ./stack.sh:exit_trap:491 : [[ True == \T\r\u\e ]] 2016-12-22
> 22:25:12.621
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621>
> | + ./stack.sh:exit_trap:492 : echo 'exit_trap: cleaning up child
> processes' 2016-12-22 22:25:12.621
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_621>
> | exit_trap: cleaning up child processes 2016-12-22 22:25:12.625
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625>
> | + ./stack.sh:exit_trap:493 : kill 31048 2016-12-22 22:25:12.625
> <http://logs.openstack.org/39/414339/2/check/gate-tempest-dsvm-ironic-ipa-partition-pxe_snmp-tinyipa-ubuntu-xenial-nv/1d5e63d/logs/devstacklog.txt.gz#_2016-12-22_22_25_12_625>
> | ./stack.sh: line 493: kill: (31048) - No such process
>
> The "Illegal instruction" message is seen in this log because of this
> test patch:
> https://review.openstack.org/414341 <https://review.openstack.org/414341>
>
>
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

It's also unit test jobs:

http://logs.openstack.org/05/378105/14/check/gate-cinder-python34-db/01d4149/console.html#_2016-12-22_22_18_35_037491

-- 

Thanks,

Matt Riedemann




More information about the OpenStack-dev mailing list