[OpenStack-Infra] [openstack-infra][External CI][dsvm-tempest-full] Jenkins job failing at "Running devstack"

Eduard Matei eduard.matei at cloudfounders.com
Thu Sep 11 08:54:51 UTC 2014 

Hi Jay,
Thx for the reply.
I had a look at nodepool but couldn't figure how to integrate it with my
setup.
The slave is already a VM connected to the master.

I looked a little more through the code, and removed a "> /dev/null" from
 /opt/stack/new/devstack-gate/devstack-vm-gate.sh so now i get more output
in the jenkins job.
It seems the job fails inside "stack.sh" but "/dev/null" was hiding all the
output.

The error is :
... snip ...

*11:43:36* 2014-09-11 08:43:36.736 | ++ openstack --os-token
10b868ee719d4525a5f02712fc4dedcf --os-url http://127.0.0.1:9292 image
create cirros-0.3.2-x86_64-uec-kernel --public --container-format aki
--disk-format aki

*11:43:40* 2014-09-11 08:43:40.973 | ERROR: glanceclient.common.http
Request returned failure status 401.

*11:43:44* 2014-09-11 08:43:44.678 | ERROR: glanceclient.common.http
Request returned failure status 401.*11:43:44* 2014-09-11 08:43:44.678
| ERROR: openstack <html>*11:43:44* 2014-09-11 08:43:44.678 |
<head>*11:43:44* 2014-09-11 08:43:44.678 |   <title>401
Unauthorized</title>*11:43:44* 2014-09-11 08:43:44.678 |
</head>*11:43:44* 2014-09-11 08:43:44.678 |  <body>*11:43:44*
2014-09-11 08:43:44.678 |   <h1>401 Unauthorized</h1>*11:43:44*
2014-09-11 08:43:44.678 |   This server could not verify that you are
authorized to access the document you requested. Either you supplied
the wrong credentials (e.g., bad password), or your browser does not
understand how to supply the credentials required.<br /><br
/>*11:43:44* 2014-09-11 08:43:44.678 | *11:43:44* 2014-09-11
08:43:44.678 |  </body>*11:43:44* 2014-09-11 08:43:44.678 | </html>
(HTTP 401)*11:43:44* 2014-09-11 08:43:44.701 | + kernel_id=*11:43:44*
2014-09-11 08:43:44.701 | + '[' -n
/opt/stack/new/devstack/files/images/cirros-0.3.2-x86_64-uec/cirros-0.3.2-x86_64-initrd
']'*11:43:44* 2014-09-11 08:43:44.702 | ++ get_field 2*11:43:44*
2014-09-11 08:43:44.703 | ++ local data field*11:43:44* 2014-09-11
08:43:44.703 | ++ grep ' id '*11:43:44* 2014-09-11 08:43:44.703 | ++
read data*11:43:44* 2014-09-11 08:43:44.705 | ++ openstack --os-token
10b868ee719d4525a5f02712fc4dedcf --os-url http://127.0.0.1:9292 image
create cirros-0.3.2-x86_64-uec-ramdisk --public --container-format ari
--disk-format ari

*11:43:48* 2014-09-11 08:43:48.962 | ERROR: glanceclient.common.http
Request returned failure status 401.

*11:43:52* 2014-09-11 08:43:52.646 | ERROR: glanceclient.common.http
Request returned failure status 401.*11:43:52* 2014-09-11 08:43:52.646
| ERROR: openstack <html>*11:43:52* 2014-09-11 08:43:52.646 |
<head>*11:43:52* 2014-09-11 08:43:52.646 |   <title>401
Unauthorized</title>*11:43:52* 2014-09-11 08:43:52.646 |
</head>*11:43:52* 2014-09-11 08:43:52.647 |  <body>*11:43:52*
2014-09-11 08:43:52.647 |   <h1>401 Unauthorized</h1>*11:43:52*
2014-09-11 08:43:52.647 |   This server could not verify that you are
authorized to access the document you requested. Either you supplied
the wrong credentials (e.g., bad password), or your browser does not
understand how to supply the credentials required.<br /><br
/>*11:43:52* 2014-09-11 08:43:52.647 | *11:43:52* 2014-09-11
08:43:52.647 |  </body>*11:43:52* 2014-09-11 08:43:52.647 | </html>
(HTTP 401)*11:43:52* 2014-09-11 08:43:52.666 | + ramdisk_id=*11:43:52*
2014-09-11 08:43:52.667 | + openstack --os-token
10b868ee719d4525a5f02712fc4dedcf --os-url http://127.0.0.1:9292 image
create cirros-0.3.2-x86_64-uec --public --container-format ami
--disk-format ami

*11:43:56* 2014-09-11 08:43:56.861 | ERROR: glanceclient.common.http
Request returned failure status 401.

*11:44:00* 2014-09-11 08:44:00.562 | ERROR: glanceclient.common.http
Request returned failure status 401.*11:44:00* 2014-09-11 08:44:00.562
| ERROR: openstack <html>*11:44:00* 2014-09-11 08:44:00.562 |
<head>*11:44:00* 2014-09-11 08:44:00.562 |   <title>401
Unauthorized</title>*11:44:00* 2014-09-11 08:44:00.562 |
</head>*11:44:00* 2014-09-11 08:44:00.562 |  <body>*11:44:00*
2014-09-11 08:44:00.562 |   <h1>401 Unauthorized</h1>*11:44:00*
2014-09-11 08:44:00.562 |   This server could not verify that you are
authorized to access the document you requested. Either you supplied
the wrong credentials (e.g., bad password), or your browser does not
understand how to supply the credentials required.<br /><br
/>*11:44:00* 2014-09-11 08:44:00.562 | *11:44:00* 2014-09-11
08:44:00.562 |  </body>*11:44:00* 2014-09-11 08:44:00.562 | </html>
(HTTP 401)*11:44:00* 2014-09-11 08:44:00.585 | + exit_trap*11:44:00*
2014-09-11 08:44:00.585 | + local r=1*11:44:00* 2014-09-11
08:44:00.585 | ++ jobs -p*11:44:00* 2014-09-11 08:44:00.586 | +
jobs='29192*11:44:00* 2014-09-11 08:44:00.586 | 29433*11:44:00*
2014-09-11 08:44:00.586 | 29441*11:44:00* 2014-09-11 08:44:00.586 |
31634*11:44:00* 2014-09-11 08:44:00.586 | 833*11:44:00* 2014-09-11
08:44:00.586 | 841*11:44:00* 2014-09-11 08:44:00.586 | 849*11:44:00*
2014-09-11 08:44:00.586 | 857*11:44:00* 2014-09-11 08:44:00.586 |
868*11:44:00* 2014-09-11 08:44:00.586 | 876'*11:44:00* 2014-09-11
08:44:00.586 | + [[ -n 29192*11:44:00* 2014-09-11 08:44:00.586 |
29433*11:44:00* 2014-09-11 08:44:00.586 | 29441*11:44:00* 2014-09-11
08:44:00.586 | 31634*11:44:00* 2014-09-11 08:44:00.586 | 833*11:44:00*
2014-09-11 08:44:00.586 | 841*11:44:00* 2014-09-11 08:44:00.586 |
849*11:44:00* 2014-09-11 08:44:00.586 | 857*11:44:00* 2014-09-11
08:44:00.586 | 868*11:44:00* 2014-09-11 08:44:00.586 | 876
]]*11:44:00* 2014-09-11 08:44:00.586 | + [[ -n
/opt/stack/new/devstacklog.txt.2014-09-11-113844 ]]*11:44:00*
2014-09-11 08:44:00.586 | + [[ True == \T\r\u\e ]]*11:44:00*
2014-09-11 08:44:00.586 | + echo 'exit_trap: cleaning up child
processes'*11:44:00* 2014-09-11 08:44:00.586 | exit_trap: cleaning up
child processes*11:44:00* 2014-09-11 08:44:00.586 | + kill 29192 29433
29441 31634 833 841 849 857 868 876*11:44:00* 2014-09-11 08:44:00.586
| ./stack.sh: line 633: kill: (29441) - Operation not
permitted*11:44:00* 2014-09-11 08:44:00.586 | + kill_spinner*11:44:00*
2014-09-11 08:44:00.587 | + '[' '!' -z '' ']'*11:44:00* 2014-09-11
08:44:00.587 | + [[ 1 -ne 0 ]]*11:44:00* 2014-09-11 08:44:00.587 | +
echo 'Error on exit'*11:44:00* 2014-09-11 08:44:00.587 | Error on
exit*11:44:00* 2014-09-11 08:44:00.587 | + [[ -z /opt/stack/new
]]*11:44:00* 2014-09-11 08:44:00.587 | + ./tools/worlddump.py -d
/opt/stack/new*11:44:00* 2014-09-11 08:44:00.617 | World dumping...
see /opt/stack/new/worlddump-2014-09-11-084400.txt for
details*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
29192 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
29433 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
31634 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
833 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
841 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
849 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
857 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
868 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | ./stack.sh: line 626:
876 Terminated              _old_run_process "$service"
"$command"*11:44:00* 2014-09-11 08:44:00.640 | + exit 1*11:44:00*
ERROR: the main setup script run by this job failed - exit code:
1*11:44:00*     please look at the relevant log files to determine the
root cause

... snip ...


If i run that command in the shell (openstack --os-token ...) i get
the same error.


Might be an error in stack.sh or the openstack script?


Any ideas?

Thanks,

Eduard


 *Eduard Biceri Mate, Senior Software Developer*

www.cloudfounders.com
 | eduard.matei at cloudfounders.com



*CloudFounders, The Private Cloud Software Company*

Disclaimer:
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed.
If you are not the named addressee or an employee or agent responsible
for delivering this message to the named addressee, you are hereby
notified that you are not authorized to read, print, retain, copy or
disseminate this message or any part of it. If you have received this
email in error we request you to notify us by reply e-mail and to
delete all electronic files of the message. If you are not the
intended recipient you are notified that disclosing, copying,
distributing or taking any action in reliance on the contents of this
information is strictly prohibited.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore does not
accept liability for any errors or omissions in the content of this
message, and shall have no liability for any loss or damage suffered
by the user, which arise as a result of e-mail transmission.



On Wed, Sep 10, 2014 at 6:30 PM, Jay Pipes <jaypipes at gmail.com> wrote:

> On 09/10/2014 04:19 AM, Eduard Matei wrote:
>
>> Hi guys,
>> I'm working on setting up a CI testing platform for the Cinder driver
>> we're developing.
>> I followed
>> http://www.joinfu.com/2014/02/setting-up-an-external-
>> openstack-testing-system/
>> and part 2, and i got both the master and the slave working, but the
>> dsvm-tempest-full job is failing.
>>
> <snip>
>
>> 2014-09-10 11:15:37.318 | error: The following untracked working tree
>> files would be overwritten by checkout:
>>
> <snip>
>
>> Any ideas how to fix this?
>>
>
> Hi Eduard,
>
> I suspect, from looking at the above error, that you are re-using the
> slave instance on subsequent runs of Jenkins jobs. If you are doing this
> (as opposed to using something like nodepool to keep a pool of single-use
> slave VMs around for running Jenkins jobs, then you will need to run some
> sort of cleanup/reset script after the Jenkins job completes.
>
> Akihiro Motoki has a nice repo with scripts you can use for this purpose:
>
> https://github.com/amotoki/devstack-tools
>
> (see the devstack-cleaner.sh script)
>
> For best results, however, the upstream infrastructure team has definitely
> found that using single-use slave VMs, managed by nodepool, is the safest
> and cleanest way to handle Jenkins jobs that involve devstack-gate (like
> the jobs detailed in my joinfu.com posts).
>
> All the best,
> -jay
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20140911/3d9ec17d/attachment-0001.html>

More information about the OpenStack-Infra mailing list