[OpenStack-Infra] Fwd: [Support] Problems installing zuul for jenkins-gerrit gateway (Jenkins does not run jobs)
Abhishek Shrivastava
abhishek at cloudbyte.com
Tue Dec 23 09:54:14 UTC 2014
---------- Forwarded message ----------
From: Abhishek Shrivastava <abhishek at cloudbyte.com>
Date: Tue, Dec 23, 2014 at 3:23 PM
Subject: Re: [OpenStack-Infra] [Support] Problems installing zuul for
jenkins-gerrit gateway (Jenkins does not run jobs)
To: Florian Schmidt <florian.schmidt.welzow at t-online.de>
Hi Florian,
Thanks for this article, but since we haven't got any issues like yours in
our setup, therefore I am sending you one *automated procedure document*
for setting up the CI environment that we are following. I hope it can
help.
On Tue, Dec 23, 2014 at 3:00 PM, Florian Schmidt <
florian.schmidt.welzow at t-online.de> wrote:
> Hello,
>
> thanks for your answer. For Jenkins i used this wiki article to install it
> on Ubuntu server [1] and added Jobs via the Jenkins web frontend (if you
> need the parameter for the test job, i can attach it, but not now, cause
> i'm in a train :/)
>
> In order to install zuul i downloaded the gearman plugin from openstack's
> git and build it with marven (clean install), uploaded the resulting plugin
> binary to jenkins (again via the web frontned), test the connection in the
> settings (works) and enabled gearman.
>
> [1]
> https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu
>
> If you have further questions, i'm happy to answer :)
>
> Kind regards,
> Florian
>
> Gesendet mit meinem HTC
>
> ----- Reply message -----
> Von: "Abhishek Shrivastava" <abhishek at cloudbyte.com>
> An: "Florian Schmidt" <florian.schmidt.welzow at t-online.de>, <
> openstack-infra at lists.openstack.org>
> Cc: "Amit Das" <amit.das at cloudbyte.com>, "Punith S" <
> punith.s at cloudbyte.com>
> Betreff: [OpenStack-Infra] [Support] Problems installing zuul for
> jenkins-gerrit gateway (Jenkins does not run jobs)
> Datum: Di., Dez. 23, 2014 07:35
>
> Hi Florian,
>
> Can you please tell me the procedure which you have followed for setting
> up the CI environment, so that I can be clear with your *zuul* problem.
>
> ---------- Forwarded message ----------
>> From: Florian Schmidt <florian.schmidt.welzow at t-online.de>
>> Date: Mon, Dec 22, 2014 at 7:04 PM
>> Subject: [OpenStack-Infra] [Support] Problems installing zuul for
>> jenkins-gerrit gateway (Jenkins does not run jobs)
>> To: openstack-infra at lists.openstack.org
>>
>>
>> Hello all together,
>>
>> i hope, someone of you can help me with my little problem. Actually I use
>> Gerrit Trigger 8Jenkins plugin) to trigger Jenkins jobs for specific
>> Gerrit
>> events. Now I want to migrate from Gerrit Trigger to zuul (easier, layout
>> based configuration and more trigger options and different pipelines).
>>
>> I installed zuul with this Blog post as a basis:
>>
>> http://ritchey98.blogspot.jp/2014/02/openstack-third-party-testing-how-to.ht
>> ml
>> <http://ritchey98.blogspot.jp/2014/02/openstack-third-party-testing-how-to.html>
>>
>> You can find my zuul configuration here[1] and my initial layout.yaml
>> here[2]
>>
>> Now I have the problem, that it seems, that Jenkins doesn't run jobs added
>> In gearman (they don't appear in Jenkins web frontend). Zuul's status page
>> lists the change (added via a "recheck" comment) and the status of the job
>> "Jenkins-test" is "queued" forever. That's why I think there is a problem
>> somewhere between Jenkins and zuul, but I can't find it. I hope, that
>> someone from here can assist me to find out the problem? I cleared the log
>> and restarted Jenkins and zuul (stopped both and started zuul first and
>> then
>> Jenkins) and added a new comment "recheck" to my test change in gerrit to
>> trigger a new build. I uploaded my log files here[3], maybe it helps to
>> find
>> out the problem. I have anonymized the domain to be "example.com" in the
>> log
>> files.
>>
>> I'm also online in #openstack-infra on freenode (nick: FlorianSW), so feel
>> free to contact me there, I would be very grateful to get help to solve
>> this
>> problem :)
>>
>> [1] https://gist.github.com/Florianschmidtwelzow/ed0e3047f0ef0c5e5554
>> [2] https://gist.github.com/Florianschmidtwelzow/a101a87d653d5bbc6de6
>> [3] https://gist.github.com/Florianschmidtwelzow/12fff7c4530805dece0c
>>
>> Kind regards,
>> Florian
>>
>>
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>>
>
>
> --
> Thanks & Regards,
> Abhishek
>
--
Thanks & Regards,
Abhishek
--
Thanks & Regards,
Abhishek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20141223/234490b1/attachment-0001.html>
-------------- next part --------------
================================================================================
README FOR SETUP CINDER CI SYSTEM FOR HDS DRIVERS
================================================================================
================================================================================
ABOUT
================================================================================
The propose of this README is to set up a real external testing plataform for
OpenStack using VMWare to handle nodes snapshots.
This README contains the step-by-step to install Jenkins, Jenkins Job Builder
(JJB), the Gerrit Jenkins plugin and auxiliary scripts and the HDS/Hitachi jobs.
The most part of scripts used in this configuration was forked from:
- jaypipes (https://github.com/jaypipes/os-ext-testing)
- rasselin (https://github.com/rasselin/os-ext-testing)
================================================================================
PRE-REQUISITES
================================================================================
To set up the external OpenStack CI system you will need:
1. At least 2 Ubuntu 12.04 machines with at least 8GB each machine
2. A Gerrit account registered for your testing system
3. Install base dependency packages on your hosts/VMs
4. Set up your data repository
2. A Gerrit account registered for your testing system
In order for your testing platform to post review comments to Gerrit code
reviews on openstack.org, you will need to have a service account registered
with the OpenStack Infra team. See this link
(http://ci.openstack.org/third_party.html#requesting-a-service-account) for
instructions on getting this account.
In short, you will need to email the OpenStack Infra mailing list
(http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra) an email
that includes:
- The email address to use for the system account
(must be different from any other Gerrit account)
- A short account username that will appear on code reviews
- (optional) A longer account name or description
- (optional but encouraged) Include your contact information
(IRC handle, your email address, and maybe an alternate contact’s email
address) to assist the upstream infrastructure team
- The public key for an SSH key pair that the service account will use for
Gerrit access.Please note that there should be no newlines in the SSH key
Don’t have an SSH key pair for your Gerrit service account?
You can create one like so:
$ ssh-keygen -t rsa -b 1024 -N '' -f gerrit_key
The above will produce the key pair: a pair of files called gerrit_key and
gerrit_key.pub. Copy the text of the gerrit_key.pub into the email you send to
the OpenStack Infra mailing list. Keep both the files handy for use in the
next step.
3. Install base dependency packages on your hosts/VMs
$ sudo apt-get update
$ sudo apt-get install git wget openssl ssl-cert ca-certificates
4. Set up your data repository
You will want to create a Git repository containing configuration data files
-- such as the Gerrit username and private SSH key file for your testing
account -- that are used in setting up the test platform.
The easiest way to get your data repository set up is to make a copy of the
example repository I set up here: http://github.com/rasselin/os-ext-testing-data
Instead of forking the repository, like you might would normally do, I recommend
instead just git clone’ing the repository to some local directory, and making it
your own data repository:
$ git clone https://github.com/rasselin/os-ext-testing-data.git \
~/os-ext-testing-data
$ cd os-ext-testing-data
$ rm -rf .git
$ git init .
$ git add .
$ git commit -a -m "My new data repository"
There are a few things you will need to do in this data repository:
I. Copy the private SSH key that you submitted when you registered with the
upstream OpenStack Infrastructure team into somewhere in this repo.
II. If you do not want to use the SSH key pair in the os-ext-testing-data
example data repository and want to create your own SSH key pair, do
this step.
Create an SSH key pair that you will use for Jenkins. This SSH key pair
will live in the /var/lib/jenkins/.ssh/ directory on the master Jenkins
host, and it will be added to the /home/jenkins/.ssh/authorized_keys
file of all slave hosts:
$ ssh-keygen -t rsa -b 1024 -N '' -f jenkins_key
Once you do the above, copy the jenkins_key and jenkins_key.pub files
into your data repository.
III. Open up vars.sh in an editor:
a) Change the value of $GIT_EMAIL to the email you registered with the
upstream OpenStack Infrastructure team
b) Change the value of the $GIT_NAME shell variable to the Gerrit
username you registered with the upstream OpenStack Infrastructure
team as detailed in these instructions
c) Change the value of the $UPSTREAM_GERRIT_USER shell variable to the
Gerrit username you registered with the upstream OpenStack
Infrastructure team as detailed in these instructions
d) Change the value of the $UPSTREAM_GERRIT_SSH_KEY_PATH shell variable
to the relative path of the private SSH key file you copied into the
repository in step #1.
For example, let's say you put your private SSH key file named
mygerritkey into a directory called ssh within the repository, you
would set the $UPSTREAM_GERRIT_SSH_KEY_PATH value to ssh/mygerritkey
e) If for some reason, in step #2 above, you either used a different
output filename than jenkins_key or put the key pair into some
subdirectory of your data repository, then change the value of the
$JENKINS_SSH_KEY_PATH variable in vars.sh to an appropriate value.
f) Paste the jenkins public key with no whitespaces (only the key) to
the $JENKINS_SSH_PUBLIC_KEY_NO_WHITESPACE
g) Rename nodepool config file sample to script used by master:
$ mv os-ext-testing-data/etc/nodepool/nodepool.yaml.erb.sample \
os-ext-testing-data/etc/nodepool/nodepool.yaml.erb
IV. Next, open up the file etc/jenkins_jobs/config/projects.yaml in your
data repository. Change the following line in it:
vendor: MyVendor
Change MyVendor to your organization’s name.
V. Save changes in your data repository:
$ git add .
$ git commit -a -m "Added Gerrit SSH key and username"
$ git push
================================================================================
Install Jenkins Master
================================================================================
On the host or virtual machine (or LXC container) you wish to run the Jenkins
Master node on, run the following (the git account need to be set before run
script - public key):
$ git clone $YOUR_DATA_REPO
$ wget https://raw.github.com/rasselin/os-ext-testing/master/puppet/install_master.sh
$ bash install_master.sh
The script will install Puppet, create an SSH key for the Jenkins master,
create self-signed certificates for Apache, and then will ask you for the URL
of the Git repository you are using as your data repository (see Prerequisites
#4 above). Enter the URL of your data repository and hit Enter.
Puppet will proceed to set up the Jenkins master.
When Puppet completes, go ahead and open up the Jenkins web UI, which by default
will be at http://$HOST_IP:8080. You will need to enable the Gearman workers
that Zuul and Jenkins use to interact. To do this:
1. Restart jenkins:
$ sudo service jenkins restart
2. Click the `Manage Jenkins` link on the left
3. Click the `Configure System` link
4. Scroll down until you see “Gearman Plugin Config”. Check the “Enable Gearman”
checkbox.
5. Click the “Test Connection” button and verify Jenkins connects to Gearman.
6. Scroll down to the bottom of the page and click `Save`
7. Once you are done with that, it’s time to load up your Jenkins jobs and
restart Zuul:
$ sudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/
$ sudo service zuul restart
If you want to test the communication between Upstream and your master,
you can do:
$ sudo tail -f /var/log/zuul/debug.log | grep sandbox
OK, now create a simple test patch in sandbox. Do this on your development
workstation, not your Jenkins master (to do it, you will need to set your
gerrit credentials):
$ git clone git at github.com:openstack-dev/sandbox /tmp/sandbox
$ cd /tmp/sandbox
$ git checkout -b testing-ext
$ touch mytest
$ git add mytest
$ git commit -a -m "Testing comms"
$ git review
After do it, You will see in the debug.log a gerrit request to run the job
pre-configured.
================================================================================
Install Jenkins Master Plugins
================================================================================
Now, you need to update the plugins installed and install anothers:
1. Click the `Manage Jenkins` link on the left
2. Click the `Manage Plugins` link
3. Update all plugins, except the LDAP Plugin
4. Go to available tab
5. Install the following plugins: Conditional-buildstep and Jenkins
Parameterized Trigger plugin
6. Manual setup of Jenkins scp 1.9 plugin as follow:
Version 1.8 is publicly available, but does not have all features (e.g. copy
console log file, copy files after failure, etc.). Follow these steps to
manually build and install the scp 1.9 plugin. Solve some dependencies is
needed to build using "mvn":
$ git clone https://github.com/jenkinsci/scp-plugin
$ cd scp-plugin; mvn package
When the plugin compilation was finished, go to Jenkins UI and upload the
plugin:
Jenkins Manage Plugins -> Advanced ->
Upload Plugin (target/scp.hpi from the maven build)
Source: http://lists.openstack.org/pipermail/openstack-infra/2013-December/000568.html
7. After install SCP Plugin, you need to configure it:
a) Click the `Manage Jenkins` link on the left
b) Click the `Configure System` link
c) Scroll down until you see “SCP repository hosts”.
d) Fill out all fields on the section (the Keyfile option is optional).
================================================================================
Install VMWare tools
================================================================================
On node that Jenkins Master was installed:
# http://trickycloud.wordpress.com/2013/11/01/using-the-vsphere-sdkapi-in-ubuntu/
configuring vmware client:
1 - Install prereqs:
$ sudo apt-get -y update && sudo apt-get install ia32-libs build-essential \
gcc uuid uuid-dev perl libssl-dev perl-doc liburi-perl libxml-libxml-perl \
libcrypt-ssleay-perl
2 - Update the deprecated modules using cpan. Hit <Enter> for all questions:
$ sudo cpan install LWP::UserAgent
$ sudo cpan install Compress::Zlib
$ sudo cpan install Compress::Raw::Zlib
$ sudo cpan install IO::Compress::Base
$ sudo cpan install IO::Compress::Zlib::Constants
3 - Checking versions:
$ perl -MLWP -e 'print "$LWP::VERSION\n"'
$ perl -MNet::HTTP -e 'print "$Net::HTTP::VERSION\n"'
4 - Download and install the versions 6.03 of Net-HTTP and LWP:
$ wget http://backpan.perl.org/authors/id/G/GA/GAAS/libwww-perl-6.03.tar.gz
$ wget http://backpan.perl.org/authors/id/G/GA/GAAS/Net-HTTP-6.03.tar.gz
PS: Unpack the files and follow the instructions of README
5 - Verify the new installed versions:
$ perl -MLWP -e 'print "$LWP::VERSION\n"'
$ perl -MNet::HTTP -e 'print "$Net::HTTP::VERSION\n"'
6 - Download and install vmware SDK client version 5.5 (https://my.vmware.com/group/vmware/details?downloadGroup=SDKPERL550&productId=353#)
To do it, You need to register yourself in VMWare website.
7 - Test with the following command:
$ vmware-cmd --server 172.24.44.14 --username 'USER' --password 'PASSWORD' \
--vihost <VCENTER IP> "[<DATASTORE>] <DIR PATH>/<FILENAME>.vmx" hassnapshot
================================================================================
Install Jenkins Slave
================================================================================
On each host or virtual machine (or LXC container) you wish to run the Jenkins
Slave node on, run the following (the git account need to be set before run
script - public key):
$ git clone $YOUR_DATA_REPO
$ wget https://raw.github.com/rasselin/os-ext-testing/master/puppet/install_slave.sh
$ bash install_slave.sh
The script will install Puppet, install a Jenkins slave, and install the Jenkins
master's public SSH key in the authorized_keys of the Jenkins slave.
Once the script completes successfully, you need to add the slave node to
Jenkins master. To do so manually, follow these steps:
1. Go to the Jenkins web UI. By default, this will be http://$IP_OF_MASTER:8080
2. Click the Credentials link on the left
3. Click the Global credentials link
4. Click the Add credentials link on the left
5. Select SSH username with private key from the dropdown labeled "Kind"
6. Enter "jenkins" in the Username textbox
7. Select the "From a file on Jenkins master" radio button and enter
/var/lib/jenkins/.ssh/id_rsa in the File textbox
8. Click the OK button
9. Click the "Jenkins" link in the upper left to go back to home page
10. Click the Manage Jenkins link on the left
11. Click the Manage Nodes link
12. Click the "New Node" link on the left
13. Enter devstack_slave1 in the Node name textbox
14. Select the Dumb Slave radio button
15. Click the OK button
16. Enter 2 in the Executors textbox
17. Enter /home/jenkins/workspaces in the Remote FS root textbox
18. Enter devstack_slave in the Labels textbox
19. Enter the IP Address of your slave host or VM in the Host textbox
20. Select jenkins from the Credentials dropdown
21. Click the Save button
22. Click the Log link on the left. The log should show the master connecting
to the slave, and at the end of the log should be: "Slave successfully
connected and online"
================================================================================
Configure Jobs on Master
================================================================================
On the Jenkins Master Node, you need to configure the desired jobs. To work with
VMWare, we need to configure a especific type of job.
In this toturial we will configure part of the jobs configuration via command
line and another configs will be set via GUI.
1. Open the /etc/jenkins_jobs/config/projects.yaml and edit:
- project:
name: <gerrit-project-name>
github-org: <git-repo-name gerrit-project>
node: <master-node-name>
vendor: <vendor>
jobs:
- <gerrit-name-job>
- <auxiliary-job>:
node: <slave-node-name>
2. Open the /etc/jenkins_jobs/config/examples.yaml and edit:
- job-template:
name: <gerrit-name-job>
node: <master-node-name>
- job-template:
name: <auxialiry-job>
node: '{node}'
wrappers:
- timeout:
timeout: 120 # Timeout in *minutes*
fail: true # A job run that exceeds the timeout will cause a failure
- timestamps
builders:
- devstack-checkout
- shell: |
<scripts to configure driver and set env vars>
- link-logs # In macros.yaml from os-ext-testing
publishers:
- devstack-logs # In macros.yaml from os-ext-testing
- console-log # In macros.yaml from os-ext-testing
PS: An auxialiry-job template can be found in
~/os-ext-testing-data/etc/jenkins_jobs/config/dsvm-cinder-driver.yaml.sample
3. Open the /etc/jenkins_jobs/config/macros.yaml and edit:
- builder:
name: link-logs
builders:
- shell: |
#!/bin/sh
echo "Detailed logs: http://<ip-log-server>/$LOG_PATH/"
- publisher:
name: console-log
publishers:
- scp:
site: '<ip-log-server>'
files:
- target: '$LOG_PATH'
copy-console: true
copy-after-failure: true
- publisher:
name: devstack-logs
publishers:
- scp:
site: '<ip-log-server>'
files:
- target: '$LOG_PATH'
source: 'logs/**'
keep-hierarchy: true
copy-after-failure: true
4. Open the /etc/zuul/zuul.conf and edit:
url_pattern=http://<ip-log-server>/{build.parameters[LOG_PATH]}
status_url=http://<ip-log-server>/zuul/status
5. Update the changes:
$ sudo jenkins-jobs update /etc/jenkins_jobs/config
6. On GUI:
a) Go to the Jenkins web UI. By default, this will be
http://$IP_OF_MASTER:8080
b) Click on <gerrit-name-job>
c) Click the Configure link
d) Configure "Dicard Olb Builds" as desired
e) Uncheck "Execute concurrent buils if necessary"
f) Set Time-out strategy to "Absolute"
g) Set Timeout minutes to "180"
h) Click the Add build step button
i) Click the Conditional steps (multiple) option
j) Set Run? to "Always"
k) Click the Add step to condition button
l) Click the Execute Shell option
m) Add to the textbox:
#!/bin/bash
vmware-cmd --server <VCENTE IP> --username 'USERNAME' --password 'PASSWORD' --vihost <VCENTER HOST IP> "[<DATASTORE>] <DATASTORE DIR PATH>/<DATASTORE FILENAME>.vmx" revertsnapshot
vmware-cmd --server <VCENTE IP> --username 'USERNAME' --password 'PASSWORD' --vihost <VCENTER HOST IP> "[<DATASTORE>] <DATASTORE DIR PATH>/<DATASTORE FILENAME>.vmx" start hard
n) Click the Add step to condition button
o) Click the Execute Shell option
p) Add to the textbox:
#!/bin/bash
sleep 200
ssh jenkins@<JENKINS SLAVE IP> "exec java -jar /home/jenkins/workspaces/slave.jar &"
sleep 160
q) Click the Add step to condition button
r) Click the Trigger/call builds on other projects option
s) Set <auxialiry-job-name> to 'Projects to build'
t) Check the Block until the triggered projects... option
u) Click the Add Parameters button
v) Click the Predefined parameters option
w) Add to the textbox:
LOG_PATH=$LOG_PATH
x) Save configuration
================================================================================
Configure Slave Node
================================================================================
1. Configure NTP
Open /etc/ntp.conf
Edit the server section
Add the Jenkins Master as first NTP server
server <JENKINS MASTER IP> iburst
restart ntp server
sudo service ntp restart
2. Install driver deps, example: nfs-common, open-iscsi-utils, tool binaries,...
3. Add all config file needed, except the cinder.conf and tempest files
4. Change the MySQL password of root user to "secretmysql". That is the default
password set on scripts.
5. Take a snapshot that will be used for CI to revert the state of the VM
=============================================================================
Configure Log Server
=============================================================================
1. Setup a Machine with public IP, Ubuntu OS, apache and ssh installed
2. Create a user for Jenkins to allow it to copy the log files to the server
More information about the OpenStack-Infra
mailing list