Open Stack

Hi Amit, Bharat
Attached the tutorial.

On Fri, Oct 31, 2014 at 10:15 AM, Amit Das <amit.das at cloudbyte.com> wrote:

> Thanks Erlon & Jeremy for the detailed responses.
>
> Erlon, the configuration doc will definitely help us. Looking forward to
> it.
>
> Regards,
> Amit
> *CloudByte Inc.* <http://www.cloudbyte.com/>
>
> On Fri, Oct 31, 2014 at 5:29 PM, Erlon Cruz <sombrafam at gmail.com> wrote:
>
>> We have a CI for Cinder drivers. Currently we use 1 instance for the
>> master and 1 instance for each slave(we test one driver on each slave). The
>> configuration is the following:
>>
>> Master: 2VCPUs + 8GB RAM
>> Slaves: 2VCPUs + 12GB RAM
>> Network: 4Mb Download + 4Mb Upload
>>
>> Our jobs usually take 1:20h to run, but that is because of the network.
>> In our setup we virtualize all hosts using VMWare, which is very handy as
>> we can save an snapshot and trigger a revert after the job is complete
>> also, if you have FC drivers, VMWare will save a lot of pain with it PCI
>> passthrought. We have an step by step to configure the env with VMWare and
>> Cinder drivers, if you need we can make it available.
>>
>>
>> On Thu, Oct 30, 2014 at 1:24 PM, Jeremy Stanley <fungi at yuggoth.org>
>> wrote:
>>
>>> On 2014-10-30 20:28:17 +0530 (+0530), Amit Das wrote:
>>> > This is exactly what we intend to do. Wanted to know if there is
>>> > anything else that we should know before placing the requests.
>>>
>>> In that case, if you're wanting to do devstack-gate based jobs we
>>> use instances with 8 VCPUs and 8 GB RAM. A typical tempest-full run
>>> takes somewhere in the vicinity of an hour to complete depending on
>>> what configuration options have been selected. Due to the inability
>>> to cleanly re-run DevStack or to trust that proposed changes haven't
>>> destroyed the usability of the worker instance, we recycle and
>>> replace them with fresh instances for each job.
>>> --
>>> Jeremy Stanley
>>>
>>> _______________________________________________
>>> OpenStack-Infra mailing list
>>> OpenStack-Infra at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20141031/f845a31f/attachment-0001.html>
-------------- next part --------------
================================================================================
             README FOR SETUP CINDER CI SYSTEM FOR HDS DRIVERS
================================================================================


================================================================================
ABOUT
================================================================================

The propose of this README is to set up a real external testing plataform for 
OpenStack using VMWare to handle nodes snapshots.

This README contains the step-by-step to install Jenkins, Jenkins Job Builder 
(JJB), the Gerrit Jenkins plugin and auxiliary scripts and the HDS/Hitachi jobs.

The most part of scripts used in this configuration was forked from:
    - jaypipes (https://github.com/jaypipes/os-ext-testing)
    - rasselin (https://github.com/rasselin/os-ext-testing)


================================================================================
PRE-REQUISITES
================================================================================

To set up the external OpenStack CI system you will need:

    1. At least 2 Ubuntu 12.04 machines with at least 8GB each machine
    2. A Gerrit account registered for your testing system
    3. Install base dependency packages on your hosts/VMs
    4. Set up your data repository


2. A Gerrit account registered for your testing system

In order for your testing platform to post review comments to Gerrit code 
reviews on openstack.org, you will need to have a service account registered 
with the OpenStack Infra team. See this link 
(http://ci.openstack.org/third_party.html#requesting-a-service-account) for 
instructions on getting this account.

In short, you will need to email the OpenStack Infra mailing list 
(http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra) an email 
that includes:

    - The email address to use for the system account 
      (must be different from any other Gerrit account)
    - A short account username that will appear on code reviews
    - (optional) A longer account name or description
    - (optional but encouraged) Include your contact information 
      (IRC handle, your email address, and maybe an alternate contact’s email 
      address) to assist the upstream infrastructure team
    - The public key for an SSH key pair that the service account will use for 
      Gerrit access.Please note that there should be no newlines in the SSH key
    
Don’t have an SSH key pair for your Gerrit service account? 
You can create one like so:

$ ssh-keygen -t rsa -b 1024 -N '' -f gerrit_key

The above will produce the key pair: a pair of files called gerrit_key and 
gerrit_key.pub. Copy the text of the gerrit_key.pub into the email you send to 
the OpenStack Infra mailing list. Keep both the files handy for use in the 
next step.


3. Install base dependency packages on your hosts/VMs

$ sudo apt-get update
$ sudo apt-get install git wget openssl ssl-cert ca-certificates


4. Set up your data repository

You will want to create a Git repository containing configuration data files 
-- such as the Gerrit username and private SSH key file for your testing 
account -- that are used in setting up the test platform.

The easiest way to get your data repository set up is to make a copy of the 
example repository I set up here: http://github.com/rasselin/os-ext-testing-data

Instead of forking the repository, like you might would normally do, I recommend
instead just git clone’ing the repository to some local directory, and making it
your own data repository:

$ git clone https://github.com/rasselin/os-ext-testing-data.git \
  ~/os-ext-testing-data
$ cd os-ext-testing-data
$ rm -rf .git
$ git init .
$ git add .
$ git commit -a -m "My new data repository"


There are a few things you will need to do in this data repository:

    I. Copy the private SSH key that you submitted when you registered with the 
       upstream OpenStack Infrastructure team into somewhere in this repo.

    II. If you do not want to use the SSH key pair in the os-ext-testing-data 
        example data repository and want to create your own SSH key pair, do 
        this step.

        Create an SSH key pair that you will use for Jenkins. This SSH key pair 
        will live in the /var/lib/jenkins/.ssh/ directory on the master Jenkins 
        host, and it will be added to the /home/jenkins/.ssh/authorized_keys 
        file of all slave hosts:

        $ ssh-keygen -t rsa -b 1024 -N '' -f jenkins_key

        Once you do the above, copy the jenkins_key and jenkins_key.pub files 
        into your data repository.
        
    III. Open up vars.sh in an editor:
        a) Change the value of $GIT_EMAIL to the email you registered with the 
           upstream OpenStack Infrastructure team

        b) Change the value of the $GIT_NAME shell variable to the Gerrit 
           username you registered with the upstream OpenStack Infrastructure 
           team as detailed in these instructions

        c) Change the value of the $UPSTREAM_GERRIT_USER shell variable to the 
           Gerrit username you registered with the upstream OpenStack 
           Infrastructure team as detailed in these instructions

        d) Change the value of the $UPSTREAM_GERRIT_SSH_KEY_PATH shell variable 
           to the relative path of the private SSH key file you copied into the 
           repository in step #1.

           For example, let's say you put your private SSH key file named 
           mygerritkey into a directory called ssh within the repository, you 
           would set the $UPSTREAM_GERRIT_SSH_KEY_PATH value to ssh/mygerritkey

        e) If for some reason, in step #2 above, you either used a different 
           output filename than jenkins_key or put the key pair into some 
           subdirectory of your data repository, then change the value of the 
           $JENKINS_SSH_KEY_PATH variable in vars.sh to an appropriate value.

        f) Paste the jenkins public key with no whitespaces (only the key) to 
           the $JENKINS_SSH_PUBLIC_KEY_NO_WHITESPACE

        g) Rename nodepool config file sample to script used by master:
           
           $ mv os-ext-testing-data/etc/nodepool/nodepool.yaml.erb.sample \
           os-ext-testing-data/etc/nodepool/nodepool.yaml.erb

    IV. Next, open up the file etc/jenkins_jobs/config/projects.yaml in your 
        data repository. Change the following line in it:

            vendor: MyVendor

        Change MyVendor to your organization’s name.
    
    V. Save changes in your data repository:
        
       $ git add .
       $ git commit -a -m "Added Gerrit SSH key and username"
       $ git push


================================================================================
Install Jenkins Master
================================================================================

On the host or virtual machine (or LXC container) you wish to run the Jenkins 
Master node on, run the following (the git account need to be set before run 
script - public key):

$ git clone $YOUR_DATA_REPO
$ wget https://raw.github.com/rasselin/os-ext-testing/master/puppet/install_master.sh
$ bash install_master.sh

The script will install Puppet, create an SSH key for the Jenkins master, 
create self-signed certificates for Apache, and then will ask you for the URL 
of the Git repository you are using as your data repository (see Prerequisites 
#4 above). Enter the URL of your data repository and hit Enter.

Puppet will proceed to set up the Jenkins master.

When Puppet completes, go ahead and open up the Jenkins web UI, which by default
will be at http://$HOST_IP:8080. You will need to enable the Gearman workers 
that Zuul and Jenkins use to interact. To do this:

1. Restart jenkins:

   $ sudo service jenkins restart
    
2. Click the `Manage Jenkins` link on the left
3. Click the `Configure System` link
4. Scroll down until you see “Gearman Plugin Config”. Check the “Enable Gearman”
   checkbox.
5. Click the “Test Connection” button and verify Jenkins connects to Gearman.
6. Scroll down to the bottom of the page and click `Save`
7. Once you are done with that, it’s time to load up your Jenkins jobs and 
   restart Zuul:
       
   $ sudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/
   $ sudo service zuul restart

If you want to test the communication between Upstream and your master, 
you can do:

    $ sudo tail -f /var/log/zuul/debug.log | grep sandbox
    
OK, now create a simple test patch in sandbox. Do this on your development 
workstation, not your Jenkins master (to do it, you will need to set your 
gerrit credentials):

    $ git clone git at github.com:openstack-dev/sandbox /tmp/sandbox
    $ cd /tmp/sandbox
    $ git checkout -b testing-ext
    $ touch mytest
    $ git add mytest
    $ git commit -a -m "Testing comms"
    $ git review


After do it, You will see in the debug.log a gerrit request to run the job 
pre-configured.


================================================================================
Install Jenkins Master Plugins
================================================================================

Now, you need to update the plugins installed and install anothers:

1. Click the `Manage Jenkins` link on the left
2. Click the `Manage Plugins` link
3. Update all plugins, except the LDAP Plugin
4. Go to available tab
5. Install the following plugins: Conditional-buildstep and Jenkins 
   Parameterized Trigger plugin
6. Manual setup of Jenkins scp 1.9 plugin as follow:

   Version 1.8 is publicly available, but does not have all features (e.g. copy 
   console log file, copy files after failure, etc.). Follow these steps to 
   manually build and install the scp 1.9 plugin. Solve some dependencies is 
   needed to build using "mvn":

   $ git clone https://github.com/jenkinsci/scp-plugin
   $ cd scp-plugin; mvn package

   When the plugin compilation was finished, go to Jenkins UI and upload the 
   plugin:

       Jenkins Manage Plugins -> Advanced -> 
       Upload Plugin (target/scp.hpi from the maven build)

   Source: http://lists.openstack.org/pipermail/openstack-infra/2013-December/000568.html

7. After install SCP Plugin, you need to configure it:
    a) Click the `Manage Jenkins` link on the left
    b) Click the `Configure System` link
    c) Scroll down until you see “SCP repository hosts”. 
    d) Fill out all fields on the section (the Keyfile option is optional).


================================================================================
Install VMWare tools
================================================================================

On node that Jenkins Master was installed:

# http://trickycloud.wordpress.com/2013/11/01/using-the-vsphere-sdkapi-in-ubuntu/
configuring vmware client:

1 - Install prereqs: 
    $ sudo apt-get -y update && sudo apt-get install ia32-libs build-essential \
    gcc uuid uuid-dev perl libssl-dev perl-doc liburi-perl libxml-libxml-perl \
    libcrypt-ssleay-perl

2 - Update the deprecated modules using cpan. Hit <Enter> for all questions:
    $ sudo cpan install LWP::UserAgent
    $ sudo cpan install Compress::Zlib
    $ sudo cpan install Compress::Raw::Zlib
    $ sudo cpan install IO::Compress::Base
    $ sudo cpan install IO::Compress::Zlib::Constants
    
3 - Checking versions: 
    $ perl -MLWP       -e 'print "$LWP::VERSION\n"'
    $ perl -MNet::HTTP -e 'print "$Net::HTTP::VERSION\n"'
   
4 - Download and install the versions 6.03 of Net-HTTP and LWP:
    $ wget http://backpan.perl.org/authors/id/G/GA/GAAS/libwww-perl-6.03.tar.gz
    $ wget http://backpan.perl.org/authors/id/G/GA/GAAS/Net-HTTP-6.03.tar.gz
    
    PS: Unpack the files and follow the instructions of README

5 - Verify the new installed versions: 
    $ perl -MLWP       -e 'print "$LWP::VERSION\n"'
    $ perl -MNet::HTTP -e 'print "$Net::HTTP::VERSION\n"'   

6 - Download and install vmware SDK client version 5.5 (https://my.vmware.com/group/vmware/details?downloadGroup=SDKPERL550&productId=353#)
    To do it, You need to register yourself in VMWare website.

7 - Test with the following command:
    $ vmware-cmd --server 172.24.44.14 --username 'USER' --password 'PASSWORD' \
    --vihost <VCENTER IP> "[<DATASTORE>] <DIR PATH>/<FILENAME>.vmx" hassnapshot


================================================================================
Install Jenkins Slave
================================================================================

On each host or virtual machine (or LXC container) you wish to run the Jenkins 
Slave node on, run the following (the git account need to be set before run 
script - public key):

$ git clone $YOUR_DATA_REPO
$ wget https://raw.github.com/rasselin/os-ext-testing/master/puppet/install_slave.sh
$ bash install_slave.sh

The script will install Puppet, install a Jenkins slave, and install the Jenkins
master's public SSH key in the authorized_keys of the Jenkins slave.
Once the script completes successfully, you need to add the slave node to 
Jenkins master. To do so manually, follow these steps:

1. Go to the Jenkins web UI. By default, this will be http://$IP_OF_MASTER:8080
2. Click the Credentials link on the left
3. Click the Global credentials link
4. Click the Add credentials link on the left
5. Select SSH username with private key from the dropdown labeled "Kind"
6. Enter "jenkins" in the Username textbox
7. Select the "From a file on Jenkins master" radio button and enter 
   /var/lib/jenkins/.ssh/id_rsa in the File textbox
8. Click the OK button
9. Click the "Jenkins" link in the upper left to go back to home page
10. Click the Manage Jenkins link on the left
11. Click the Manage Nodes link
12. Click the "New Node" link on the left
13. Enter devstack_slave1 in the Node name textbox
14. Select the Dumb Slave radio button
15. Click the OK button
16. Enter 2 in the Executors textbox
17. Enter /home/jenkins/workspaces in the Remote FS root textbox
18. Enter devstack_slave in the Labels textbox
19. Enter the IP Address of your slave host or VM in the Host textbox
20. Select jenkins from the Credentials dropdown
21. Click the Save button
22. Click the Log link on the left. The log should show the master connecting 
    to the slave, and at the end of the log should be: "Slave successfully 
    connected and online"


================================================================================
Configure Jobs on Master
================================================================================

On the Jenkins Master Node, you need to configure the desired jobs. To work with
VMWare, we need to configure a especific type of job.

In this toturial we will configure part of the jobs configuration via command 
line and another configs will be set via GUI.

1. Open the /etc/jenkins_jobs/config/projects.yaml and edit:

- project:
    name: <gerrit-project-name>
    github-org: <git-repo-name gerrit-project>
    node: <master-node-name>
    vendor: <vendor>

    jobs:
        - <gerrit-name-job>
        - <auxiliary-job>:
            node: <slave-node-name>


2. Open the /etc/jenkins_jobs/config/examples.yaml and edit:

- job-template:
    name: <gerrit-name-job>
    node: <master-node-name>
    
- job-template:
    name: <auxialiry-job>
    node: '{node}'

    wrappers:
      - timeout:
          timeout: 120  # Timeout in *minutes*
          fail: true  # A job run that exceeds the timeout will cause a failure
      - timestamps

    builders:
      - devstack-checkout
      - shell: |
          <scripts to configure driver and set env vars>
      - link-logs  # In macros.yaml from os-ext-testing

    publishers:
      - devstack-logs  # In macros.yaml from os-ext-testing
      - console-log  # In macros.yaml from os-ext-testing

PS: An auxialiry-job template can be found in 
~/os-ext-testing-data/etc/jenkins_jobs/config/dsvm-cinder-driver.yaml.sample

3. Open the /etc/jenkins_jobs/config/macros.yaml and edit:

- builder:
    name: link-logs
    builders:
      - shell: |
          #!/bin/sh
          echo "Detailed logs: http://<ip-log-server>/$LOG_PATH/"

- publisher:
    name: console-log
    publishers:
      - scp:
          site: '<ip-log-server>'
          files:
            - target: '$LOG_PATH'
              copy-console: true
              copy-after-failure: true

- publisher:
    name: devstack-logs
    publishers:
      - scp:
          site: '<ip-log-server>'
          files:
            - target: '$LOG_PATH'
              source: 'logs/**'
              keep-hierarchy: true
              copy-after-failure: true
   
4. Open the /etc/zuul/zuul.conf and edit:    

url_pattern=http://<ip-log-server>/{build.parameters[LOG_PATH]}
status_url=http://<ip-log-server>/zuul/status


5. Update the changes:

$ sudo jenkins-jobs update /etc/jenkins_jobs/config

6. On GUI:

    a) Go to the Jenkins web UI. By default, this will be 
       http://$IP_OF_MASTER:8080
    b) Click on  <gerrit-name-job>
    c) Click the Configure link
    d) Configure "Dicard Olb Builds" as desired
    e) Uncheck "Execute concurrent buils if necessary"
    f) Set Time-out strategy to "Absolute"
    g) Set Timeout minutes to "180"
    h) Click the Add build step button
    i) Click the Conditional steps (multiple) option
    j) Set Run? to "Always"
    k) Click the Add step to condition button
    l) Click the Execute Shell option
    m) Add to the textbox:
        #!/bin/bash
        vmware-cmd --server <VCENTE IP> --username 'USERNAME' --password 'PASSWORD'  --vihost <VCENTER HOST IP> "[<DATASTORE>] <DATASTORE DIR PATH>/<DATASTORE FILENAME>.vmx" revertsnapshot
        vmware-cmd --server <VCENTE IP> --username 'USERNAME' --password 'PASSWORD'  --vihost <VCENTER HOST IP> "[<DATASTORE>] <DATASTORE DIR PATH>/<DATASTORE FILENAME>.vmx" start hard
    n) Click the Add step to condition button
    o) Click the Execute Shell option
    p) Add to the textbox:
        #!/bin/bash
        sleep 200
        ssh jenkins@<JENKINS SLAVE IP> "exec java -jar /home/jenkins/workspaces/slave.jar &"
        sleep 160
    q) Click the Add step to condition button
    r) Click the Trigger/call builds on other projects option
    s) Set <auxialiry-job-name> to 'Projects to build'
    t) Check the Block until the triggered projects... option
    u) Click the Add Parameters button
    v) Click the Predefined parameters option
    w) Add to the textbox:
            LOG_PATH=$LOG_PATH
    x) Save configuration


================================================================================
Configure Slave Node
================================================================================

1. Configure NTP

    Open /etc/ntp.conf
    Edit the server section
    Add the Jenkins Master as first NTP server 
        server <JENKINS MASTER IP> iburst

    restart ntp server
        sudo service ntp restart

2. Install driver deps, example: nfs-common, open-iscsi-utils, tool binaries,...

3. Add all config file needed, except the cinder.conf and tempest files

4. Change the MySQL password of root user to "secretmysql". That is the default 
   password set on scripts.

5. Take a snapshot that will be used for CI to revert the state of the VM


=============================================================================
Configure Log Server
=============================================================================

1. Setup a Machine with public IP, Ubuntu OS, apache and ssh installed

2. Create a user for Jenkins to allow it to copy the log files to the server