[openstack-dev] [tripleo] CI Squad Meeting Summary (week 5 - for real now)
Attila Darazs
adarazs at redhat.com
Fri Feb 3 17:15:01 UTC 2017
As always, if these topics interest you and you want to contribute to
the discussion, feel free to join the next meeting:
Time: Thursdays, 15:30-16:30 UTC
Place: https://bluejeans.com/4113567798/
Note: last two weeks I used the incorrect week number. Getting back on
track, this is the 5th week of 2017.
Yesterday's meeting focused almost entirely on figuring out what's the
"feature delta" between the current TripleO CI and the functionality of
our Quickstart based CI. In the spirit of aviation we call this
"preflight checklist"[1].
It contains:
* the various variables that turn functionality on and off in upstream tests
* a short description of the variables
* a "Quickstart" section for each, describing if it's supported or not
currently. If yes, usually there's a link at the respective part, if
not, we add a Trello card to track the work, or a bug if we plan to take
care of it a bit later
* proposed new Quickstart jobs, where we combine the existing features
into fewer jobs with same coverage
* the existing upstream jobs with the features they currently cover
If you're somewhat familiar with the current CI system, please look over
these and let us know if there's any mistake in it.
Other than this, the new nodepool and OVB Quickstart jobs are working,
apart from the OVB HA job -- Sagi is working on it.
I'm not sure the link for the checklist is accessible for everyone, so
I'm going to paste it here after the link. Probably the formatting is
not perfect, so if you can, check the google doc.
Best regards,
Attila
[1]
https://docs.google.com/document/d/1Mb_t5Qe-Lnh0uaXy0ubX9y4k65Q4D_aw-49eZOqoviQ/edit?pli=1#
---
TripleO CI Quickstart Transition
Preflight Checklist - All the items must be ready by 10th March
This document describes:
* Existing features in the current TripleO CI
* Their support in Quickstart
* The current CI job feature sets (what features are tested in specific
jobs)
* The new proposed job feature sets (to reduce the amount of jobs while
keeping the same coverage)
Feature index
Each list item represents a “toci_gate_test.sh” variable that
enable/disable features in the CI jobs.
* 1-CONTROLLER
* Use 1 controller (and maybe other type of nodes)
* Quickstart: supported
* 3-CONTROLLERS
* Use 3 controllers (and maybe other type of nodes)
* Quickstart: supported
* 1-COMPUTE
* Use 1 compute node (and maybe other type of nodes)
* Quickstart: supported
* 1-CEPH
* Use 1 ceph node (and maybe other type of nodes)
* Quickstart: supported
* CONTAINERS
* Container based undercloud (feature under development)
* Container based overcloud
* Quickstart: supported (undercloud in progress)[a]
* DISABLE_IRONIC
* This is just a label that denotes the ability to skip ironic steps
entirely during multinode based jobs
* Quickstart: not needed, implemented elsewhere
* DELOREAN_HTTP_URLS
* We can't use squid to cache https urls, so don't use them
* Quickstart: make sure we use only http in build-test-package [card]
* IRONIC_DRIVERS_DEPLOY_HTTPPORT
* sets http port unconditionally to
ironic::drivers::deploy::http_port: 3816 in undercloud overrides
* Quickstart: not supported [card]
* IMAGE_CACHE_SELECT
* This feature enables to select whether or not using an image from
cache when gating specific project (i.e. projects that alter the image
creation process)
* TODO(gcerami) propose a change to unify the list of project for
every image to build (specific project gated -> all images will be
recreated from scratch)
* Quickstart: work in progress (trown, image build role) [card]
* IMAGE_CACHE_UPLOAD
* Ability to promote image, uploading it to the image cache server
* We can leave the current implementation in bash, but work on which
job type combination will activate the upload
* Quickstart: not needed, can be handled up the current script
* INTROSPECT
* perform overcloud nodes introspection
* Quickstart: supported, but we are still performing bulk
introspection while we should use new format as in
http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/tripleo.sh#n608
instead of
https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-images/templates/overcloud-prep-images.sh.j2#L90
* [card]
* METRICS
* Tripleo ci is sprinkled with metric sections, surrounded with
start_metric - stop_metric primitives that gather section duration
informations throughout various steps of the deployment. (they really
just set timers). This metrics are then sent to graphite host for graph
rendering at the end of the run
* Quickstart: not needed
* MULTINODE_SETUP
* MULTINODE_NODES_BOOTSTRAP
* multiple nodes are consumed from openstack nodes pool
* A setup to create a network between nodepool nodes is needed
* All nodes must contain proper nodepool configurations in /etc/nodepool
* Nodepool nodes are not created starting from overcloud images, so
they have to be aligned to the overcloud image content before starting
the deployment
* MULTINODE_ENV_MULTINODE
* multiple uses multinode environment from
tripleo-ci/test-environments
* Quickstart: to be done in configuration step
* [card]
* MULTINODE_ENV_JOBTYPE
* use multinode environment from tht/ci/environments[b][c]
* Quickstart: not supported, not sure if needed or we can leave the
implementation to the bash scripts; we should pull in the bash script
part in quickstart -> [card]
* NETISO_MULTINIC
* request network environment with basic multiple nic
* Quickstart: [d]supported, OVB jobs are already doing netiso-multinic
* NETISO_PUBLICBOND[e][f][g]
* request network environment like "multi-nic" but with bonded
public interfaces
* Quickstart: [h][i]not needed
* IPV4
* deploy using ipv4
* Quickstart: supported
* IPV6
* deploy using ipv6
* Quickstart: supported
* OSINFRA
* Base for multinode jobs
* If present without MULTINODE a single node is consumed from
openstack nodes pool: This is only used in undercloud only testing
* Overrides net config in undercloud.conf[j][k]
* Quickstart: needed for undercloud-only jobs, job not implemented
* OVB
* nodes are consumed from “openstack virtual baremetal” host
environments e.g. rh1, qeos7, rdo-cloud
* Quickstart: supported
* OVERCLOUD_PACEMAKER
* deploy overcloud to use pacemaker
* Quickstart: supported
* OVERCLOUD_PACEMAKER_CHECK
* This feature check the availability of openstack-heat-api and
openstack-heat-engine before proceeding with validation
* Quickstart: not needed anymore
* OVERCLOUD_PREDICTABLE_PLACEMENT
* allow to choose IPs to be placed on node network interfaces using
nova scheduler hints
* Quickstart: not supported, [bug]
* OVERCLOUD_HOSTNAME_MAP (depends on predictable placement)
* This features tests HostnameMap directive that maps arbitrary
names to default host names, and checks if they are respected.
* Quickstart: not supported, [bug]
* OVERCLOUD_IMAGE_UPDATE
* This feature gives the ability to update an overcloud image before
use it as a base for undercloud nodes[l]
* Quickstart: supported on master, but we should use repo setup role
instead of hardcoded repos.
* OVERCLOUD_MAJOR_UPGRADE[m]
* Quickstart: work in progress
* OVERCLOUD_NO_DEPLOY
* does not deploy the overcloud
* Quickstart: supported with tags
* OVERCLOUD_DELETE (contains branch specific steps)
* launch openstack overcloud delete <stack>
* Quickstart: supported, not thoroughly tested, especially node logs
collection before deletion
* OVERCLOUD UPDATE
* perform overcloud packages update, this is just a deploy command
that includes
tripleo-heat-templates/overcloud-resource-registry-puppet.yml after
updating undercloud tripleo-heat-templates package[n][o][p]
* Quickstart: not supported
* OVERCLOUD_SSL
* deploy overcloud to support ssl
* Quickstart: mostly supported[q], need to check certificate presence
* OVERCLOUD_VALIDATE_PING
* Quickstart: supported but we should use template from THT - [card]
* OVERCLOUD_VALIDATE_TEMPEST_SMOKE
* Quickstart: supported, runs already on the OVB jobs
* OVERCLOUD_VALIDATE_TEMPEST_FULL
* Quickstart: supported, runs in the periodic OVB jobs
* OVERCLOUD_VALIDATE_SANITY_CHECK
* Validation performed after an overcloud upgrade
* Quickstart: not supported, and should be responsibility of upgrade
team implementing this in quickstart[r]
* PROJECT_PIN
* PROJECT_TEMPREVERT
* PROJECT_CHERRYPICK
* These three features give the jobs the ability to select specific
commits for any project to be applied to the delorean package build
process, typically as a workaround for long standing bugs.
* They automatically detects when a bug is solved, and skip the
operation
* Quickstart: not supported - if not offering the same functions,
modifications to the documentation are needed
* Revert and cherry pick can be handled by creating proper changes
* Pinning can be implemented in the build-test-packages as a new
variable that checks out repos at a certain hash
* RELEASE_NEWTON
* check out just newton
* RELEASE_MASTER
* check out just master
* RELEASE_MITAKA
* check out just mitaka
* RELEASE_OCATA
* check out just ocata
* Quickstart: add ocata.yml to release config under
tripleo-quickstart/config/release/tripleoci/
* RELEASE_AGNOSTIC (usually have no suffix, but there are exceptions)
* checks out ZUUL_BRANCH and works well with any branch
* Quickstart: not needed, should be implemented in the configuration
scripts
* UNDERCLOUD
* Installs the undercloud
* Quickstart: widely supported, may be missing /etc/hosts correct
handling like in
http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/tripleo.sh#n496,
low priority, open a bug
* UNDERCLOUD_SSL
* install undercloud with ssl support
* Quickstart: supported, incompatible with multinode even in
upstream, so it’s ok
* UNDERCLOUD_HEAT_CONVERGENCE
* This is a feature tested against openstack/heat and tests a deploy
using the experimental heat convergence engine (just adds a override in
undercloud hiera)
* Currently disabled in undercloud because of high memory usage,
should be enabled in overcloud
* Quickstart: not supported, open a bug.
* UNDERCLOUD_SANITY_CHECK
* checks if essential undercloud services are running
* Quickstart: supported, not thoroughly tested
* UNDERCLOUD_IDEMPOTENT
* installs undercloud twice
* Quickstart: supported, not thoroughly tested, must be properly
placed in a new job
* UNDERCLOUD_MAJOR_UPGRADE
* Stops all services and ovs and issue and openstack undercloud
upgrade command
* Quickstart: not supported.
* UNDERCLOUD_TELEMETRY
* install telemetry packages and legacy api
* Quickstart: not supported. It’s possible that this is enabled by
default, so any implementation should take into account that we need to
ensure that undercloud conf is setting the flag to FALSE if this is
disabled.
* UNDERCLOUD_FREEIPA
* Installs and activate freeipa components
* Sets up CA server in extra node
* Quickstart: supported, not working. ask oooq implementation status
details to Juan Antonio Osorio Robles, not sure how urgent this is, open
a bug
* UNDERCLOUD_UI
* Enables ui, just a flag in undercloud.conf
* Quickstart: not supported. It’s possible that this is enabled by
default, so any implementation should take into account that we need to
ensure that undercloud conf is setting the flag to FALSE if this is
disabled.
* UNDERCLOUD_VALIDATIONS
* Enables validations, just a flag in undercloud.conf, usually
enabled together with UI
* Quickstart: not supported. It’s possible that this is enabled by
default, so any implementation should take into account that we need to
ensure that undercloud conf is setting the flag to FALSE if this is
disabled.
Proposed new Quickstart jobs
Note: We should try to maintain the run time of each gate job as low as
possible, so don’t put too many features in just one job. Periodic jobs
can take instead all the time needed.
This is a list of toci_jobtype (or type: in
project-config/jenkins/jobs/projects.yaml)
BRACHES will probably be handled in the same way as tripleo-ci
* OVB FEATURESET001 - parser will add conf for OVB and eventually periodic
* As a gate job
* -e foo -R ZUUL_BRANCH
* As a periodic (or gate for the next-release branches)
* -e bar -R OVERRIDE_BRANCH[s][t][u][v][w][x][y]
* quickstart.sh -n 3ctrl-1comp.yaml -c featureset001.yaml -e
net-iso-type=multi-nic
* INTROSPECT - Step_introspect: true
* NETISO_MULTINIC - Network_isolation: true
* IN OVB USE MULTI_NIC
* IN LIBVIRT USE SINGLE _NIC_VLAN
* UNDERCLOUD_SSL
* OVERCLOUD_VALIDATE_PING_TEST
* OVERCLOUD_PACEMAKER
* IPV4
* OVB FEATURESET002
* quickstart.sh -n 1ctrl-1comp.yaml -c featureset002.yaml -e
net-iso-type=multi-nic
* IPV6
* NETISO_MULTINIC (should be PUBLICBOND when implemented)
* OVERCLOUD_SSL
* UNDERCLOUD_VALIDATIONS
* UNDERCLOUD_UI
* OVERCLOUD_VALIDATE_PING_TEST
* MULTINODE SCENARIO001 FEATURESET003
* DISABLE_IRONIC
* 1-CONTROLLER
* 1-COMPUTE
* SCENARIO001
* UNDERCLOUD_IDEMPOTENT
* HOSTMAP
* MULTINODE SCENARIO002 FEATURESET004
* UNDERCLOUD_MAJOR_UPGRADE
* UNDERCLOUD_SANITY_CHECK
* MULTINODE FEATURESET005
* CONTAINERS
* UNDERCLOUD_SANITY_CHECK
JOB EQUIVALENCE (by order of importance)
TODO: review this list, bar all the jobs we don't need anymore,
redistribute features over remaining jobs
Check/gate jobs
1. gate-tripleo-ci-centos-7-ovb-ha
* oooq-job: gate-tripleo-ci-centos-7-ovb-ha-oooq-nv
* current features:
* OVB
* 3-CONTROLLERS
* 1-COMPUTE
* IPV4
* NETISO_MULTINIC
* OVERCLOUD_PACEMAKER
* RELEASE_AGNOSTIC
* OVERCLOUD_VALIDATE_PING
1. gate-tripleo-ci-centos-7-ovb-nonha
* oooq-job: gate-tripleo-ci-centos-7-ovb-nonha-oooq-nv
* Current features
* OVB
* 1-CONTROLLER
* 1-COMPUTE
* 1-CEPH
* IPV4
* NETISO_MULTINIC
* INTROSPECT
* UNDERCLOUD_SSL
* UNDERCLOUD_TELEMETRY
* UNDERCLOUD_UI
* UNDERCLOUD_VALIDATIONS
* RELEASE_AGNOSTIC
* OVERCLOUD_VALIDATE_PING
1. gate-tripleo-ci-centos-7-ovb-updates
* oooq-job:
* current features
* OVB
* 1-CONTROLLER
* 1-COMPUTE
* IPV6
* NETISO_MULTINIC
* OVERCLOUD_SSL
* OVERCLOUD_PACEMAKER
* OVERCLOUD_UPDATE
* RELEASE_AGNOSTIC
* OVERCLOUD_VALIDATE_PING
* oooq features
1. gate-tripleo-ci-centos-7-nonha-multinode
* oooq-job: gate-tripleo-ci-centos-7-nonha-multinode-oooq-nv
* current features:
* MULTINODE
* OSINFRA
* 1-CONTROLLER
* UNDERCLOUD_UI
* UNDERCLOUD_TELEMETRY
* UNDERCLOUD_VALIDATIONS
1. gate-tripleo-ci-centos-7-nonha-multinode-updates-nv
* oooq-job:
* current features
* MULTINODE
* OSINFRA
* 1-CONTROLLER
* UNDERCLOUD_UI
* UNDERCLOUD_TELEMETRY
* UNDERCLOUD_VALIDATIONS
* OVERCLOUD_UPDATE
* oooq features
1. gate-tripleo-ci-centos-7-scenario001-multinode
1. oooq-job:
1. gate-tripleo-ci-centos-7-scenario002-multinode
1. oooq-job:
1. gate-tripleo-ci-centos-7-scenario003-multinode
1. oooq-job:
1. gate-tripleo-ci-centos-7-scenario004-multinode[z]
1. oooq-job:
1. gate-tripleo-ci-centos-7-ovb-ha-newton
[aa]
* oooq-job:
* current features
* OVB
* 3-CONTROLLERS
* 1-COMPUTE
* IPV4
* NETISO_MULTINIC
* OVERCLOUD_PACEMAKER
* RELEASE_NEWTON
* OVERCLOUD_VALIDATE_PING
1. gate-tripleo-ci-centos-7-ovb-nonha-newton
[ab]
* oooq-job:
* current features
* OVB
* 1-CONTROLLER
* 1-COMPUTE
* IPV4
* NETISO_MULTINIC
* INTROSPECT
* UNDERCLOUD_SSL
* UNDERCLOUD_TELEMETRY
* UNDERCLOUD_UI
* UNDERCLOUD_VALIDATIONS
* RELEASE_NEWTON
* OVERCLOUD_VALIDATE_PING
1. gate-tripleo-ci-centos-7-ovb-ha-mitaka
1. Oooq-job:
* current features:
[ac]
* OVB
* 3-CONTROLLERS
* 1-COMPUTE
* IPV4
* NETISO_MULTINIC
* OVERCLOUD_PACEMAKER
* RELEASE_MITAKA
* OVERCLOUD_VALIDATE_PING
1. gate-tripleo-ci-centos-7-ovb-nonha-mitaka[ad]
* oooq-job:
* current features
* OVB
* 1-CONTROLLER
* 1-COMPUTE
* IPV4
* NETISO_MULTINIC
* INTROSPECT
* UNDERCLOUD_SSL
* UNDERCLOUD_TELEMETRY
* UNDERCLOUD_UI
* UNDERCLOUD_VALIDATIONS
* RELEASE_MITAKA
* OVERCLOUD_VALIDATE_PING
1. gate-tripleo-ci-centos-7-undercloud
* oooq-job:
* current features
* OSINFRA
* UNDERCLOUD_SSL
* UNDERCLOUD_UI
* UNDERCLOUD_VALIDATIONS
* UNDERCLOUD_TELEMETRY
* UNDERCLOUD_SANITY_CHECK
* OVERCLOUD_NO_DEPLOY
* OVERCLOUD_VALIDATE_NONE
1. gate-tripleo-ci-centos-7-undercloud-upgrades-nv
* oooq-job:
* current features
* OSINFRA
* UNDERCLOUD_SSL
* UNDERCLOUD_UI
* UNDERCLOUD_VALIDATIONS
* UNDERCLOUD_TELEMETRY
* OVERCLOUD_NO_DEPLOY
* OVERCLOUD_VALIDATE_NONE
* UNDERCLOUD_MAJOR_UPGRADE
* UNDERCLOUD_SANITY_CHECK
1. gate-tripleo-ci-centos-7-3nodes-multinode-nv
* Oooq_job
* current features
* MULTINODE
* OSINFRA
* 1-CONTROLLER
* 1-COMPUTE
* OVERCLOUD_PACEMAKER
1. gate-tripleo-ci-centos-7-multinode-upgrades-nv
* oooq-job
* current features
* MULTINODE
* OSINFRA
* 1-CONTROLLER
* OVERCLOUD_MAJOR_UPGRADE
* OVERCLOUD_VALIDATION_SANITY_CHECKS
Periodic jobs
1. periodic-tripleo-ci-centos-7-ovb-ha
* oooq-job:
* current features
* OVERCLOUD_DELETE
* oooq features
1. periodic-tripleo-ci-centos-7-ovb-ha-tempest
1. oooq-job: periodic-tripleo-ci-centos-7-ovb-nonha-tempest-oooq
1. periodic-tripleo-ci-centos-7-ovb-nonha
* oooq-job:
* current features
* UNDERCLOUD_IDEMPOTENT
* oooq features
1. periodic-tripleo-ci-centos-7-ovb-updates
* oooq-job:
* current features
* OVB
* 1-CONTROLLER
* 1-COMPUTE
* IPV6
* NETISO_MULTINIC
* OVERCLOUD_SSL branch specific configuration
* OVERCLOUD_PACEMAKER
* OVERCLOUD_UPDATE
* oooq features
1. periodic-tripleo-ci-centos-7-ovb-ha-newton
* ooo-job:
* current features
* OVERCLOUD_DELETE
1. Periodic-tripleo-ci-centos-7-ovb-ha-mitaka
* Ooo-job:
* current features
* OVERCLOUD_DELETE
1. periodic-tripleo-ci-centos-7-ovb-ha-ocata (present at time of transition)
* ooo-job:
* current features
* OVERCLOUD_DELETE
1. periodic-tripleo-ci-centos-7-scenario001-multinode
* current features
* oooq features
1. periodic-tripleo-ci-centos-7-scenario002-multinode
* current features
* oooq features
1. periodic-tripleo-ci-centos-7-scenario003-multinode
* current features
* oooq features
1. Periodic-tripleo-ci-centos-7-scenario004-multinode
* current features
* oooq features
1. periodic-tripleo-ci-centos-7-promote-ovb-ha
* oooq job:
* current features
* OVERCLOUD_DELETE
* oooq features
1. periodic-tripleo-ci-centos-7-promote-ovb-nonha
* current features
* UNDERCLOUD_IDEMPOTENT
* oooq features
Experimental jobs
1. gate-tripleo-ci-centos-7-ovb-ha-ipv6
1. oooq-job:
2. current features
1. OVB
2. 3-CONTROLLERS
3. 1-COMPUTE
4. IPV6
5. NETISO_MULTINIC
6. OVERCLOUD_PACEMAKER
7. RELEASE_MASTER
8. OVERCLOUD_VALIDATE_PING
1. 2. gate-tripleo-ci-centos-7-ovb-ha-tempest
1. oooq-job: every oooq job runs tempest, so:
gate-tripleo-ci-centos-7-ovb-ha-oooq-nv
2. current features
1. OVB
2. 3-CONTROLLERS
3. 1-COMPUTE
4. IPV4
5. NETISO_MULTINIC
6. OVERCLOUD_PACEMAKER
7. RELEASE_MASTER
8. OVERCLOUD_VALIDATE_TEMPEST_SMOKE?
1. gate-tripleo-ci-centos-7-undercloud-containers
1. oooq-job:
2. current features
1. OSINFRA
2. CONTAINERS
3. UNDERCLOUD_SSL
4. UNDERCLOUD_SANITY_CHECK
5. OVERCLOUD_NO_DEPLOY
6. OVERCLOUD_VALIDATE_NONE
1. gate-tripleo-ci-centos-7-ovb-fakeha-caserver
1. Related to Security DFG work -- adding Free IPA server to
deployments for auth testing
1. (is developed by Juan): https://review.openstack.org/#/c/391738/
2. https://review.openstack.org/#/q/topic:bp/tls-via-certmonger
1. gate-tripleo-ci-centos-7-ovb-nonha-convergence
Jobs that don’t need transition
1. gate-tripleo-ci-centos-7-ovb-nonha-puppet-mitaka
2. gate-tripleo-ci-centos-7-ovb-nonha-puppet-newton
3. gate-tripleo-ci-centos-7-scenario001-multinode-puppet
1. this is a clone of classic scenario001-multinode, but it runs
only in Puppet OpenStack CI. We had to duplicate it because of specific
files rules on zuul layout.
1. gate-tripleo-ci-centos-7-scenario001-multinode-puppet-nv
1. ditto
1. gate-tripleo-ci-centos-7-scenario002-multinode-puppet
1. ditto
1. gate-tripleo-ci-centos-7-scenario003-multinode-puppet
1. ditto
1. gate-tripleo-ci-centos-7-scenario004-multinode-puppet
1. ditto
1. gate-tripleo-ci-centos-7-scenario005-multinode
PROMOTION JOBS
* ensure the promotion jobs are an accurately reflecting the check gates
and providing coverage
LOGGING EQUIVALENCE
* console log - compare usability
* postci.txt
* https://review.openstack.org/#/c/413224/
*
https://trello.com/c/91FQaapF/516-q-to-u-add-the-equivalent-for-postci-information
* It’s one big file used for everything, should we do the same?
* rpm log
* Nova list ( https://review.openstack.org/#/c/428187/ )
* Hostinfo.txt: https://review.openstack.org/#/c/413224/
* journal-text.txt
* Package list is sorted alphabetically ? (very useful in case a package
diff is needed )
* undercloud
* etc
* var
* home
* overcloud nodes
* etc
* var
Log position equivalence (try to put everything on the same place
(underloud/var/logs/oooq/extras/… may not be a good place)
TOCI review
* review oooq changes to original toci scripts
* review new oooq toci scripts
* email changes to openstack-dev[tripleo] with summary of changes
FEATURES TO REVIEW
* containers
* composable roles
More information about the OpenStack-dev
mailing list