Open Stack

Hi, everyone,

<tl;dr>

in Fuel project we run two HA deployment tests for every commit in
openstack/fuel-library repository. Currently these tests are run by
Fuel CI - the standalone Third-Party CI system, which can vote on
openstack/fuel* projects. But we'd like it to be the part of the gate.
As these tests require several vms to be available for the test run,
we'd like to use nodepool multinode support for that.

</tl;dr>

As I don't have deep understanding about multinode setup at the
moment, and I haven't found enough info in docs or specs, I think the
better way to start this discussion is to explain the workflow used in
Fuel HA tests. Then we can consider which of the steps are already
there, or can be implemented, and which items are potential blockers.
Any feedback is welcome.

Generic deployment test
=======================

To run deployment for Fuel you need:
  - ISO image for Fuel node [1]
  - fuel-devops [2] - the cli tool which manages virtual machines and
stores state (vm names, network interfaces..) in a PostgreSQL database
  - fuel-qa [3] - test framework based on proboscis

Basic setup is described in [4].

And the test flow is as follows:

- with fuel-devops tool:

create several vm's connected via internal network - a so called
'devops environment'

- with fuel-qa framework:

1. install Fuel node on first vm using the ISO image provided by the
local path on the host server

2. bootstrap other vms with basic OS image provided on Fuel ISO

3. configure Fuel environment via API according to certain scenario

4. run deployment


Test scenarios are described in fuel-qa documentation, see for example [5].

Deployment test on CI
=====================

fuel-library code is essentially a set of puppet manifests which are
used to deploy the enviroment configuration defined via Fuel
interface. These manifests are delivered to Fuel node as RPM package
[6].

To save time and resources on CI we don't recreate environment from
scratch for every tests but regularly take a "stable enough" ISO,
upload it to Jenkins slaves, create base environment (steps 1. and 2.)
and snapshot all its vms.

Then, on every commit we

1) rebuild a fuel-library package in a CentOS-based docker container

2) revert devops environment from snapshot

3) upload and install package on Fuel node

4) run the deployment test scenario (steps 3. and 4.)


You can refer to detailed logs in [7]

How this can be addressed in nodepool
=====================================

The nodepool driver approach
----------------------------

fuel-devops is essentially a wrapper and vm's manager, and it was
originally planned as a tool which can use multiple backends, taking
libvirt as a default one. There is an still-on-discussion task to
implement the 'bare-metal driver' for fuel-devops, which would make it
possible to use vm's from different servers for one particular test
run.

We can consider implementing nodepool as a driver, so it provides
vm's, which then are wrapped by fuel-devops and are sent further to
fuel-qa framework.

Then to run the test we would need a 'manager vm' where fuel-devops
code is executed, and several empty nodes from nodepool. We'd register
those empty nodes in fuel-devops database and run the test as usual.

No fuel-devops approach
-----------------------

Direct approach would be to use the nodepool's service for pre-built images.

Given by Fuel ISO image, we regularly generate one vm with deployed
Fuel node (step 1.), and one with the basic node deployed with
bootstrap image (step 2.). This images are stored in Glance or another
storage as usual.

Then for each fuel-library test we request 1 Fuel node and several
basic nodes, and then operate on them directly without wrappers.

For this scenario to work we need to change a lot in fuel-qa code. But
this approach seems to be aligned with the initiative [8], which is
currently in development: if we manage to describe devops environments
in YAML files, we'd probably be able to map these descriptions to the
multinode configurations, and then support them in nodepool.

Side Question
=============

Can we build package from a change request so that package is then
used in test? Are there any best practices?


P.S. All code used for current implementation of deployment tests in
Fuel CI is open, including puppet manifests for Jenkins master and
slaves and Jenkins Job Builder configs for actual jobs. See links
below.

[1] https://ci.fuel-infra.org/view/ISO/ - nightly ISO builds
[2] https://github.com/openstack/fuel-devops
[3] https://github.com/openstack/fuel-qa
[4] https://docs.fuel-infra.org/fuel-dev/devops.html - test environment setup
[5] https://docs.fuel-infra.org/fuel-qa/base_tests.html#module-fuelweb_test.tests.test_neutron
- test scenario
[6] https://github.com/openstack/fuel-library/blob/master/specs/fuel-library8.0.spec
[7] https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/2699/console
- test run example
[8] https://review.openstack.org/#/c/240170/ - YAML-based templates
for devops environments

Fuel CI links:

[A] https://github.com/fuel-infra/jenkins-jobs/tree/master/servers/fuel-ci
- JJB configs
[B] https://github.com/fuel-infra/puppet-manifests/tree/master/modules/jenkins/manifests
- Puppet manifests
[C] https://ci.fuel-infra.org - Jenkins instance


-- 
Aleksandra Fedorova
Fuel CI Team Lead
bookwar