Open Stack

Hey everyone!

During the Newton cycle, we have added a new repository: 
tripleo-validations. This is a little guide on how to review the patches 
there.

All the validations are YAML files under the `validations` directory. 
They can optionally add files in `validations/library` (for Ansible 
modules) or `validations/files` for additional files or scripts to run.

Each validation is a self-contained check for something that could 
potentially break the deployment. Some of them verify the undercloud 
hardware, the introspected nodes for overcloud deployment, the heat 
templates, and some post-deployment checks.

Here's a list of things to pay attention to. You can look at this sample 
patch as you follow along:

https://review.openstack.org/#/c/353943/

* Check that the `hosts` section has a known value that fits the validation

This is currently: undercloud, overcloud, compute, controller or all. It 
describes the nodes the validation should run on. We need to update this 
to support the composable/custom roles[1].

You can see the source for these labels by running 
`scripts/tripleo-ansible-inventory --list` from the repo on the 
undercloud. You need to run `source stackrc` first.

* The `vars/metadata/name` section must exist and have a unique and 
descriptive name -- it should make it clear what the validation is for. 
This is displayed in the UI.

* The `vars/metadata/description` section must exist and describe what 
the validation is going to check and why. Where possible, the name and 
description should be enough to tell you where to look for when the 
validation fails. This is displayed in the UI.

* The `vars/metadata/groups` section should exist and have a list of 
stages throughout the deployment (pre-deployment, pre-introspection, 
post-deployment, etc.). These can be brand-new but they should make 
sense given what the validation is supposed to be doing.

* There may be additional `vars` after these -- those are variables that 
are going to be used within the validation steps.[2]

* The `tasks` section is a list of steps that Ansible is going to 
perform. They gather data, process it and verify what needs to happen.

Each task should have a name. It must also have a module name that's 
either the official Ansible module[3] or a custom one from the 
`validations/library` directory.

In addition, a task may also have a `failed_when` condition, possibly a 
`fail` message explaining the error, or a `register` section which takes 
the task's output and stores it in a variable available for late tasks.


* Ansible has three modules that seem to invoke a custom command, but 
each works differently: `command`, `shell` and `script`:

`command` invokes a single command with arguments you pass in. It 
doesn't do multiple commands, pipes or redirection. If you need any of 
those, use the `shell` module.

The `script` module takes a script on the local compute (usually from 
the  `validations/files` directory) copies it to the host and executes 
it there.

Oh and finally, if you need to run a shell builtin, you'll have to 
execute the shell manually (e.g.: `command: sh -c "ulimit -n"`)

When reviewing, if you see any of those, make sure the correct module is 
used.

* A task may rely on a custom module (i.e. one that doesn't come with 
Ansible but we ship it directly ourselves). These modules live in 
`validations/library` and the file name without the extension must be 
the same as the module invocation.


Testing validations
-------------------

If you have set `enable_validations` in `undercloud.conf` before 
installing the undercloud, all dependdencies, etc. should be ready.

Otherwise you need to do this on the undercloud:
1. install the Ansible package (yum install ansible)
2. sudo useradd validations
3. mistral execution-create tripleo.validations.v1.copy_ssh_key


After that, to test a validation:
1. clone the repo: git clone 
https://git.openstack.org/openstack/tripleo-validations
2. apply the patch
3. source stackrc
4. run-validation /path/to/validations/some-validation.yaml ~/.ssh/id_rsa

run-validation is a script that takes the full path to a validation and 
an SSH key to connect to the nodes being validated. This should all be 
available under the stack user in the typical dev environment.

If you wish to run the validation via Mistral, create a context.json file:

     { "validation_name": "some-validation" }

and run:

$ mistral execution-create tripleo.validations.v1.run_validation 
context.json

Note that for this to work, the validation must be under:

/usr/share/openstack-tripleo-validations/validations/some-validation.yaml

After that, call `mistral execution-get-output` to see the output of the 
validation.


Additional notes
----------------

* Documentation:

We have http://docs.openstack.org/developer/tripleo-validations/ which 
lists the existing validations. And the info on how to enable & run 
validations is in the tripleo-common readme, but all of this should be 
exposed in tripleo.org.

I'll try to get to that in the coming weeks.

* We do not test any of the validations in the CI. That needs to change.

* If possible, the deployers shouldn't have to run the validations 
manually at the right time. They should be run as part of the tripleo 
workflows.

* The validation bugs are stored in the tripleo launchpad[4] with the 
`validations` tag.

If you have any questions or suggestions, don't hesitate to reply here 
or talk to me (shadower) on IRC.

Cheers,
Tomas Sedovic

[1]: https://bugs.launchpad.net/tripleo/+bug/1620617
[2]: http://docs.ansible.com/ansible/playbooks_variables.html
[3]: http://docs.ansible.com/ansible/list_of_all_modules.html
[4]: https://bugs.launchpad.net/tripleo/