Open Stack

Hi everyone,

thanks to everyone who joined the QA track, hands-on session and
discussions at the first PTG in Atlanta.
I think it was a very productive week, it was great to meet so many of you
in person.
We also received a lot of feedback from the community [0].

The list of sessions is available at [1].
The list of priorities for the QA team during the Pike cycle is available
at [2].

Below a summary of the areas we will be working on during Pike.
This is not an exhaustive list of all the topics we discussed at the PTG,
see the session etherpads [1] for more.

Andrea

Tempest Stable Interfaces
====================

Credential providers are widely used in Tempest plugins [3], so they're top
of the list in migrating to stable interfaces.
Tempest test base class is used by more than 45 different repos [3], so
after as little as possible rework we will declare that class "as-is" a
stable interface for plugins.
Other items planned to stable interfaces are: swift service client, waiters
and potentially scenario helper methods.

Tempest Plugins
=============

We discussed about Tempest plugins and the proposed goal [4] to maintain
them in branchless repos.
We clarified the implications of hosting the plugin in a branched repo / in
the project repo [5].

Documentation
============

One recurring items in the feedback was documentation. We will be working
on generating documentation from all Tempest stable interfaces, and to add
examples in there. We will enrich the documentation about writing plugins
and contributing tests as well.

Scenario Tests
===========

Scenario tests have a long due refactor to be done, so we'll work on it
during Pike.
We will be trying to make the process as smooth as possible for teams that
depend on scenario test base classes today.

Policy Testing - Patrole
=================

During the Ocata cycle we introduced a new QA framework - Patrole [6] - for
policy testing.
It's already discovered some inconsistencies in error codes, which is good
news.
We have a lot of work planned on Patrole for the Pike cycle:
- setup CI gates
- make Patrole portable - similar to Tempest - so that it may be used to
test existing deployments (public / private clouds)
- define Patrole interfaces for plugins

Schema Validation
==============

We plan to extend schema validation in Tempest to the six projects hosted
there.
Plugins are encouraged to implement schema validation for the service they
represent as well. No additional plugin interfaces are required for this.

Microversion Testing
================

We will stick to the current plan of updating micro version schemas in
Tempest only when needed.
However we will monitor drift between implemented versions and tested ones,
and we will make sure we fill the gap at least at the end of each cycle.

Grenade and Upgrade Testing
=======================

The current implementation of upgrade testing in Grenade provides a great
service in our gates.
However it is not meant to fit more complex testing scenarios such as
rolling upgrade, zero-downtime testing, validation using custom topologies
and cross project upgrade dependencies.
We will investigate synergies with other projects to deploy topologies
which are interesting for rolling upgrades, as well as for orchestrating
the upgrade process itself, while keeping the existing upgrade test
strategy available and stable via Grenade.

HA / Destructive Testing
==================

Interest on automated HA / Destructive testing has revived - see the email
thread [7].
Starting with a third party CI service may be a good approach to asses the
potential of this for upstream testing.

[0] https://etherpad.openstack.org/p/qa-ptg-pike-community-input
[1] https://etherpad.openstack.org/p/qa-ptg-pike
[2] https://etherpad.openstack.org/p/pike-qa-priorities
[3] http://imgur.com/KDFeAf6
[4] https://review.openstack.org/#/c/369749/
[5] https://etherpad.openstack.org/p/qa-ptg-pike-tempest-plugins
[6] https://github.com/openstack/patrole
[7]
http://lists.openstack.org/pipermail/openstack-dev/2017-February/112848.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170227/54caa707/attachment.html>