Open Stack

On Mon, Mar 21, 2016 at 10:19:47AM -0400, Emilien Macchi wrote:
> On Mon, Mar 21, 2016 at 9:59 AM, Steven Hardy <shardy at redhat.com> wrote:
> > On Mon, Mar 21, 2016 at 09:41:42AM -0400, Emilien Macchi wrote:
> >> On Mon, Mar 21, 2016 at 6:57 AM, Steven Hardy <shardy at redhat.com> wrote:
> >> > On Fri, Mar 18, 2016 at 01:27:33PM +0000, Arkady_Kanevsky at DELL.com wrote:
> >> >>    Emilien,
> >> >>
> >> >>    Agree on the rant. But not clear on concrete proposal to fix it.
> >> >>
> >> >>    Spend more time “fixing” CI and use Tempest as a gate is a bit wage.
> >> >>
> >> >>    Unless we test known working version of each project in TripleO CI you are
> >> >>    dependent on health of other components.
> >> >
> >> > I've so far resisted replying to this thread, because while valid, many of
> >> > the concerns expressed by Emilien are quite general complaints, and it's
> >> > hard to reply with specific solutions.
> >> >
> >> > However work *is* going on to improve many of these problems, let's see if
> >> > I can provide a summary, to clarify the various "concrete proposals" which
> >> > do exist.
> >> >
> >> > 1. Core team & review velocity
> >> >
> >> > We've had a small and very overloaded core team for a while now, and this
> >> > will be helped by expanding our community to include those who've been
> >> > regularly contributing excellent work and reviews as core reviewers:
> >> >
> >> > http://lists.openstack.org/pipermail/openstack-dev/2016-February/087774.html
> >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089235.html
> >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089912.html
> >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089913.html
> >> >
> >> > Note that I personally think it's absolutely fine for folks to be more
> >> > expert in some subsystem and to focus review extra attention on e.g API,
> >> > UI, Puppet or whatever.  This subsystem-core model has been well proven in
> >> > other projects, and folks will naturally broaden their areas of deeper
> >> > knowledge over time.
> >> >
> >> > Related to this is movement of code, such as the puppet-tripleo refactoring
> >> > mentioned by Michael - this has already started, and will help with
> >> > providing a cleaner interface between the puppet and heat pieces (which
> >> > will also help focus reviewer attention appropriately).
> >>
> >> Indeed, Michael, Dan & I are working on moving out the Puppet code
> >> from THT to puppet-tripleo.
> >> That's a nice move, and I appreciate TripleO team support on it.
> >>
> >> > 2. Day 1 developer experience
> >> >
> >> > This is closely related to the CI failure rate - there are efforts to
> >> > integrate with the RDO tripleo-quickstart tooling, which simplifies the
> >> > initial undercloud setup, and potentially makes consuming pre-built,
> >> > validated undercloud images (probably output artefacts from our new
> >> > periodic CI job) much easier.
> >> >
> >> > So, this will mean that both developers and CI can potentially be less
> >> > regularly impacted by trunk regressions which often cause CI to fail, and
> >> > break developer environments.
> >> >
> >> > https://review.openstack.org/#/c/276810/5
> >> >
> >> > 3. CI coverage and trunk failure rate
> >> >
> >> > We've been working really hard to improve things here, which are really
> >> > several inter-related issues:
> >> >
> >> > - Lack of Hardware capacity in the tripleo CI cloud
> >> > - Frequent trunk regressions breaking our CI
> >> > - Lack of coverage of some key features (network isolation, SSL, IPv6, upgrades)
> >> > - Lack of coverage for vendor plugin templates/puppet code
> >> >
> >> > There's work ongoing to improve this from multiple perspectives:
> >> >
> >> > New periodic CI job (to be used for automated promotion of the
> >> > current-tripleo repo, and for pre-built undercloud images):
> >> > https://review.openstack.org/#/c/271370/
> >> >
> >> > Add network isolation support to CI:
> >> > https://review.openstack.org/#/c/288163/
> >> >
> >> > Test SSL enabled in overcloud:
> >> > https://review.openstack.org/#/c/281988/
> >> >
> >> > CI coverage of IPv6:
> >> > https://review.openstack.org/#/c/289445/
> >> >
> >> > Discussion around better documented integration for third-party CI:
> >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/088972.html
> >>
> >> Do we have plans to execute Tempest?
> >
> > This is something which has been discussed several times, but right now we
> > don't have the time available to run it per-commit because we'll hit the
> > job timeout.
> >
> > This situation will improve as we gain time e.g through use of cached
> > pre-built images, but right now I think we could look at enabling it only
> > on the periodic job when that is fully proven.
> >
> > Having said that, I should point out that tempest doesn't get us great
> > coverage of some newer projects - e.g all Heat scenario coverage was moved
> > out of the tempest tree, and other projects have done similar AFAIK, so we
> > may end up with very sparse API surface tests (or nothing at all) in these cases.
> 
> In Puppet OpenStack CI, we execute smoke tests (a few tests of each
> service, and some scenarios), and some tests not in smoke, for Ironic,
> Aodh, Horizon.
> It takes ~15 minutes to execute them.

Thanks, useful datapoint - this kind of time budget may become possible
after we've optimized things like the undercloud image building.

> This is how we proceed:
> https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L116-L134
> 
> I'm asking because to me Tempest is the official project for OpenStack
> testing and if we enable third party CI later, with external CI, we
> need to find a common way to test TripleO clouds.

I guess this is the point of my earlier comment - despite being "the"
official project for OpenStack testing, tempest has very deep testing of
certain projects (e.g Nova) and very shallow (API only) testing for others.

So, while it's probably a good start, it's not a single solution unless you
can configure things to also run per-project functional/integration tests.

Steve