Open Stack

On Thu, Jul 24, 2014 at 3:54 PM, Sean Dague <sean at dague.net> wrote:

> On 07/24/2014 05:57 PM, Matthew Treinish wrote:
> > On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote:
> >> OpenStack has a substantial CI system that is core to its development
> >> process.  The goals of the system are to facilitate merging good code,
> >> prevent regressions, and ensure that there is at least one configuration
> >> of upstream OpenStack that we know works as a whole.  The "project
> >> gating" technique that we use is effective at preventing many kinds of
> >> regressions from landing, however more subtle, non-deterministic bugs
> >> can still get through, and these are the bugs that are currently
> >> plaguing developers with seemingly random test failures.
> >>
> >> Most of these bugs are not failures of the test system; they are real
> >> bugs.  Many of them have even been in OpenStack for a long time, but are
> >> only becoming visible now due to improvements in our tests.  That's not
> >> much help to developers whose patches are being hit with negative test
> >> results from unrelated failures.  We need to find a way to address the
> >> non-deterministic bugs that are lurking in OpenStack without making it
> >> easier for new bugs to creep in.
> >>
> >> The CI system and project infrastructure are not static.  They have
> >> evolved with the project to get to where they are today, and the
> >> challenge now is to continue to evolve them to address the problems
> >> we're seeing now.  The QA and Infrastructure teams recently hosted a
> >> sprint where we discussed some of these issues in depth.  This post from
> >> Sean Dague goes into a bit of the background: [1].  The rest of this
> >> email outlines the medium and long-term changes we would like to make to
> >> address these problems.
> >>
> >> [1] https://dague.net/2014/07/22/openstack-failures/
> >>
> >> ==Things we're already doing==
> >>
> >> The elastic-recheck tool[2] is used to identify "random" failures in
> >> test runs.  It tries to match failures to known bugs using signatures
> >> created from log messages.  It helps developers prioritize bugs by how
> >> frequently they manifest as test failures.  It also collects information
> >> on unclassified errors -- we can see how many (and which) test runs
> >> failed for an unknown reason and our overall progress on finding
> >> fingerprints for random failures.
> >>
> >> [2] http://status.openstack.org/elastic-recheck/
> >>
> >> We added a feature to Zuul that lets us manually "promote" changes to
> >> the top of the Gate pipeline.  When the QA team identifies a change that
> >> fixes a bug that is affecting overall gate stability, we can move that
> >> change to the top of the queue so that it may merge more quickly.
> >>
> >> We added the clean check facility in reaction to the January gate break
> >> down. While it does mean that any individual patch might see more tests
> >> run on it, it's now largely kept the gate queue at a countable number of
> >> hours, instead of regularly growing to more than a work day in
> >> length. It also means that a developer can Approve a code merge before
> >> tests have returned, and not ruin it for everyone else if there turned
> >> out to be a bug that the tests could catch.
> >>
> >> ==Future changes==
> >>
> >> ===Communication===
> >> We used to be better at communicating about the CI system.  As it and
> >> the project grew, we incrementally added to our institutional knowledge,
> >> but we haven't been good about maintaining that information in a form
> >> that new or existing contributors can consume to understand what's going
> >> on and why.
> >>
> >> We have started on a major effort in that direction that we call the
> >> "infra-manual" project -- it's designed to be a comprehensive "user
> >> manual" for the project infrastructure, including the CI process.  Even
> >> before that project is complete, we will write a document that
> >> summarizes the CI system and ensure it is included in new developer
> >> documentation and linked to from test results.
> >>
> >> There are also a number of ways for people to get involved in the CI
> >> system, whether focused on Infrastructure or QA, but it is not always
> >> clear how to do so.  We will improve our documentation to highlight how
> >> to contribute.
> >>
> >> ===Fixing Faster===
> >>
> >> We introduce bugs to OpenStack at some constant rate, which piles up
> >> over time. Our systems currently treat all changes as equally risky and
> >> important to the health of the system, which makes landing code changes
> >> to fix key bugs slow when we're at a high reset rate. We've got a manual
> >> process of promoting changes today to get around this, but that's
> >> actually quite costly in people time, and takes getting all the right
> >> people together at once to promote changes. You can see a number of the
> >> changes we promoted during the gate storm in June [3], and it was no
> >> small number of fixes to get us back to a reasonably passing gate. We
> >> think that optimizing this system will help us land fixes to critical
> >> bugs faster.
> >>
> >> [3] https://etherpad.openstack.org/p/gatetriage-june2014
> >>
> >> The basic idea is to use the data from elastic recheck to identify that
> >> a patch is fixing a critical gate related bug. When one of these is
> >> found in the queues it will be given higher priority, including bubbling
> >> up to the top of the gate queue automatically. The manual promote
> >> process should no longer be needed, and instead bugs fixing elastic
> >> recheck tracked issues will be promoted automatically.
> >>
> >> At the same time we'll also promote review on critical gate bugs through
> >> making them visible in a number of different channels (like on elastic
> >> recheck pages, review day, and in the gerrit dashboards). The idea here
> >> again is to make the reviews that fix key bugs pop to the top of
> >> everyone's views.
> >>
> >> ===Testing more tactically===
> >>
> >> One of the challenges that exists today is that we've got basically 2
> >> levels of testing in most of OpenStack: unit tests, and running a whole
> >> OpenStack cloud. Over time we've focused on adding more and more
> >> configurations and tests to the latter, but as we've seen, when things
> >> fail in a whole OpenStack cloud, getting to the root cause is often
> >> quite hard. So hard in fact that most people throw up their hands and
> >> just run 'recheck'. If a test run fails, and no one looks at why, does
> >> it provide any value?
> >>
> >> We need to get to a balance where we are testing that OpenStack works as
> >> a whole in some configuration, but as we've seen, even our best and
> >> brightest can't seem to make OpenStack reliably boot a compute that has
> >> working networking 100% the time if we happen to be running more than 1
> >> API request at once.
> >>
> >> Getting there is a multi party process:
> >>
> >>   * Reduce the gating configurations down to some gold standard
> >>     configuration(s). This will be a small number of configurations that
> >>     we all agree that everything will gate on. This means things like
> >>     postgresql, cells, different environments will all get dropped from
> >>     the gate as we know it.
> >>
> >>   * Put the burden for a bunch of these tests back on the projects as
> >>     "functional" tests. Basically a custom devstack environment that a
> >>     project can create with a set of services that they minimally need
> >>     to do their job. These functional tests will live in the project
> >>     tree, not in Tempest, so can be atomically landed as part of the
> >>     project normal development process.
> >>
> >>   * For all non gold standard configurations, we'll dedicate a part of
> >>     our infrastructure to running them in a continuous background loop,
> >>     as well as making these configs available as experimental jobs. The
> >>     idea here is that we'll actually be able to provide more
> >>     configurations that are operating in a more traditional CI (post
> >>     merge) context. People that are interested in keeping these bits
> >>     functional can monitor those jobs and help with fixes when needed.
> >>     The experimental jobs mean that if developers are concerned about
> >>     the effect of a particular change on one of these configs, it's easy
> >>     to request a pre-merge test run.  In the near term we might imagine
> >>     this would allow for things like ceph, mongodb, docker, and possibly
> >>     very new libvirt to be validated in some way upstream.
> >>
> >>   * Provide some kind of easy to view dashboards of these jobs, as well
> >>     as a policy that if some job is failing for > some period of time,
> >>     it's removed from the system. We want to provide whatever feedback
> >>     we can to engaged parties, but people do need to realize that
> >>     engagement is key. The biggest part of putting tests into OpenStack
> >>     isn't landing the tests, but dealing with their failures.
> >>
> >>   * Encourage projects to specifically land interface tests in other
> >>     projects when they depend on certain behavior.
> >
> > So I think we (or least I do) need clarification around this item. My
> question
> > is which interfaces are we depending on that need these specific types of
> > tests? Projects shouldn't be depending on another project's unstable
> interfaces.
> > If specific behavior is required for a cross-project interaction it
> should be
> > part of defined stable API, hopefully the REST API, and then that
> behavior
> > should be enforced for everyone not just the cross-project interaction.
> >
> > If I'm interpreting this correctly the what is actually needed here is to
> > actually ensure that there is test coverage somewhere for the APIs that
> should
> > already be tested where there is a cross-project dependency. This is
> actually
> > the same thing we see all the time because there is a lack of test
> coverage
> > on certain APIs that are being used. (the nova default quotas example
> comes to
> > mind) I just think calling this a special class of test is a bit
> misleading.
> > Since it shouldn't actually differ than any other API test. Or am I
> missing
> > something?
>
> Projects are consuming the behavior of other projects far beyond just
> the formal REST APIs. Notifications is another great instance of that.
>

I think the fact that notification aren't versioned or really 'contractual'
but are being used as such to be a huge issue.


>
> This is also more of a pragmatic organic approach to figuring out the
> interfaces we need to lock down. When one projects breaks depending on
> an interface in another project, that should trigger this kind of
> contract growth, which hopefully formally turns into a document later
> for a stable interface.
>

This approach sounds like a recipe for us playing a never ending game of
catch up, when issues like this have arrisen in the past we rarely get
around to creating a stable interface (see notifications). I would rather
push on using stable APIs (and creating them as needed) instead.


>
> >> Let's imagine an example of how this works in the real world.
> >>
> >>   * The heat-slow job is deleted.
> >>
> >>   * The heat team creates a specific functional job which tests some of
> >>     their deeper function in Heat, all the tests live in Heat, and
> >>     because of these the tests can include white/grey box testing of the
> >>     DB and queues while things are progressing.
> >>
> >>   * Nova lands a change which neither Tempest or our configs exercise,
> >>     but breaks Heat.
> >>
> >>   * The Heat project can now decide if it's more important to keep the
> >>     test in place (preventing them from landing code), or to skip it to
> >>     get back to work.
> >>
> >>   * The Heat team then works on the right fix for Nova, or communicates
> >>     with the Nova team on the issue at hand. The fix to Nova *also*
> >>     should include tests which locks down that interface so that Nova
> >>     won't break them again in the future (the ironic team did this with
> >>     their test_ironic_contract patch). These tests could be unit tests,
> >>     if they are testable that way, or functional tests in the Nova tree.
> >
> > The one thing I want to point out here is that ironic_contract test
> should be
> > an exception, I don't feel that we want that to be the norm. It's not a
> good
> > example for a few reasons, mostly around the fact that ironic tree
> depends on
> > the purposefully unstable nova driver api as temporary measure until the
> ironic
> > driver is merged into the nova tree. The contract api tests will go away
> once
> > the driver is in the nova tree. It should not be necessary for something
> over
> > the REST API, since the contact should be enforced through tempest.
> (even under
> > this new model, I expect this to still be true)
> >
> > There was that comment which someone (I can't remember who) brought up
> at the
> > Portland summit that tempest is acting like double book accounting for
> the api
> > contract, and that has been something we've seen as extremely valuable
> > historically. Which is why I don't want to see this aspect of tempest's
> role in
> > the gate altered.
>
> I've been the holder of the double book accounting pov in the past.
> However, after the last six months of fragility, I just don't see how
> that's a sustainable point of view. The QA team remains somewhat
> constant size, and the number of interfaces and projects grows at a good
> clip.
>
> > Although, all I think we actually need is an api definition for testing
> in an
> > external repo, just to prevent inadvertent changes. (whether that gets
> used in
> > tempest or not) So another alternative I see here is something that I've
> started
> > to outline in [4] to address the potential for code duplication and
> effort in
> > the new functional test suites. If all the project specific functional
> tests are
> > using clients from an external functional testing library repo then this
> concern
> > goes away.
>
> Actually, I don't think these wold be using external clients. This is in
> tree testing.
>
> This will definitely be an experiment to get the API testing closer to
> the source. That being said, Swift really has done this fine for a long
> time, and I think we need to revisit the premise that projects can't be
> trusted.
>
> > Now, if something like this example were to be exposed because of a
> coverage
> > gap I think it's fair game to have a specific test in nova's functional
> test
> > suite. But, I also think there should be an external audit of that API
> somewhere
> > too. Ideally I think what I'd like to see is probably a write-once test
> > graduation procedure for moving appropriate things into tempest (or
> somewhere
> > else) from the project specific functional tests. Basically like what we
> > discussed during Maru's summit session on Neutron functional testing in
> Atlanta.
>
> Right, and I think basically we shouldn't graduate most of those tests.
> They are neutron tests, in the neutron tree. A few key ones we decide
> should be run outside that context.
>
> > For the other, more social, goal of this step in fostering communication
> between
> > the projects and not using QA and/or Infra as a middle man I fully
> support. I
> > agree that we probably have too proxying going on between projects using
> QA
> > and/or infra instead of necessarily talking directly.
>
> Our current model leans far too much on the idea of the only time we
> ever try to test things for real is when we throw all 1 million lines of
> source code into one pot and stir. It really shouldn't be surprising how
> many bugs shake out there. And this is the wrong layer to debug from, so
> I firmly believe we need to change this back to something we can
> actually manage to shake the bugs out with. Because right now we're
> finding them, but our infrastructure isn't optimized for fixing them,
> and we need to change that.
>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140725/0a936ca9/attachment-0001.html>