Open Stack

On Mon, Mar 10, 2014 at 12:21 PM, John Dennis <jdennis at redhat.com> wrote:
> On 03/10/2014 02:31 PM, Zane Bitter wrote:
>> Fewer logs is hardly ever what you want when debugging a unit test.
>>
>> I think what John is looking for is a report at the end of each test run
>> that just lists the tests that failed instead of all the details (like
>> `testr failing --list`), or perhaps the complete list of tests with the
>> coloured pass/fail like IIRC nose does. Since testr's killer feature is
>> to helpfully store all of the results for later, maybe this output is
>> all you need in the first instance (along with a message telling the
>> user what command to run to see the full output, of course), or at least
>> that could be an option.
>
>> It sounds like what John wants to do is pass a filter to something like
>> `testr failing` or `testr last` to only report a subset of the results,
>> in much the same way as it's possible to pass a filter to `testr` to
>> only run a subset of the tests.
>
> Common vocabulary is essential to discuss this. A test result as emitted
> by subunit is a metadata collection indexed by keys. To get the list of
> failing tests one iterates over the set of results and looks for the
> absense of the "successful" key in the result. That's the set of test
> failures, as such it's a filter on the test results.
>
> Therefore I see filtering as the act of producing a subset of the test
> results (i.e, only those failing, or only those whose names match a
> regexp, or the intersection of those). That is a filtered result set.
> The filtering is performed by examining the key/value in each result
> metadata to yield a subset of the results.
>
> Next you have to display that filtered result set. When each result is
> displayed one should be able to specify which pieces of metadata get
> displayed. In my mind that's not filtering, it's a display option. One
> common display option would be to emit only the test name. Another might
> be to display the test name and the captured log data. As it stands now
> it seems to display every piece of metadata in the result which is what
> is producing the excessive verbosity.
>
> Hopefully I'm making sense, yes/no?
>
Makes sense to me. We should poke lifeless and get him to chime in.

Clark