Open Stack

On 11/08/14 15:44, Andreas Jaeger wrote:
> On 08/11/2014 09:36 AM, Tom Fifield wrote:
>> On 11/08/14 13:20, Andreas Jaeger wrote:
>>> On 08/11/2014 05:26 AM, Tom Fifield wrote:
>>>> On 10/08/14 22:21, Andreas Jaeger wrote:
>>>>> With the addition of a non-voting check for linked URLs, we need to
>>>>> touch Jenkins jobs and our tox.ini everywhere. I suggest to do this
>>>>> differently: Combine the tiny voting jobs in one job and do the same
>>>>> for
>>>>> non-voting jobs.
>>>>>
>>>>> So, instead of:
>>>>> * check-syntax
>>>>> * check-niceness
>>>>> * check-deletions
>>>>> * check-links
>>>>>
>>>>> we would have:
>>>>> * check-voting
>>>>> * check-non-voting
>>>>
>>>> May I ask what would be posted on the review comment? Would we still
>>>> have a posting of all of the various job results {syntax, niceness,
>>>> deletions, links} and FAIL/PASS next to them? Or would it be just
>>>> {voting, non-voting} PASS/FAIL.
>>>
>>> It would be
>>> voting: PASS
>>> non-voting: PASS
>>> check-build: PASS
>>> check-lang: PASS
>>>
>>> And then you need to click on the link for details - like you do today -
>>> but also to read out which check failed,
>>
>> Ok, if that's the case, my preference would be to leave it as it is -
>> with each check giving its own explicit pass/fail.
>>
>> Aside from my personal preference, I believe explicitly separating the
>> tests is more friendly for new users.
>
> Let me cite from the blueprint for the reasons:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Reorganize the gating setup to combine the tiny jobs into a single
> voting and a single non-voting check. This will give more flexibility in
> adding new jobs and reduce load of Jenkins.
> Problem description
>
> Most of the current jobs are very small and it takes far longer to setup
> the virtual environment than running the tests. For example the niceness
> check runs in two seconds - while Jenkins needs at least 3 minutes for
> the whole job. The difference is that Jenkins needs to start a new
> virtual machine, install the git repository and all packages.
>
> With the addition of checking for links, we would have the following
> tiny jobs where each normally runs in less than 2 seconds:
>
>      niceness check
>      syntax check
>      deletions check
>      linked URL check
>
> Let’s combine these tiny jobs into voting and non-voting jobs.
>
> This gives us also the flexibility to add additional jobs easily and
> decide per repository on which of these jobs are voting and which are
> non-voting.
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The flexibility and the reduced work load are nice - but indeed this is
> nothing we have to change.
>
> That's why I wanted the discussion - to weight the pros and cons.


 From my perspective, user experience is paramount over working around 
infrastructure problems.

Yes, it sucks that the checks take a long time to setup and it would be 
amazing if we could fix that - it's also a user experience problem.

However, a solution that makes it more convoluted than it currently is 
to find "what went wrong" is really not nice :)


So, I'd propose option #3, which is fix the problem, while retaining or 
improving the experience of the user to determine "what went wrong" :)

Two potential ways I can see this moving forward:

* work out a way to keep the virtual environment set up/constantly 
running. Docs doesn't have the same requirements for fresh environments 
as the python projects, so this could certainly be one way.

* alternately, work out a way to make a really awesome index page when 
clicking on the pass/fail link. It should have itemised which tests 
passed/failed, and also the excerpts from the relevent bits of the logs 
(for example the line about which lines have trailing whitespace, the 
deleted file line, the actual docbook build error)



Regards,


Tom