Open Stack

David Kranz wrote:
> On 08/27/2014 03:43 PM, Sean Dague wrote:
>> On 08/27/2014 03:33 PM, David Kranz wrote:
>>> Race conditions are what makes debugging very hard. I think we are in
>>> the process of experimenting with such an idea: asymetric gating by
>>> moving functional tests to projects, making them deeper and more
>>> extensive, and gating against their own projects. The result should be
>>> that when a code change is made, we will spend much more time running
>>> tests of code that is most likely to be growing a race bug from the
>>> change. Of course there is a risk that we will impair integration
>>> testing and we will have to be vigilant about that. One mitigating
>>> factor is that if cross-project interaction uses apis (official or not)
>>> that are well tested by the functional tests, there is less risk that a
>>> bug will only show up only when those apis are used by another project.
>>
>> So, sorry, this is really not about systemic changes (we're running
>> those in parallel), but more about skills transfer in people getting
>> engaged. Because we need both. I guess that's the danger of breaking the
>> thread is apparently I lost part of the context.
>>
> I agree we need both. I made the comment because if we can make gate
> debugging less daunting
> then less skill will be needed and I think that is key. Honestly, I am
> not sure the full skill you have can be transferred. It was gained
> partly through learning in simpler times.

I think we could develop tools and visualizations that would help the
debugging tasks. We could make those tasks more visible, and therefore
more appealing to the brave souls that step up to tackle them. Sean and
Joe did a ton of work improving the raw data, deriving graphs from it,
highlighting log syntax or adding helpful Apache footers. But those days
they spend so much time fixing the issues themselves, they can't
continue on improving those tools.

And that's part of where the gate burnout comes from: spending so much
time on the issues themselves that you can no longer work on preventing
them from happening, or making the job of handling the issues easier, or
documenting/mentoring other people so that they can do it in your place.

-- 
Thierry Carrez (ttx)