[openstack-dev] [all] gate debugging
anteaya at anteaya.info
Wed Aug 27 20:01:28 UTC 2014
On 08/27/2014 03:43 PM, Sean Dague wrote:
> On 08/27/2014 03:33 PM, David Kranz wrote:
>> On 08/27/2014 02:54 PM, Sean Dague wrote:
>>> Note: thread intentionally broken, this is really a different topic.
>>> On 08/27/2014 02:30 PM, Doug Hellmann wrote:>
>>>> On Aug 27, 2014, at 1:30 PM, Chris Dent <chdent at redhat.com> wrote:
>>>>> On Wed, 27 Aug 2014, Doug Hellmann wrote:
>>>>>> I have found it immensely helpful, for example, to have a written set
>>>>>> of the steps involved in creating a new library, from importing the
>>>>>> git repo all the way through to making it available to other projects.
>>>>>> Without those instructions, it would have been much harder to split up
>>>>>> the work. The team would have had to train each other by word of
>>>>>> mouth, and we would have had constant issues with inconsistent
>>>>>> approaches triggering different failures. The time we spent building
>>>>>> and verifying the instructions has paid off to the extent that we even
>>>>>> had one developer not on the core team handle a graduation for us.
>>>>> +many more for the relatively simple act of just writing stuff down
>>>> "Write it down.” is my theme for Kilo.
>>> I definitely get the sentiment. "Write it down" is also hard when you
>>> are talking about things that do change around quite a bit. OpenStack as
>>> a whole sees 250 - 500 changes a week, so the interaction pattern moves
>>> around enough that it's really easy to have *very* stale information
>>> written down. Stale information is even more dangerous than no
>>> information some times, as it takes people down very wrong paths.
>>> I think we break down on communication when we get into a conversation
>>> of "I want to learn gate debugging" because I don't quite know what that
>>> means, or where the starting point of understanding is. So those
>>> intentions are well meaning, but tend to stall. The reality was there
>>> was no road map for those of us that dive in, it's just understanding
>>> how OpenStack holds together as a whole and where some of the high risk
>>> parts are. And a lot of that comes with days staring at code and logs
>>> until patterns emerge.
>>> Maybe if we can get smaller more targeted questions, we can help folks
>>> better? I'm personally a big fan of answering the targeted questions
>>> because then I also know that the time spent exposing that information
>>> was directly useful.
>>> I'm more than happy to mentor folks. But I just end up finding the "I
>>> want to learn" at the generic level something that's hard to grasp onto
>>> or figure out how we turn it into action. I'd love to hear more ideas
>>> from folks about ways we might do that better.
>> Race conditions are what makes debugging very hard. I think we are in
>> the process of experimenting with such an idea: asymetric gating by
>> moving functional tests to projects, making them deeper and more
>> extensive, and gating against their own projects. The result should be
>> that when a code change is made, we will spend much more time running
>> tests of code that is most likely to be growing a race bug from theby
>> change. Of course there is a risk that we will impair integration
>> testing and we will have to be vigilant about that. One mitigating
>> factor is that if cross-project interaction uses apis (official or not)
>> that are well tested by the functional tests, there is less risk that a
>> bug will only show up only when those apis are used by another project.
> So, sorry, this is really not about systemic changes (we're running
> those in parallel), but more about skills transfer in people getting
> engaged. Because we need both. I guess that's the danger of breaking the
> thread is apparently I lost part of the context.
I love mentoring it is my favourite skills transfer pattern.
The optimal pattern is I agree to mentor someone and they are focused on
what I task them with, I evaluate it and we review it and not only do
they learn a skill but they have their own personal experience as a
foundation for having that skill.
Here is the part that breaks down in OpenStack - then the person I
mentor agrees to mentor someone else. Now I am mentoring one person plus
another by proxy, which is great because now in addition to technical
skills like searching and finding and offering patches and reviewing,
the person I'm mentoring learns the mentoring skills to be able to pass
on what they learn. For some reason I don't seem to make much headway (a
little but not much) in getting any traction in the second layer of
mentoring. For whatever reason, it just doesn't work and I am having to
teach everything all over from scratch one at a time to people. This is
not what I am used to and is really exhausting.
I wish I had answers but I don't. I don't know why this structure
doesn't pick up and scale out, but it doesn't.
Perhaps you might figure it out, Sean. I don't know.
More information about the OpenStack-dev