[openstack-dev] Learning to Debug the Gate

Sean Dague sean at dague.net
Tue Nov 10 16:21:38 UTC 2015


There is also a neutron issue which is causing a 35% failure rate in
neutron jobs -
http://tinyurl.com/ne3ex4v

That one still needs resolution.

On 11/10/2015 10:54 AM, Davanum Srinivas wrote:
> Took about 35 mins or so :)
> 
> -- Dims
> 
> On Tue, Nov 10, 2015 at 10:45 AM, Matt Riedemann
> <mriedem at linux.vnet.ibm.com> wrote:
>>
>>
>> On 11/9/2015 3:54 PM, Anita Kuno wrote:
>>>
>>> On 11/05/2015 07:45 PM, Anita Kuno wrote:
>>>>
>>>> On 11/03/2015 05:30 PM, Anita Kuno wrote:
>>>>>
>>>>> On 11/02/2015 12:39 PM, Anita Kuno wrote:
>>>>>>
>>>>>> On 10/29/2015 10:42 PM, Anita Kuno wrote:
>>>>>>>
>>>>>>> On 10/29/2015 08:27 AM, Anita Kuno wrote:
>>>>>>>>
>>>>>>>> On 10/28/2015 12:14 AM, Matt Riedemann wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/27/2015 4:08 AM, Anita Kuno wrote:
>>>>>>>>>>
>>>>>>>>>> Learning how to debug the gate was identified as a theme at the
>>>>>>>>>> "Establish Key Themes for the Mitaka Cycle" cross-project session:
>>>>>>>>>> https://etherpad.openstack.org/p/mitaka-crossproject-themes
>>>>>>>>>>
>>>>>>>>>> I agreed to take on this item and facilitate the process.
>>>>>>>>>>
>>>>>>>>>> Part one of the conversation includes referencing this video
>>>>>>>>>> created by
>>>>>>>>>> Sean Dague and Dan Smith:
>>>>>>>>>> https://www.youtube.com/watch?v=fowBDdLGBlU
>>>>>>>>>>
>>>>>>>>>> Please consume this as you are able.
>>>>>>>>>>
>>>>>>>>>> Other suggestions for how to build on this resource were mentioned
>>>>>>>>>> and
>>>>>>>>>> will be coming in the future but this was an easy, actionable first
>>>>>>>>>> step.
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>> Anita.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> __________________________________________________________________________
>>>>>>>>>>
>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>> Unsubscribe:
>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/tales-from-the-gate-how-debugging-the-gate-helps-your-enterprise
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> The source for the definition of "the gate":
>>>>>>>>
>>>>>>>> http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n34
>>>>>>>>
>>>>>>>> Thanks for following along,
>>>>>>>> Anita.
>>>>>>>>
>>>>>>>
>>>>>>> This is the status page showing the status of our running jobs,
>>>>>>> including patches in the gate pipeline:
>>>>>>> http://status.openstack.org/zuul/
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Anita.
>>>>>>>
>>>>>>
>>>>>> This is a simulation of how the gate tests patches:
>>>>>> http://docs.openstack.org/infra/publications/zuul/#%2818%29
>>>>>>
>>>>>> Click in the browser window to advance the simulation.
>>>>>>
>>>>>> Thank you,
>>>>>> Anita.
>>>>>>
>>>>>
>>>>> Here is a presentation that uses the slide deck linked above, I
>>>>> recommend watching: https://www.youtube.com/watch?v=WDoSCGPiFDQ
>>>>>
>>>>> Thank you,
>>>>> Anita.
>>>>>
>>>>
>>>> Three links in this edition of Learning to Debug the Gate:
>>>>
>>>> The view that tracks our top bugs:
>>>> http://status.openstack.org/elastic-recheck/
>>>>
>>>> The logstash queries that create the above view:
>>>>
>>>> http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries
>>>>
>>>> Logstash itself, where you too can practice creating queries:
>>>> http://logstash.openstack.org
>>>>
>>>> Note: in logstash the query is the transferable piece of information.
>>>> Filters can help you create a query, they do not populate a query. The
>>>> information that is in the query bar is what is important here.
>>>>
>>>> Practice making some queries of your own.
>>>>
>>>> Thanks for reading,
>>>> Anita.
>>>>
>>>
>>> Today the elastic search cluster was upgraded to 1.7.3:
>>>
>>> http://lists.openstack.org/pipermail/openstack-dev/2015-November/078314.html
>>>
>>> Go over the 1.7 elastic search docs:
>>> https://www.elastic.co/guide/en/elasticsearch/reference/1.7/index.html
>>> and try a few queries.
>>>
>>> Thanks for following along,
>>> Anita.
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> If you want a fun little play by play of a critical blocking gate issue for
>> nova, it's in the nova irc logs starting here:
>>
>> http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2015-11-10.log.html#t2015-11-10T14:59:27
>>
>> Which led to a block of oslo.messaging 2.8.0 in g-r and a revert in the o.m
>> repo. The point isn't the outcome, but the process in which we figured out
>> what the regression was and quickly resolved it using logstash and seeing
>> what recent dependent library releases impacted nova.
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 


-- 
Sean Dague
http://dague.net



More information about the OpenStack-dev mailing list