[openstack-dev] Learning to Debug the Gate

Mikhail Medvedev mihailmed at gmail.com
Wed Nov 18 23:03:32 UTC 2015


We had a mini tutorial today in #openstack-infra, were Clark Boylan
explained how one can bring up an environment to debug logstash-2.0.
This is tangential to "debugging the gate", but still could be useful to
better understand logstash/es pipeline. 

I did condense the conversation into the doc, excuse any grammar /
punctuation that I missed:

http://paste.openstack.org/show/479346/

-- 
Mikhail Medvedev

On Tue, Nov 10, 2015, at 10:21, Sean Dague wrote:
> There is also a neutron issue which is causing a 35% failure rate in
> neutron jobs -
> http://tinyurl.com/ne3ex4v
> 
> That one still needs resolution.
> 
> On 11/10/2015 10:54 AM, Davanum Srinivas wrote:
> > Took about 35 mins or so :)
> > 
> > -- Dims
> > 
> > On Tue, Nov 10, 2015 at 10:45 AM, Matt Riedemann
> > <mriedem at linux.vnet.ibm.com> wrote:
> >>
> >>
> >> On 11/9/2015 3:54 PM, Anita Kuno wrote:
> >>>
> >>> On 11/05/2015 07:45 PM, Anita Kuno wrote:
> >>>>
> >>>> On 11/03/2015 05:30 PM, Anita Kuno wrote:
> >>>>>
> >>>>> On 11/02/2015 12:39 PM, Anita Kuno wrote:
> >>>>>>
> >>>>>> On 10/29/2015 10:42 PM, Anita Kuno wrote:
> >>>>>>>
> >>>>>>> On 10/29/2015 08:27 AM, Anita Kuno wrote:
> >>>>>>>>
> >>>>>>>> On 10/28/2015 12:14 AM, Matt Riedemann wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 10/27/2015 4:08 AM, Anita Kuno wrote:
> >>>>>>>>>>
> >>>>>>>>>> Learning how to debug the gate was identified as a theme at the
> >>>>>>>>>> "Establish Key Themes for the Mitaka Cycle" cross-project session:
> >>>>>>>>>> https://etherpad.openstack.org/p/mitaka-crossproject-themes
> >>>>>>>>>>
> >>>>>>>>>> I agreed to take on this item and facilitate the process.
> >>>>>>>>>>
> >>>>>>>>>> Part one of the conversation includes referencing this video
> >>>>>>>>>> created by
> >>>>>>>>>> Sean Dague and Dan Smith:
> >>>>>>>>>> https://www.youtube.com/watch?v=fowBDdLGBlU
> >>>>>>>>>>
> >>>>>>>>>> Please consume this as you are able.
> >>>>>>>>>>
> >>>>>>>>>> Other suggestions for how to build on this resource were mentioned
> >>>>>>>>>> and
> >>>>>>>>>> will be coming in the future but this was an easy, actionable first
> >>>>>>>>>> step.
> >>>>>>>>>>
> >>>>>>>>>> Thank you,
> >>>>>>>>>> Anita.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> __________________________________________________________________________
> >>>>>>>>>>
> >>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>>>>>> Unsubscribe:
> >>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/tales-from-the-gate-how-debugging-the-gate-helps-your-enterprise
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> The source for the definition of "the gate":
> >>>>>>>>
> >>>>>>>> http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n34
> >>>>>>>>
> >>>>>>>> Thanks for following along,
> >>>>>>>> Anita.
> >>>>>>>>
> >>>>>>>
> >>>>>>> This is the status page showing the status of our running jobs,
> >>>>>>> including patches in the gate pipeline:
> >>>>>>> http://status.openstack.org/zuul/
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> Anita.
> >>>>>>>
> >>>>>>
> >>>>>> This is a simulation of how the gate tests patches:
> >>>>>> http://docs.openstack.org/infra/publications/zuul/#%2818%29
> >>>>>>
> >>>>>> Click in the browser window to advance the simulation.
> >>>>>>
> >>>>>> Thank you,
> >>>>>> Anita.
> >>>>>>
> >>>>>
> >>>>> Here is a presentation that uses the slide deck linked above, I
> >>>>> recommend watching: https://www.youtube.com/watch?v=WDoSCGPiFDQ
> >>>>>
> >>>>> Thank you,
> >>>>> Anita.
> >>>>>
> >>>>
> >>>> Three links in this edition of Learning to Debug the Gate:
> >>>>
> >>>> The view that tracks our top bugs:
> >>>> http://status.openstack.org/elastic-recheck/
> >>>>
> >>>> The logstash queries that create the above view:
> >>>>
> >>>> http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries
> >>>>
> >>>> Logstash itself, where you too can practice creating queries:
> >>>> http://logstash.openstack.org
> >>>>
> >>>> Note: in logstash the query is the transferable piece of information.
> >>>> Filters can help you create a query, they do not populate a query. The
> >>>> information that is in the query bar is what is important here.
> >>>>
> >>>> Practice making some queries of your own.
> >>>>
> >>>> Thanks for reading,
> >>>> Anita.
> >>>>
> >>>
> >>> Today the elastic search cluster was upgraded to 1.7.3:
> >>>
> >>> http://lists.openstack.org/pipermail/openstack-dev/2015-November/078314.html
> >>>
> >>> Go over the 1.7 elastic search docs:
> >>> https://www.elastic.co/guide/en/elasticsearch/reference/1.7/index.html
> >>> and try a few queries.
> >>>
> >>> Thanks for following along,
> >>> Anita.
> >>>
> >>> __________________________________________________________________________
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>
> >> If you want a fun little play by play of a critical blocking gate issue for
> >> nova, it's in the nova irc logs starting here:
> >>
> >> http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2015-11-10.log.html#t2015-11-10T14:59:27
> >>
> >> Which led to a block of oslo.messaging 2.8.0 in g-r and a revert in the o.m
> >> repo. The point isn't the outcome, but the process in which we figured out
> >> what the regression was and quickly resolved it using logstash and seeing
> >> what recent dependent library releases impacted nova.
> >>
> >> --
> >>
> >> Thanks,
> >>
> >> Matt Riedemann
> >>
> >>
> >> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > 
> > 
> 
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list