Open Stack

On Thursday, January 30, 2014 9:38:48 PM, Joe Gordon wrote:
> On Thu, Jan 30, 2014 at 6:06 PM, Vijay Venkatachalam
> <Vijay.Venkatachalam at citrix.com> wrote:
>> Thanks for the prompt reply Joe!
>>
>> So what is the purpose of trying recheck bug 'x'? Would it run the tests again mask the undeterministic error if it were to occur. In my case it did not mask.
>
> See https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures
>
>>
>> Also, what if the tests fail due to multiple undeterministic errors? Can I run recheck bug 'x1' 'x2'?
>
> No, and that won't make a difference.
>
>>
>> Test Results:
>> http://logs.openstack.org/24/57524/18/check/check-tempest-dsvm-neutron-pg/ed48142/logs/testr_results.html.gz
>> yaml for the bug:
>> https://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries/1254890.yaml
>>
>>
>> Thanks,
>> Vijay V.
>>
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: jagman4000 at gmail.com [mailto:jagman4000 at gmail.com] On Behalf
>>> Of Joe Gordon
>>> Sent: Friday, January 31, 2014 2:08 AM
>>> To: Vijay Venkatachalam
>>> Cc: openstack-infra at lists.openstack.org; Youcef Laribi
>>> Subject: Re: [OpenStack-Infra] Elastic Recheck not adding comments
>>>
>>> On Thu, Jan 30, 2014 at 11:30 AM, Vijay Venkatachalam
>>> <Vijay.Venkatachalam at citrix.com> wrote:
>>>> Hi Joe,
>>>>
>>>>
>>>>
>>>>>> if it happens again let me know.
>>>>
>>>>
>>>>
>>>> Yes, this has happened again.
>>>
>>> I am not sure why this is happening to your patch, but elastic-recheck not
>>> commenting on your patch doesn't mean you can't run 'recheck bug x'
>>>
>>> here is the information to find your logs in elasticsearch
>>> (logstash.openstack.org):
>>>
>>> build_change:"57524" AND
>>> build_uuid:"c50ce3b6d3d34e268e1eba7d6660d99b"
>>>
>>>>
>>>>
>>>>
>>>> Each time the tests fail in either 1249065 or 1254890 or both.  Tried
>>>> to abandon and restore the change in order to rerun the Jenkins checks
>>>> without code changes, but each time it's different tests that fail.
>>>>
>>>>
>>>>
>>>> Both of these are listed in
>>>>
>>>> status.openstack.org/elastic-recheck/
>>>>
>>>>
>>>>
>>>> Would appreciate any help/pointers in this regard.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Vijay V.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: jagman4000 at gmail.com [mailto:jagman4000 at gmail.com] On Behalf
>>> Of
>>>> Joe Gordon
>>>> Sent: Wednesday, January 29, 2014 2:36 AM
>>>> To: Vijay Venkatachalam
>>>> Cc: openstack-infra at lists.openstack.org; Youcef Laribi
>>>> Subject: Re: [OpenStack-Infra] Elastic Recheck not adding comments
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Jan 28, 2014 at 3:52 AM, Vijay Venkatachalam
>>>> <Vijay.Venkatachalam at citrix.com> wrote:
>>>>
>>>> Hi:
>>>>
>>>> Please pardon my ignorance on OpenStack infra.
>>>>
>>>> "Elastic Recheck" has not added entries to commit
>>>> https://review.openstack.org/#/c/57524/
>>>> for bugs  1249065 & 1254890.
>>>>
>>>>
>>>>
>>>> I am not sure why Elastic Recheck didn't run against your commit, if
>>>> it happens again let me know.
>>>>
>>>>
>>>>
>>>>
>>>> Both are listed in http://status.openstack.org/rechecks/
>>>>
>>>>
>>>>
>>>> The list of bugs Elastic Recheck is tracking can be found here:
>>>>
>>>>
>>>>
>>>> status.openstack.org/elastic-recheck/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Also, test results from Jenkins has marked test "process-returncode"
>>>> of suite "check-tempest-dsvm-neutron-pg" as failure. What is expected
>>>> as part of this test?
>>>>
>>>>
>>>> Thanks,
>>>> Vijay V.
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-Infra mailing list
>>>> OpenStack-Infra at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>
>>>>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>

I was seeing some real weirdness with logstash today too.  I have a 
stable/havana patch in nova that was hitting bug 1265495 but 
elastic-recheck wasn't commenting on it.  I know we have an e-r query 
for it because I wrote it and Joe merged it.  When I was searching with 
it in logstash though using build_branch:"stable/havana" for the 
filter, nothing was showing up.

But then after talking to clarkb about it, within a few minutes things 
started working, and as far as I know neither of us did anything 
different.  Spooky.

This is the conversation with Clark.

(2:19:29 PM) mriedem: is there a way to filter logstash queries based 
on stable branches only?
(2:20:15 PM) mriedem: i want to see if we backported a race bug from 
master
(2:21:11 PM) clarkb: mriedem: yes there is a new build_branch field
(2:21:22 PM) fungi: mriedem: AND build_branch:"stable/havana"
(2:21:56 PM) mriedem: nice, thanks
(2:23:10 PM) mriedem: jog0: does e-r report on stable branch patches? 
not seeing this https://review.openstack.org/#/c/70556/ reporting on 
https://review.openstack.org/#/c/70474/
(2:23:39 PM) mriedem: hmm, logstash isn't finding it either
(2:23:39 PM) mriedem: 
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU1NIRXhjZXB0aW9uOiBFcnJvciByZWFkaW5nIFNTSCBwcm90b2NvbCBiYW5uZXJcIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiBBTkQgYnVpbGRfYnJhbmNoOlwic3RhYmxlL2hhdmFuYVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiJhbGwiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkxNzE4MTU0MjQ4LCJtb2RlIjoidGVybXMiLCJhbmFseXplX2ZpZWxkIjoiYnVpbGRfbmFtZSJ9
(2:23:57 PM) mriedem: on stable/havana at least?
(2:24:24 PM) clarkb: mriedem: please don't search all time...
(2:24:30 PM) clarkb: I am going to rip all time out of the options list
(2:24:45 PM) mriedem: ok, 7 days doesn't hit either
(2:25:12 PM) clarkb: right it won't affect the content of results, but 
all time is basically go nuts mode for elasticsearch
(2:25:17 PM) mriedem: yeah
(2:25:22 PM) clarkb: when you give it boundaries queries are much more 
responsive
(2:25:41 PM) clarkb: mriedem: can you link to a log file that should 
match that?
(2:25:46 PM) mriedem: ok, so back to my main issue - i'm seeing an 
error we have an e-r query for in a stable/havana patch but it's not 
hitting in logstash
(2:25:51 PM) mriedem: 
http://logs.openstack.org/74/70474/1/check/check-tempest-dsvm-neutron/df2d911/console.html
(2:26:20 PM) mriedem: well damn
(2:26:26 PM) mriedem: i think our e-r query must be busted....
(2:26:42 PM) mriedem: i think jog0 and i must have been drunk one night 
when we changed it...
(2:29:09 PM) malini is now known as malini_afk
(2:30:31 PM) mriedem: doesn't matter anyway, stable/havana isn't 
showing up in logstash for this:
(2:30:31 PM) mriedem: filename:"console.html" AND 
message:"SSHException: Error reading SSH protocol banner"
(2:30:38 PM) mriedem: but it's in that log above
(2:31:22 PM) clarkb: and the log is indexed
(2:31:35 PM) openstackgerrit: A change was merged to 
openstack/requirements: Start renaming oslo.sphinx to oslosphinx  
https://review.openstack.org/71612
(2:31:51 PM) clarkb: mriedem: filename:"console.html" AND 
build_branch:"stable/havana" AND message:"SSHException: Error reading 
SSH protocol banner" AND build_uuid:"df2d911e4e674e0088ce3340a208baa4"
(2:32:46 PM) mriedem: hmm, why is build_uuid needed to make it work?
(2:33:03 PM) clarkb: mriedem: I was just using it to filter more, It 
shouldn't be necessary
(2:33:22 PM) mriedem: ah weird, this works filename:"console.html" AND 
build_branch:"stable/havana" AND message:"SSHException: Error reading 
SSH protocol banner"
(2:33:29 PM) mriedem: it seems that ordering of the filters matters?
(2:33:34 PM) clarkb: maybe? that would be weird
(2:35:56 PM) mriedem: clarkb: well now this is working for me: 
message:"raise SSHException(\'Error reading SSH protocol banner\' + 
str(x))" AND filename:"console.html" AND build_branch:"stable/havana"
(2:36:00 PM) russellb: oops, looks like we have gantt in the integrated 
gate ... probably should take it out for now
(2:36:04 PM) mriedem: clarkb: did you hit the "pay clarkb today" button?

--

Thanks,

Matt Riedemann