Open Stack

On 06/25/2015 02:44 PM, Nikolay Fedotov (nfedotov) wrote:
> Hello!
> 
> The cisco_ml2_ci CI has been reconfigured and tested using ci-sandbox repo. 
> 
> Patchset: https://review.openstack.org/#/c/192152/ 
> 
> The cisco_ml2_ci CI uses my own account in order to  post build status:
> "Nikolay Fedotov		10:53 PM
> Patch Set 9: Verified+1
> Build succeeded.
> ml2-nexus SUCCESS in 20m 15s
> python27 SUCCESS in 5m 27s (non-voting)"
> 
> Additional builds could be found here: http://128.107.233.10:8080/job/ml2-nexus/ 
> 
> Could you re-enable "cisco_ml2_ci" account ? 
> 
> Thanks
> Nikolay

Thank you Nikolav, this account is now re-enabled.

Please ensure you have read and are following the requirements listed in
this documentation:
http://docs.openstack.org/infra/system-config/third_party.html

Thank you,
Anita.


> 
> -----Original Message-----
> From: Anita Kuno [mailto:anteaya at anteaya.info] 
> Sent: Wednesday, June 17, 2015 10:17 PM
> To: third-party-announce at lists.openstack.org
> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
> 
> On 06/17/2015 12:23 PM, Nikolay Fedotov (nfedotov) wrote:
>> Hello Anita
>>
>> Here is results of my investigation:
>> * Jenkins died at " Jun 11, 2015 8:15:45 PM " . Part of Jenkins log is in attachment
>> * Zuul experienced issues with memory too as it is reported by "apport". Log is in attachment. " Signal 11, or officially know as "segmentation fault", means that the program accessed a memory location that was not assigned ". Not sure why zuul did not died as Jenkins did.
>> * "ml2-nexus"  got EXCEPTION status because zuul was not able to submit new job. Code where zuul sets EXCEPTION status here: https://github.com/openstack-infra/zuul/blob/master/zuul/launcher/gearman.py#L368-L370 . I could find other places where a job gets EXCEPTION status. I think it failed to add job to Gearman queue because of RAM issue. (Gearman is managed by zuul)
>>
>> In order to avoid such issue I will move Jenkins and other services to a machine (virtual) that has more RAM.
>>
>> Thanks
> 
> Wonderful, thank you Nidolav for sharing the results of your
> investigation. I look forward to hearing when your services are on new
> machines.
> 
> Something you may want to consider is to have your Jenkins server use
> utc time, your zuul server apparently already does. It is up to you and
> isn't a requirement, but when sharing logs with others in different
> timezones it just makes it easier for folks to understand the chronology
> of an event.
> 
> I'll wait to hear back from you that your services are moved and running
> before I re-enable your account. I will need your assurance that you
> will test your system using our ci-sandbox repo prior to resuming
> commenting on other repositories.
> 
> Thanks for your follow-up,
> Anita.
> 
>>
>> -----Original Message-----
>> From: Anita Kuno [mailto:anteaya at anteaya.info] 
>> Sent: Saturday, June 13, 2015 2:41 AM
>> To: third-party-announce at lists.openstack.org
>> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
>>
>> On 06/12/2015 06:20 PM, Nikolay Fedotov (nfedotov) wrote:
>>> Hello Anita
>>>
>>> Thank you for pointing me out to the cacti and IRC log.
>>>
>>> I did not change comment text of the "Cisco ml2 CI". If tests failed the CI should reply :
>>> " Build failed. For help on isolating this failure, please contact cisco-openstack-neutron-ci at cisco.com. To re-run, post a 'recheck cisco-ml2' comment. "
>>
>> The belief is it is the recheck syntax in the comment text that is the problem.
>>
>> Have you logs for your system activity from yesterday?
>>
>> Is anyone associated with this account active on irc?
>>
>> Thank you,
>> Anita
>>
>> anteaya on irc
>>
>>
>>>
>>> I see there are guesses that looping might be caused by wrong " comment_filter " (regexp for check text) value. The CI uses:
>>>         - event: comment-added
>>>           comment_filter: (?i)^(Patch Set [0-9]+:\n\n)?\s*recheck(( (?:bug|lp)[\s#:]*(\d+))|( no bug))\s*$
>>>         - event: comment-added
>>>           comment_filter: (?i)^(Patch Set [0-9]+:\n\n)?\s*recheck 
>>> cisco-ml2\s*$
>>>
>>> My guess is: Jenkins died at " Jun 11, 2015 8:15:45 PM"  due to error='Cannot allocate memory'. And, maybe, it caused zuul to start looping. But I am not sure.
>>>
>>> Could you advice further steps that help me to get the CI account enabled?
>>>
>>> Thanks.
>>>
>>> -----Original Message-----
>>> From: Anita Kuno [mailto:anteaya at anteaya.info]
>>> Sent: Friday, June 12, 2015 11:13 PM
>>> To: third-party-announce at lists.openstack.org
>>> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
>>>
>>> On 06/12/2015 04:06 PM, Anita Kuno wrote:
>>>> On 06/12/2015 03:47 PM, Nikolay Fedotov (nfedotov) wrote:
>>>>> Hello All!
>>>>>
>>>>> I was wondering what kind of looping are you talking about? 
>>>>>
>>>>> I see the change https://review.openstack.org/#/c/188221/ has 5  patchset and each Cisco CI replied :
>>>>> - Cisco Tail-f CI - 4 times
>>>>> - Cisco N1kv CI - 4 times
>>>>> - Cisco APIC CI - 4 times
>>>>> - Cisco ml2 CI - 0 times
>>>>
>>>> We had to perform database surgery to remove the hundreds of comments 
>>>> left by Cisco ml2 CI. Trust us it was looping.
>>>>
>>>> We also had to restart gerrit to close the connection as our gerrit 
>>>> input spiked because of it. See the inbound spike from yesterday?
>>>> http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id
>>>> =
>>>> 34&rra_id=all
>>>> that was you.
>>>>
>>>> Your comment text changed yesterday sometime. No other patches were 
>>>> affected by this.
>>>>
>>>> Please confirm your comment text changed yesterday.
>>>>
>>>> Thank you,
>>>> Anita.
>>>
>>> Also here is the log of the incident:
>>>
>>> http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack
>>> -infra.2015-06-11.log.html#t2015-06-11T20:34:08
>>>
>>>>
>>>>>
>>>>> So each CI systems tested every (except the 5th) patchset only once.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Asselin, Ramy [mailto:ramy.asselin at hp.com]
>>>>> Sent: Thursday, June 11, 2015 11:55 PM
>>>>> To: Announcements for third party CI operators.
>>>>> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
>>>>>
>>>>> Wow, good call. We should document that and find a way to protect that from happening.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Doug Wiegley [mailto:dougw at a10networks.com]
>>>>> Sent: Thursday, June 11, 2015 1:49 PM
>>>>> To: Announcements for third party CI operators.
>>>>> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
>>>>>
>>>>> Not sure if this is your issue, but this looks similar to a looping issue that I inflicted on myself, by including my recheck text in my CI comment.
>>>>>
>>>>>
>>>>> "Cisco ml2 CI2:22 PMPatch Set 4:
>>>>> Build failed. For help on isolating this failure, please contact cisco-openstack-neutron-ci at cisco.com. To re-run, post a 'recheck cisco-ml2' comment.
>>>>>
>>>>> * ml2-nexus <http://128.107.233.10:8080/job/ml2-nexus/None> 
>>>>> EXCEPTION
>>>>> * python27 <http://128.107.233.10:8080/job/python27/225> SUCCESS in 6m 06s (non-voting) "
>>>>>
>>>>> Thanks,
>>>>> Doug
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 6/11/15, 2:40 PM, "Anita Kuno" <anteaya at anteaya.info> wrote:
>>>>>
>>>>>> https://review.openstack.org/#/c/188221/
>>>>>>
>>>>>> The patch speaks for itself.
>>>>>>
>>>>>> Thank you, Russell Bryant for the alert.
>>>>>>
>>>>>> Before this system is enabled we will need to hear the results of 
>>>>>> your investigation as to why this happened as well as you assurance 
>>>>>> that it won't happen again.
>>>>>>
>>>>>> Thank you,
>>>>>> Anita.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Third-party-announce mailing list
>>>>>> Third-party-announce at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-ann
>>>>>> o
>>>>>> unc
>>>>>> e
>>>>> _______________________________________________
>>>>> Third-party-announce mailing list
>>>>> Third-party-announce at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-anno
>>>>> u
>>>>> nce
>>>>>
>>>>> _______________________________________________
>>>>> Third-party-announce mailing list
>>>>> Third-party-announce at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-anno
>>>>> u
>>>>> nce
>>>>>
>>>>> _______________________________________________
>>>>> Third-party-announce mailing list
>>>>> Third-party-announce at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-anno
>>>>> u
>>>>> nce
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Third-party-announce mailing list
>>> Third-party-announce at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announ
>>> ce
>>>
>>> _______________________________________________
>>> Third-party-announce mailing list
>>> Third-party-announce at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announ
>>> ce
>>>
>>
>>
>> _______________________________________________
>> Third-party-announce mailing list
>> Third-party-announce at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
>>
>>
>>
>> _______________________________________________
>> Third-party-announce mailing list
>> Third-party-announce at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
>> Please attend the third party meetings: http://eavesdrop.openstack.org/#Third_Party_Meeting
>>
> 
> 
> _______________________________________________
> Third-party-announce mailing list
> Third-party-announce at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> Please attend the third party meetings: http://eavesdrop.openstack.org/#Third_Party_Meeting
> 
> _______________________________________________
> Third-party-announce mailing list
> Third-party-announce at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> Please attend the third party meetings: http://eavesdrop.openstack.org/#Third_Party_Meeting
>