<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 14, 2015 at 1:06 PM, Anita Kuno <span dir="ltr"><<a href="mailto:anteaya@anteaya.info" target="_blank">anteaya@anteaya.info</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">On 07/14/2015 11:39 AM, Erlon Cruz wrote:<br>
> Hi Anita,<br>
><br>
</span><span class="">> I think I knew what happened, we had a power failure this weekend, and the<br>
> VM that was running the Jenkins master for this CI was automatically<br>
> booted, if you see on the CI reports all of then where sent on 13/07.<br>
<br>
</span>Great are you able to take any steps so that if you have a power failure<br>
in future that a human needs to evaluate the system prior to it<br>
reporting on patches.<br>
<span class=""><br></span></blockquote><div><br></div><div>Sure, we have done it!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
><br>
> Whats is there any criteria for removing CIs?<br>
<br>
</span>This gerrit account has been disabled. The CI has not be removed. There<br>
is a difference.<br>
<br></blockquote><div>That was what I meant. Disable.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The criteria for disabling CIs is meeting infra requirements as outlined<br>
here:<br>
<a href="http://docs.openstack.org/infra/system-config/third_party.html#requirements" rel="noreferrer" target="_blank">http://docs.openstack.org/infra/system-config/third_party.html#requirements</a><br>
as well as infra having discretion at disabling a system at any time:<br>
"The OpenStack Infrastructure team disables mis-behaving third-party ci<br>
accounts at its discretion." from<br>
<a href="http://docs.openstack.org/infra/system-config/third_party.html#permissions-on-your-third-party-system" rel="noreferrer" target="_blank">http://docs.openstack.org/infra/system-config/third_party.html#permissions-on-your-third-party-system</a><br>
<span class=""><br>
> I fell this removal was very<br>
> unfair.<br>
<br>
</span>That's fine and I appreciate you sharing your thoughts with me.<br>
<span class=""><br>
<br>
> I have seen CIs reporting errors for weeks<br>
<br>
</span>Have you communicated those errors to anyone? The core reviewers of the<br>
project? The operators of the CI in question? Shared them at a third<br>
party meeting?<br>
<br></blockquote><div><br></div><div>Yes, we talked[1] to cinder guys and informed about the problem when it happened, also informed about the patch that was ready to fix the problem[2], that was only merged after I asked again.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Part of the reason why my actions may seem so heavy handed is that my<br>
vision of CI operators working collaboratively with each other to ensure<br>
high quality of all systems has not come to pass. Quality as a whole is<br>
low and folks tend to ignore CI reports as noise, even the useful ones.<br>
As a survival technique I disable systems when I find out about them.<br>
This system came to my attention and I diabled it.<br></blockquote><div><br></div><div>We worked in a collaboration scheme since the problem came up. Since we detected the problem, that was not our fault BTW, we are working with Cinder team to get it fixed. Collaboration is opposed to the action that was taken here. You see a problem and before informing the people that could solve the problem or trying to understand the status of the problem just disable the CI.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<a href="http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-07-13.log.html#t2015-07-13T21:36:48" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-07-13.log.html#t2015-07-13T21:36:48</a><br>
<span class=""><br>
<br>
<br>
> and then, only after<br>
> warnings to operators in this mailing list, they are removed.<br>
<br>
</span>Nothing has been removed. Infra takes care of the gerrit accounts,<br>
nothing more.<br>
<span class=""><br>
> In our case,<br>
> it was less than 24h, we got no warnings and got this account disabled.<br>
> Today, we can't test the CI against the patch that entered (yesterday) and<br>
> will have to do all the process to get the account again. Is it possible to<br>
> re-enable the account so we can continue the process to test the patch that<br>
> should fix the problem?<br>
<br>
</span>Let's get some confirmation a power outage in future will have a<br>
different outcome.<br></blockquote><div><br></div><div>I have done this. If you re-enable it, it will run in silent mode until we assure that it is stable enough to post to gerrit.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Thanks Erlon,<br>
Anita.<br>
<div class=""><div class="h5"><br>
><br>
> Thanks,<br>
> Erlon<br>
><br>
> On Tue, Jul 14, 2015 at 12:06 PM, Anita Kuno <<a href="mailto:anteaya@anteaya.info">anteaya@anteaya.info</a>> wrote:<br>
><br>
>> On 07/14/2015 10:18 AM, Erlon Cruz wrote:<br>
>>> Hi Anita,<br>
>>><br>
>>> There was a change in an external library (lockutils) that was causing<br>
>> the<br>
>>> patches run in our CI to fail. We have reported the problem[1],<br>
>> submitted a<br>
>>> fix[2], and shut down the CI while the fix wasn't merged.<br>
>><br>
>> Okay well somehow you shut down your system but it was still leaving<br>
>> comments on gerrit patches. How did that happen?<br>
>><br>
>> Thanks Erlon,<br>
>> Anita.<br>
>><br>
>><br>
>>> That was about 10<br>
>>> days ago, and the patch only landed yesterday. We will re-run the above<br>
>>> tests and submit the successful results.<br>
>>><br>
>>> [1] <a href="https://bugs.launchpad.net/cinder/+bug/1470112" rel="noreferrer" target="_blank">https://bugs.launchpad.net/cinder/+bug/1470112</a><br>
>>> [2] <a href="https://review.openstack.org/#/c/197686/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/197686/</a><br>
>>><br>
>>> On Mon, Jul 13, 2015 at 7:06 PM, Anita Kuno <<a href="mailto:anteaya@anteaya.info">anteaya@anteaya.info</a>><br>
>> wrote:<br>
>>><br>
>>>> <a href="https://review.openstack.org/#/c/199537/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/199537/</a><br>
>>>><br>
>>>> <a href="https://review.openstack.org/#/c/198853/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/198853/</a><br>
>>>><br>
>>>> <a href="https://review.openstack.org/#/c/193937/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/193937/</a><br>
>>>><br>
>>>> <a href="https://review.openstack.org/#/c/201241/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/201241/</a><br>
>>>><br>
>>>> This account is posting NOT_REGISTERED on patches.<br>
>>>><br>
>>>> This is an indication that zuul (if you are using zuul) is unable to<br>
>>>> clearly communicate with Jenkins nodes. This is an internal error and<br>
>>>> has nothing to do with the patch.<br>
>>>><br>
>>>> Please fix your system and post your incident report with details as a<br>
>>>> response.<br>
>>>><br>
>>>> Please test your system on ci-sandbox after it is working again.<br>
>>>><br>
>>>> Thank you,<br>
>>>> Anita.<br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Third-party-announce mailing list<br>
>>>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
>>>><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>>>> Please attend the third party meetings:<br>
>>>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Third-party-announce mailing list<br>
>>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>>> Please attend the third party meetings:<br>
>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Third-party-announce mailing list<br>
>> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
>> Please attend the third party meetings:<br>
>> <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
>><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Third-party-announce mailing list<br>
> <a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
> Please attend the third party meetings: <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
><br>
<br>
<br>
_______________________________________________<br>
Third-party-announce mailing list<br>
<a href="mailto:Third-party-announce@lists.openstack.org">Third-party-announce@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce</a><br>
Please attend the third party meetings: <a href="http://eavesdrop.openstack.org/#Third_Party_Meeting" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/#Third_Party_Meeting</a><br>
</div></div></blockquote></div><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">[1] <a href="http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-02.log.html">http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-02.log.html</a></div><div class="gmail_extra">[2  <a href="http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-08.log.html">http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-08.log.html</a></div></div>