Open Stack

Hi Anita, et. all,

I understand if the -1 is being voted because of testing framework
failures, it is a real pain.

Assuming the framework is working fine, I have noticed that there could be
genuine failures because of devstack failing to fully stack up or the test
failures. In my specific case, I have observed that
test_network_basic_ops() have 70% success rate and devstack fails once in a
while as well - and these failures are not because of the patch being
reviewed.
Therefore, being sensitive to patch submitters, and not causing -1 votes, I
am not running basic ops test. However, I would eventually like to start
running this test. This means we will start to vote -1 once in while (or
lots of time - depending upon the test results).

My question is as long as a -1 vote is casted with logs of the results, is
it OK? - even if the failures are because of known bugs/issues?


regards..
-Sukhdev







On Thu, Jan 2, 2014 at 6:11 AM, Anita Kuno <anteaya at anteaya.info> wrote:

> On 12/30/2013 09:32 PM, Anita Kuno wrote:
> > Please.
> >
> > When your third party testing structure votes on patches and your
> > testing structure is not stable, it will vote with a -1 on patches.
> >
> > This results in three consequences:
> > 1. The patch it votes on starts a countdown for abandonment, this is
> > frustrating for developers.
> > 2. Reviewers who use -Verified-1 as a filter criteria will not review a
> > patch with a -1 in the verification column in the Gerrit dashboard. This
> > prevents developers from progressing in their work, and this also
> > prevents reviewers from reviewing patches that need to be assessed.
> > 3. Third party testing that does not provide publicly accessible logs
> > leave developers with no way to diagnose the issue, which makes it very
> > difficult for a developer to fix, leaving the patch in a state of limbo.
> >
> > You can post messages to the patches, including a stable working url to
> > the logs for your tests, as you continue to work on your third party
> tests.
> >
> > You are also welcome to post a success of failure message, just please
> > refrain from allowing your testing infrastructure to vote on patches
> > until your testing infrastructure is working, reliably, and has logs
> > that developers can use to fix problems.
> >
> > The list of third party testing accounts are found here.[0]
> >
> > Right now there are three plugins that need to remove voting until they
> > are stable.
> >
> > Please be active in #openstack-neutron, #openstack-qa, and the mailing
> > list so that if there is an issue with your testing structure, people
> > can come and talk to you.
> >
> > Thank you,
> > Anita.
> >
> > [0]https://review.openstack.org/#/admin/groups/91,members
> >
> Keep in mind that the email provided in this list is expected to be an
> email people can use to contact you with questions and concerns about
> your testing interactions. [0]
>
> The point of the exercise is to provide useful and helpful information
> to developers and reviewers so that we all improve our code and create a
> better, more integrated product than we have now.
>
> Please check the email inboxes of the email addresses you have provided
> and please respond to inquires in a timely fashion.
>
> Please also remember people will look for you on irc, so having a
> representative available on irc for discussion will give you some useful
> feedback on ensuring your third party testing structure is behaving as
> efficiently as possible.
>
> We have a certain level of tolerance for unproductive noise while you
> are getting the bugs knocked out of your system. If developers are
> trying to contact you for more information and there is no response,
> third party testing structures that fail to comply with the expectation
> that they will respond to questions, will be addressed on a case by case
> basis.
>
> Thank you in advance for your kind attention,
> Anita.
>
> [0] https://review.openstack.org/#/admin/groups/91,members
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140109/cad20eab/attachment.html>