<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 28, 2014 at 5:17 AM, Thierry Carrez <span dir="ltr"><<a href="mailto:thierry@openstack.org" target="_blank">thierry@openstack.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>David Kranz wrote:<br>
> On 08/27/2014 03:43 PM, Sean Dague wrote:<br>
>> On 08/27/2014 03:33 PM, David Kranz wrote:<br>
</div><div>>>> Race conditions are what makes debugging very hard. I think we are in<br>
>>> the process of experimenting with such an idea: asymetric gating by<br>
>>> moving functional tests to projects, making them deeper and more<br>
>>> extensive, and gating against their own projects. The result should be<br>
>>> that when a code change is made, we will spend much more time running<br>
>>> tests of code that is most likely to be growing a race bug from the<br>
>>> change. Of course there is a risk that we will impair integration<br>
>>> testing and we will have to be vigilant about that. One mitigating<br>
>>> factor is that if cross-project interaction uses apis (official or not)<br>
>>> that are well tested by the functional tests, there is less risk that a<br>
>>> bug will only show up only when those apis are used by another project.<br>
>><br>
>> So, sorry, this is really not about systemic changes (we're running<br>
>> those in parallel), but more about skills transfer in people getting<br>
>> engaged. Because we need both. I guess that's the danger of breaking the<br>
>> thread is apparently I lost part of the context.<br>
>><br>
</div><div>> I agree we need both. I made the comment because if we can make gate<br>
> debugging less daunting<br>
> then less skill will be needed and I think that is key. Honestly, I am<br>
> not sure the full skill you have can be transferred. It was gained<br>
> partly through learning in simpler times.<br>
<br>
</div>I think we could develop tools and visualizations that would help the<br>
debugging tasks. We could make those tasks more visible, and therefore<br>
more appealing to the brave souls that step up to tackle them. Sean and<br>
Joe did a ton of work improving the raw data, deriving graphs from it,<br>
highlighting log syntax or adding helpful Apache footers. But those days<br>
they spend so much time fixing the issues themselves, they can't<br>
continue on improving those tools.<br></blockquote><div><br></div><div>Some tooling improvements I would like to do but probably don't have the time for:</div><div><br></div><div>* Add the ability to filter <a href="http://status.openstack.org/elastic-recheck/" target="_blank">http://status.openstack.org/elastic-recheck/</a> by project. So a neutron dev can see the list of bugs that are neutron related</div>
<div>* Make the list of open reviews on <a href="http://status.openstack.org/elastic-recheck/" target="_blank">http://status.openstack.org/elastic-recheck/</a> easier to find</div><div>* Create an up to date diagram of what OpenStack looks like when running, how services interact etc. <a href="http://docs.openstack.org/training-guides/content/figures/5/figures/image31.jpg" target="_blank">http://docs.openstack.org/training-guides/content/figures/5/figures/image31.jpg</a> and<a href="http://docs.openstack.org/admin-guide-cloud/content/figures/2/figures/openstack-arch-havana-logical-v1.jpg"> http://docs.openstack.org/admin-guide-cloud/content/figures/2/figures/openstack-arch-havana-logical-v1.jpg</a> are out of date</div>
<div>* Make <a href="http://jogo.github.io/gate" target="_blank">http://jogo.github.io/gate</a> easier to understand. This is what I check to see the health of the gate.</div><div>* Build a request-id tracker for logs. Make it easier to find the logs for a given request-id across multiple services (nova-api,nova-scheduler etc.)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
And that's part of where the gate burnout comes from: spending so much<br>
time on the issues themselves that you can no longer work on preventing<br>
them from happening, or making the job of handling the issues easier, or<br>
documenting/mentoring other people so that they can do it in your place.<br>
<span><font color="#888888"><br>
--<br>
Thierry Carrez (ttx)<br>
</font></span><div><div><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>