Re: [placement][tripleo][infra] zuul job dependencies for greater good?
On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote:
On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote:
snip
That said, I wouldn't push too hard in either direction until someone crunched the numbers and figured out how much time it would have saved to not run long tests on patch sets with failing unit tests. I feel like it's probably possible to figure that out, and if so then we should do it before making any big decisions on this.
For numbers the elastic-recheck tool [0] gives us fairly accurate tracking of which issues in the system cause tests to fail. You can use this as a starting point to potentially figure out how expensive indentation errors caught by the pep8 jobs ends up being or how often unittests fail. You probably need to tweak the queries there to get that specific though.
Periodically I also dump node resource utilization by project, repo, and job [1]. I haven't automated this because Tobiash has written a much better thing that has Zuul inject this into graphite and we should be able to set up a grafana dashboard for that in the future instead.
These numbers won't tell a whole story, but should paint a fairly accurate high level picture of the types of things we should look at to be more node efficient and "time in gate" efficient. Looking at these two really quickly myself it seems that job timeouts are a big cost (anyone looking into why our jobs timeout?).
[0] http://status.openstack.org/elastic-recheck/index.html [1] http://paste.openstack.org/show/746083/
Hope this helps, Clark
Here is some numbers [0] extracted via elastic-recheck console queries. It shows 6% of wasted failures because of tox issues in general, and 3% for tripleo projects in particular. My final take is, given some middle-ground solution, like I illustrated earlier this sub-thread, it might be worth it, and the effort for boosting up the total throughput of openstack CI system by a 6% is not so bad idea. [0] http://paste.openstack.org/show/746503/ -- Best regards, Bogdan Dobrelya, Irc #bogdando
On 28.02.2019 15:16, Bogdan Dobrelya wrote:
On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote:
On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote:
snip
That said, I wouldn't push too hard in either direction until someone crunched the numbers and figured out how much time it would have saved to not run long tests on patch sets with failing unit tests. I feel like it's probably possible to figure that out, and if so then we should do it before making any big decisions on this.
For numbers the elastic-recheck tool [0] gives us fairly accurate tracking of which issues in the system cause tests to fail. You can use this as a starting point to potentially figure out how expensive indentation errors caught by the pep8 jobs ends up being or how often unittests fail. You probably need to tweak the queries there to get that specific though.
Periodically I also dump node resource utilization by project, repo, and job [1]. I haven't automated this because Tobiash has written a much better thing that has Zuul inject this into graphite and we should be able to set up a grafana dashboard for that in the future instead.
These numbers won't tell a whole story, but should paint a fairly accurate high level picture of the types of things we should look at to be more node efficient and "time in gate" efficient. Looking at these two really quickly myself it seems that job timeouts are a big cost (anyone looking into why our jobs timeout?).
[0] http://status.openstack.org/elastic-recheck/index.html [1] http://paste.openstack.org/show/746083/
Hope this helps, Clark
Here is some numbers [0] extracted via elastic-recheck console queries. It shows 6% of wasted failures because of tox issues in general, and 3% for tripleo projects in particular.
My final take is, given some middle-ground solution, like I illustrated earlier this sub-thread, it might be worth it, and the effort for boosting up the total throughput of openstack CI system by a 6% is not so bad idea.
Also I forgot to mention that the 6% for numbers of failed jobs give a non linear function for the saved resources pool, given the numbers [1] from Clark: Top 20 repos by resource usage: openstack/neutron: 198367474.73s, 16.95% openstack/tripleo-heat-templates: 174221496.83s, 14.89% ... And a 3% cut off from t-h-t (14.89%) frees out 0.45% of total resources into the pool, which could shorten total pipeline wait times for CI jobs for, say manila by a *half*: openstack/manila: 11137905.75s, 0.95% [1] http://paste.openstack.org/show/746083/
-- Best regards, Bogdan Dobrelya, Irc #bogdando
On Thu, Feb 28, 2019, at 6:16 AM, Bogdan Dobrelya wrote:
On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote:
On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote:
snip
That said, I wouldn't push too hard in either direction until someone crunched the numbers and figured out how much time it would have saved to not run long tests on patch sets with failing unit tests. I feel like it's probably possible to figure that out, and if so then we should do it before making any big decisions on this.
For numbers the elastic-recheck tool [0] gives us fairly accurate tracking of which issues in the system cause tests to fail. You can use this as a starting point to potentially figure out how expensive indentation errors caught by the pep8 jobs ends up being or how often unittests fail. You probably need to tweak the queries there to get that specific though.
Periodically I also dump node resource utilization by project, repo, and job [1]. I haven't automated this because Tobiash has written a much better thing that has Zuul inject this into graphite and we should be able to set up a grafana dashboard for that in the future instead.
These numbers won't tell a whole story, but should paint a fairly accurate high level picture of the types of things we should look at to be more node efficient and "time in gate" efficient. Looking at these two really quickly myself it seems that job timeouts are a big cost (anyone looking into why our jobs timeout?).
[0] http://status.openstack.org/elastic-recheck/index.html [1] http://paste.openstack.org/show/746083/
Hope this helps, Clark
Here is some numbers [0] extracted via elastic-recheck console queries. It shows 6% of wasted failures because of tox issues in general, and 3% for tripleo projects in particular.
Are these wasted failures? The queries appear to be tracking valid failures of those jobs. These valid failures are then actionable feedback for developers to fix their changes. If we expect these failures to go away we'll need to be much more forceful about getting developers to run tox locally before they push. We need to compare (and this is a rough example) the resource usage and developer time of complete batch results where you have a pep8 issue and an integration job issue in patchset one, fix both and all tests pass in patchset two against pep8 failure in patchset one, integration failure in patchset two, all tests pass in patchset three. Today: patchset one: pep8 FAILURE unittest SUCCESS integration FAILURE patchset two: pep8 SUCCESS unittest SUCCESS integration SUCCESS Proposed Future: patchset one: pep8 FAILURE unittest SUCCESS patchset two: pep8 SUCCESS unittest SUCCESS integration FAILURE patchset three: pep8 SUCCESS unittest SUCCESS integration SUCCESS There are strictly more patchsets (developer roundtrips) and tests run in my contrived example. Reality will depend on how many iterations we actually see in the real world (eg are we fixing bugs reliably based on test feedback and how often do unittest and integration tests fail for different reasons). Someone with a better background in statistics will probably tell me this approach is wrong, but using the elasticsearch tooling one approach may be to pick say 1k changes, then for each change identify which tests failed on subsequent patchsets? Then we'd be able to infer with some confidence the behaviors we have in the test suites around catching failures, whether they are independent across jobs and whether or not we fix them in batches. Finally, I think we might also want to consider what the ideal is here. If we find we can optimize the current system for current behaviors, we also need to consider if that is worthwhile given an ideal. Should developers be expected to run unittests and linters locally before pushing? If so then optimizing for when they don't might be effort better spent on making it easier to run the unittests and linters locally and educating developers on how to do so. I think we'd also ideally expect our tests to pass once they run in the gate pipeline. Unfortunately I think our elastic-recheck data shows they often don't and more effort in fixing those failures would provide a dramatic increase in throughput (due to the compounding costs of subsequent gate resets).
My final take is, given some middle-ground solution, like I illustrated earlier this sub-thread, it might be worth it, and the effort for boosting up the total throughput of openstack CI system by a 6% is not so bad idea.
[0] http://paste.openstack.org/show/746503/
-- Best regards, Bogdan Dobrelya, Irc #bogdando
On 28.02.2019 17:22, Clark Boylan wrote:
On Thu, Feb 28, 2019, at 6:16 AM, Bogdan Dobrelya wrote:
On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote:
On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote:
snip
That said, I wouldn't push too hard in either direction until someone crunched the numbers and figured out how much time it would have saved to not run long tests on patch sets with failing unit tests. I feel like it's probably possible to figure that out, and if so then we should do it before making any big decisions on this.
For numbers the elastic-recheck tool [0] gives us fairly accurate tracking of which issues in the system cause tests to fail. You can use this as a starting point to potentially figure out how expensive indentation errors caught by the pep8 jobs ends up being or how often unittests fail. You probably need to tweak the queries there to get that specific though.
Periodically I also dump node resource utilization by project, repo, and job [1]. I haven't automated this because Tobiash has written a much better thing that has Zuul inject this into graphite and we should be able to set up a grafana dashboard for that in the future instead.
These numbers won't tell a whole story, but should paint a fairly accurate high level picture of the types of things we should look at to be more node efficient and "time in gate" efficient. Looking at these two really quickly myself it seems that job timeouts are a big cost (anyone looking into why our jobs timeout?).
[0] http://status.openstack.org/elastic-recheck/index.html [1] http://paste.openstack.org/show/746083/
Hope this helps, Clark
Here is some numbers [0] extracted via elastic-recheck console queries. It shows 6% of wasted failures because of tox issues in general, and 3% for tripleo projects in particular.
Are these wasted failures? The queries appear to be tracking valid failures of those jobs. These valid failures are then actionable feedback for developers to fix their changes.
I'll need more time to get through the ideas below and wrap it around my heaf, but for this particular statement, I understand the CI pool resources as "wasted" if executed for either of the failed tox jobs. The reason why it fails is secondary from that standpoint...
If we expect these failures to go away we'll need to be much more forceful about getting developers to run tox locally before they push.
We need to compare (and this is a rough example) the resource usage and developer time of complete batch results where you have a pep8 issue and an integration job issue in patchset one, fix both and all tests pass in patchset two against pep8 failure in patchset one, integration failure in patchset two, all tests pass in patchset three.
Today: patchset one: pep8 FAILURE unittest SUCCESS integration FAILURE patchset two: pep8 SUCCESS unittest SUCCESS integration SUCCESS
Proposed Future: patchset one: pep8 FAILURE unittest SUCCESS patchset two: pep8 SUCCESS unittest SUCCESS integration FAILURE patchset three: pep8 SUCCESS unittest SUCCESS integration SUCCESS
There are strictly more patchsets (developer roundtrips) and tests run in my contrived example. Reality will depend on how many iterations we actually see in the real world (eg are we fixing bugs reliably based on test feedback and how often do unittest and integration tests fail for different reasons).
Someone with a better background in statistics will probably tell me this approach is wrong, but using the elasticsearch tooling one approach may be to pick say 1k changes, then for each change identify which tests failed on subsequent patchsets? Then we'd be able to infer with some confidence the behaviors we have in the test suites around catching failures, whether they are independent across jobs and whether or not we fix them in batches.
Finally, I think we might also want to consider what the ideal is here. If we find we can optimize the current system for current behaviors, we also need to consider if that is worthwhile given an ideal. Should developers be expected to run unittests and linters locally before pushing? If so then optimizing for when they don't might be effort better spent on making it easier to run the unittests and linters locally and educating developers on how to do so. I think we'd also ideally expect our tests to pass once they run in the gate pipeline. Unfortunately I think our elastic-recheck data shows they often don't and more effort in fixing those failures would provide a dramatic increase in throughput (due to the compounding costs of subsequent gate resets).
My final take is, given some middle-ground solution, like I illustrated earlier this sub-thread, it might be worth it, and the effort for boosting up the total throughput of openstack CI system by a 6% is not so bad idea.
[0] http://paste.openstack.org/show/746503/
-- Best regards, Bogdan Dobrelya, Irc #bogdando
-- Best regards, Bogdan Dobrelya, Irc #bogdando
On 28.02.2019 17:35, Bogdan Dobrelya wrote:
On 28.02.2019 17:22, Clark Boylan wrote:
On Thu, Feb 28, 2019, at 6:16 AM, Bogdan Dobrelya wrote:
On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote:
On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote:
snip
That said, I wouldn't push too hard in either direction until someone crunched the numbers and figured out how much time it would have saved to not run long tests on patch sets with failing unit tests. I feel like it's probably possible to figure that out, and if so then we should do it before making any big decisions on this.
For numbers the elastic-recheck tool [0] gives us fairly accurate tracking of which issues in the system cause tests to fail. You can use this as a starting point to potentially figure out how expensive indentation errors caught by the pep8 jobs ends up being or how often unittests fail. You probably need to tweak the queries there to get that specific though.
Periodically I also dump node resource utilization by project, repo, and job [1]. I haven't automated this because Tobiash has written a much better thing that has Zuul inject this into graphite and we should be able to set up a grafana dashboard for that in the future instead.
These numbers won't tell a whole story, but should paint a fairly accurate high level picture of the types of things we should look at to be more node efficient and "time in gate" efficient. Looking at these two really quickly myself it seems that job timeouts are a big cost (anyone looking into why our jobs timeout?).
[0] http://status.openstack.org/elastic-recheck/index.html [1] http://paste.openstack.org/show/746083/
Hope this helps, Clark
Here is some numbers [0] extracted via elastic-recheck console queries. It shows 6% of wasted failures because of tox issues in general, and 3% for tripleo projects in particular.
Are these wasted failures? The queries appear to be tracking valid failures of those jobs. These valid failures are then actionable feedback for developers to fix their changes.
I'll need more time to get through the ideas below and wrap it around my heaf, but for this particular statement, I understand the CI pool resources as "wasted" if executed for either of the failed tox jobs. The reason why it fails is secondary from that standpoint...
ugh... I meant the CI resources spent for running other integration jobs, while still having an unmergable patch here. As I noted above, *some* of the integration jobs may still be there in early results, just that probably not *all* of them. That is I called "middle-ground".
If we expect these failures to go away we'll need to be much more forceful about getting developers to run tox locally before they push.
We need to compare (and this is a rough example) the resource usage and developer time of complete batch results where you have a pep8 issue and an integration job issue in patchset one, fix both and all tests pass in patchset two against pep8 failure in patchset one, integration failure in patchset two, all tests pass in patchset three.
Today: patchset one: pep8 FAILURE unittest SUCCESS integration FAILURE patchset two: pep8 SUCCESS unittest SUCCESS integration SUCCESS
Proposed Future: patchset one: pep8 FAILURE unittest SUCCESS patchset two: pep8 SUCCESS unittest SUCCESS integration FAILURE patchset three: pep8 SUCCESS unittest SUCCESS integration SUCCESS
There are strictly more patchsets (developer roundtrips) and tests run in my contrived example. Reality will depend on how many iterations we actually see in the real world (eg are we fixing bugs reliably based on test feedback and how often do unittest and integration tests fail for different reasons).
Someone with a better background in statistics will probably tell me this approach is wrong, but using the elasticsearch tooling one approach may be to pick say 1k changes, then for each change identify which tests failed on subsequent patchsets? Then we'd be able to infer with some confidence the behaviors we have in the test suites around catching failures, whether they are independent across jobs and whether or not we fix them in batches.
Finally, I think we might also want to consider what the ideal is here. If we find we can optimize the current system for current behaviors, we also need to consider if that is worthwhile given an ideal. Should developers be expected to run unittests and linters locally before pushing? If so then optimizing for when they don't might be effort better spent on making it easier to run the unittests and linters locally and educating developers on how to do so. I think we'd also ideally expect our tests to pass once they run in the gate pipeline. Unfortunately I think our elastic-recheck data shows they often don't and more effort in fixing those failures would provide a dramatic increase in throughput (due to the compounding costs of subsequent gate resets).
My final take is, given some middle-ground solution, like I illustrated earlier this sub-thread, it might be worth it, and the effort for boosting up the total throughput of openstack CI system by a 6% is not so bad idea.
[0] http://paste.openstack.org/show/746503/
-- Best regards, Bogdan Dobrelya, Irc #bogdando
-- Best regards, Bogdan Dobrelya, Irc #bogdando
On Thu, Feb 28, 2019 at 9:27 AM Clark Boylan <cboylan@sapwetik.org> wrote:
On Thu, Feb 28, 2019, at 6:16 AM, Bogdan Dobrelya wrote:
On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote:
On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote:
snip
That said, I wouldn't push too hard in either direction until someone crunched the numbers and figured out how much time it would have saved to not run long tests on patch sets with failing unit tests. I feel like it's probably possible to figure that out, and if so then we should do it before making any big decisions on this.
For numbers the elastic-recheck tool [0] gives us fairly accurate tracking of which issues in the system cause tests to fail. You can use this as a starting point to potentially figure out how expensive indentation errors caught by the pep8 jobs ends up being or how often unittests fail. You probably need to tweak the queries there to get that specific though.
Periodically I also dump node resource utilization by project, repo, and job [1]. I haven't automated this because Tobiash has written a much better thing that has Zuul inject this into graphite and we should be able to set up a grafana dashboard for that in the future instead.
These numbers won't tell a whole story, but should paint a fairly accurate high level picture of the types of things we should look at to be more node efficient and "time in gate" efficient. Looking at these two really quickly myself it seems that job timeouts are a big cost (anyone looking into why our jobs timeout?).
[0] http://status.openstack.org/elastic-recheck/index.html [1] http://paste.openstack.org/show/746083/
Hope this helps, Clark
Here is some numbers [0] extracted via elastic-recheck console queries. It shows 6% of wasted failures because of tox issues in general, and 3% for tripleo projects in particular.
Are these wasted failures? The queries appear to be tracking valid failures of those jobs. These valid failures are then actionable feedback for developers to fix their changes.
If we expect these failures to go away we'll need to be much more forceful about getting developers to run tox locally before they push.
We need to compare (and this is a rough example) the resource usage and developer time of complete batch results where you have a pep8 issue and an integration job issue in patchset one, fix both and all tests pass in patchset two against pep8 failure in patchset one, integration failure in patchset two, all tests pass in patchset three.
Today: patchset one: pep8 FAILURE unittest SUCCESS integration FAILURE patchset two: pep8 SUCCESS unittest SUCCESS integration SUCCESS
Proposed Future: patchset one: pep8 FAILURE unittest SUCCESS patchset two: pep8 SUCCESS unittest SUCCESS integration FAILURE patchset three: pep8 SUCCESS unittest SUCCESS integration SUCCESS
There are strictly more patchsets (developer roundtrips) and tests run in my contrived example. Reality will depend on how many iterations we actually see in the real world (eg are we fixing bugs reliably based on test feedback and how often do unittest and integration tests fail for different reasons).
So I think the thought behind splitting this up is really project specific. For core openstack python projects it might make sense to not split them apart and run them all together. For things like TripleO/Kolla/Puppet/etc where we have layers of interactions that can be affected by the results from the linters/unit jobs it might make sense to split them out as Bogdan proposes. For example, in tripleo since we use packages, if the unit test fails the integration test may fail because when we go to built the package with the new source, the unit test int he package build fails. Thus we know that'll be a wasted execution and you won't actually get any results. An alternative Today: patchset one: pep8 SUCCESS unittest FAILURE integration FAILURE patchset two: pep8 SUCCESS unittest SUCCESS integration FAILURE patchset three: pep8 SUCCESS unittest SUCCESS integration SUCCESS Future: patchset one: pep8 SUCCESS unittest FAILURE integration SKIPPED patchset two: pep8 SUCCESS unittest SUCCESS integration FAILURE patchset three: pep8 SUCCESS unittest SUCCESS integration SUCCESS This may not be true for devstack but IMHO I would argue if the unit tests are failing, then the code is likely bad (backwards compatibility/wrong assumptions about change/etc) and we shouldn't be running an actual deployment.
Someone with a better background in statistics will probably tell me this approach is wrong, but using the elasticsearch tooling one approach may be to pick say 1k changes, then for each change identify which tests failed on subsequent patchsets? Then we'd be able to infer with some confidence the behaviors we have in the test suites around catching failures, whether they are independent across jobs and whether or not we fix them in batches.
Finally, I think we might also want to consider what the ideal is here. If we find we can optimize the current system for current behaviors, we also need to consider if that is worthwhile given an ideal. Should developers be expected to run unittests and linters locally before pushing? If so then optimizing for when they don't might be effort better spent on making it easier to run the unittests and linters locally and educating developers on how to do so. I think we'd also ideally expect our tests to pass once they run in the gate pipeline. Unfortunately I think our elastic-recheck data shows they often don't and more effort in fixing those failures would provide a dramatic increase in throughput (due to the compounding costs of subsequent gate resets).
Yes I would hope that developers are running tox -e pep8/tox -e py36 prior to pushing code. Should they be expected to run integration tests? Probably not. <rant> I feel that allowing integration tests without pushing for better unit tests actually reduces the coverage of the code in unit tests. If I make a change and don't bother it covering it with a decent set of unit tests but it passes the integration tests, would we is it tested well enough? From what I've seen the integration tests only test a fraction of actual functionality so I feel we really should be pushing people to improve the unit tests and I wonder if splitting this up would force the issue? </rant> Thanks, -Alex
My final take is, given some middle-ground solution, like I illustrated earlier this sub-thread, it might be worth it, and the effort for boosting up the total throughput of openstack CI system by a 6% is not so bad idea.
[0] http://paste.openstack.org/show/746503/
-- Best regards, Bogdan Dobrelya, Irc #bogdando
participants (3)
-
Alex Schultz
-
Bogdan Dobrelya
-
Clark Boylan