<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 19, 2013 at 10:47 AM, Daniel P. Berrange <span dir="ltr"><<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">In this thread about code review:<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html</a><br>
<br>
I mentioned that I thought there were too many blueprints created without<br>
sufficient supporting design information and were being used for "tickbox"<br>
process compliance only. I based this assertion on a gut feeling I have<br>
from experiance in reviewing.<br>
<br>
To try and get a handle on whether there is truely a problem, I used the<br>
launchpadlib API to extract some data on blueprints [1].<br>
<br>
In particular I was interested in seeing:<br>
<br>
- What portion of blueprints have an URL containing an associated<br>
design doc,<br>
<br>
- How long the descriptive text was in typical blueprints<br>
<br>
- Whether a blueprint was created before or after the dev period<br>
started for that major release.<br>
<br>
<br>
The first two items are easy to get data on. On the second point, I redid<br>
line wrapping on description text to normalize the line count across all<br>
blueprints. This is because many blueprints had all their text on one<br>
giant long line, which would skew results. I thus wrapped all blueprints<br>
at 70 characters.<br>
<br>
The blueprint creation date vs release cycle dev start date is a little<br>
harder. I inferred the start date of each release, by using the end date<br>
of the previous release. This is probably a little out but hopefully not<br>
by enough to totally invalidate the usefulness of the stats below. Below,<br>
"Early" means created before start of devel, "Late" means created after<br>
the start of devel period.<br>
<br>
The data for the last 3 releases is:<br>
<br>
Series: folsom<br>
Specs: 178<br>
Specs (no URL): 144<br>
Specs (w/ URL): 34<br>
Specs (Early): 38<br>
Specs (Late): 140<br>
Average lines: 5<br>
Average words: 55<br>
<br>
<br>
Series: grizzly<br>
Specs: 227<br>
Specs (no URL): 175<br>
Specs (w/ URL): 52<br>
Specs (Early): 42<br>
Specs (Late): 185<br>
Average lines: 5<br>
Average words: 56<br>
<br>
<br>
Series: havana<br>
Specs: 415<br>
Specs (no URL): 336<br>
Specs (w/ URL): 79<br>
Specs (Early): 117<br>
Specs (Late): 298<br>
Average lines: 6<br>
Average words: 68<br>
<br>
<br>
Looking at this data there are 4 key take away points<br>
<br>
- We're creating more blueprints in every release.<br>
<br>
- Less than 1 in 4 blueprints has a link to a design document.<br>
<br>
- The description text for blueprints is consistently short<br>
(6 lines) across releases.<br>
<br></blockquote><div><br></div><div>Thanks for running the numbers. My instinct told me this was the case, but the data is especially helpful. Sometimes six lines is enough, but mostly I rely on the linked spec for writing docs. If those links are at 25% that's a bad trend.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
- Less than 1 in 4 blueprints is created before the devel<br>
period starts for a release.<br>
<br></blockquote><div><br></div><div>I find this date mismatch especially intriguing, because the Foundation and member company sponsors spend millions on Design Summits annually and caters so much to getting people together in person. Yet the blueprints aren't created in enough detail for discussion before the Summit dates? Is that really what the data says? Is any one project skewing this (as in, they haven't been at a Summit or they don't follow integrated release dates?)</div>
<div><br></div><div>I'll dig in further to the data set below.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
You can view the full data set + the script to generate the<br>
data which you can look at to see if I made any logic mistakes:<br>
<br>
<a href="http://berrange.fedorapeople.org/openstack-blueprints/" target="_blank">http://berrange.fedorapeople.org/openstack-blueprints/</a><br>
<br>
<br></blockquote><div><br></div><div>I wouldn't think to include marconi in the dataset as they've just asked about incubation in June 2013. I think you excluded keystone. You also want ceilometer and oslo if you already included heat. Is it fairly easy to re-run? I'd like to see it re-run with the correct program listings.</div>
<div><br></div><div>Also please rerun with Swift excluded as their release dates are not on the mark with the other projects. I'd like more info around the actual timing.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
There's only so much you can infer from stats like this, but IMHO think the<br>
stats show that we ought to think about how well we are using blueprints as<br>
design / feature approval / planning tools.<br>
<br>
<br>
That 3 in 4 blueprint lack any link to a design doc and have only 6 lines of<br>
text description, is a cause for concern IMHO. The blueprints should be giving<br>
code reviewers useful background on the motivation of the dev work & any<br>
design planning that took place. While there are no doubt some simple features<br>
where 6 lines of text is sufficient info in the blueprint, I don't think that<br>
holds true for the majority.<br>
<br>
In addition to helping code reviewers, the blueprints are also arguably a<br>
source of info for QA people testing OpenStack and for the docs teams<br>
documenting new features in each release. I'm not convinced that there is<br>
enough info in many of the blueprints to be of use to QA / docs people.<br>
<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
The creation dates of the blueprints are also an interesting data point.<br>
If the design summit is our place for reviewing blueprints and 3 in 4<br>
blueprints in a release are created after the summit, that's alot of<br>
blueprints potentially missing summit discussions. On the other hand many<br>
blueprints will have corresponding discussions on mailing lists too,<br>
which is arguably just as good, or even better than, summit discussions.<br>
<br>
Based on the creation dates though & terseness of design info, I think<br>
there is a valid concern here that blueprints are being created just for<br>
reason of "tickbox" process compliance.<br></blockquote><div><br></div><div>Possibly. Blueprints are created because it's part of our process, but I also agree that blueprints themselves are not featured enough for the increasing complexity of integrated projects.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
In theory we have an approval process for blueprints, but are we ever<br>
rejecting code submissions for blueprints which are not yet approved ?<br>
I've only noticed that happen a couple of times in Nova for things that<br>
were pretty clearly controversial.<br>
<br>
I don't intend to suggest that we have strict rules that all blueprints<br>
must be min X lines of text, or be created by date Y. It is important<br>
to keep the flexibility there to avoid development being drowned in<br>
process without benefits.<br></blockquote><div><br></div><div>Agreed. </div><div><br></div><div>Anne</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I do think we have scope for being more rigourous in our review of<br>
blueprints, asking people to expand on the design info associated with<br>
a blueprint. Perhaps also require that a blueprint is actually approved<br>
by the core team before we go to the trouble of reviewing & approving<br>
the code implementing a blueprint in Gerrit.<br>
<br>
Regards,<br>
Daniel<br>
<br>
[1] <a href="http://berrange.fedorapeople.org/openstack-blueprints/blueprint.py" target="_blank">http://berrange.fedorapeople.org/openstack-blueprints/blueprint.py</a><br>
<span><font color="#888888">--<br>
|: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Anne Gentle<br><a href="mailto:annegentle@justwriteclick.com" target="_blank">annegentle@justwriteclick.com</a>
</div></div>