<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 19, 2013 at 11:47 AM, Daniel P. Berrange <span dir="ltr"><<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">In this thread about code review:<br>
<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html</a><br>
<br>
I mentioned that I thought there were too many blueprints created without<br>
sufficient supporting design information and were being used for "tickbox"<br>
process compliance only. I based this assertion on a gut feeling I have<br>
from experiance in reviewing.<br>
<br>
To try and get a handle on whether there is truely a problem, I used the<br>
launchpadlib API to extract some data on blueprints [1].<br>
<br>
In particular I was interested in seeing:<br>
<br>
- What portion of blueprints have an URL containing an associated<br>
design doc,<br>
<br>
- How long the descriptive text was in typical blueprints<br>
<br>
- Whether a blueprint was created before or after the dev period<br>
started for that major release.<br>
<br>
<br>
The first two items are easy to get data on. On the second point, I redid<br>
line wrapping on description text to normalize the line count across all<br>
blueprints. This is because many blueprints had all their text on one<br>
giant long line, which would skew results. I thus wrapped all blueprints<br>
at 70 characters.<br>
<br>
The blueprint creation date vs release cycle dev start date is a little<br>
harder. I inferred the start date of each release, by using the end date<br>
of the previous release. This is probably a little out but hopefully not<br>
by enough to totally invalidate the usefulness of the stats below. Below,<br>
"Early" means created before start of devel, "Late" means created after<br>
the start of devel period.<br>
<br>
The data for the last 3 releases is:<br>
<br>
Series: folsom<br>
Specs: 178<br>
Specs (no URL): 144<br>
Specs (w/ URL): 34<br>
Specs (Early): 38<br>
Specs (Late): 140<br>
Average lines: 5<br>
Average words: 55<br>
<br>
<br>
Series: grizzly<br>
Specs: 227<br>
Specs (no URL): 175<br>
Specs (w/ URL): 52<br>
Specs (Early): 42<br>
Specs (Late): 185<br>
Average lines: 5<br>
Average words: 56<br>
<br>
<br>
Series: havana<br>
Specs: 415<br>
Specs (no URL): 336<br>
Specs (w/ URL): 79<br>
Specs (Early): 117<br>
Specs (Late): 298<br>
Average lines: 6<br>
Average words: 68<br>
<br>
<br>
Looking at this data there are 4 key take away points<br>
<br>
- We're creating more blueprints in every release.<br>
<br>
- Less than 1 in 4 blueprints has a link to a design document.<br>
<br>
- The description text for blueprints is consistently short<br>
(6 lines) across releases.<br>
<br>
- Less than 1 in 4 blueprints is created before the devel<br>
period starts for a release.<br>
<br>
<br>
You can view the full data set + the script to generate the<br>
data which you can look at to see if I made any logic mistakes:<br>
<br>
<a href="http://berrange.fedorapeople.org/openstack-blueprints/" target="_blank">http://berrange.fedorapeople.org/openstack-blueprints/</a><br>
<br>
<br>
There's only so much you can infer from stats like this, but IMHO think the<br>
stats show that we ought to think about how well we are using blueprints as<br>
design / feature approval / planning tools.<br>
<br>
<br>
That 3 in 4 blueprint lack any link to a design doc and have only 6 lines of<br>
text description, is a cause for concern IMHO. The blueprints should be giving<br>
code reviewers useful background on the motivation of the dev work & any<br>
design planning that took place. While there are no doubt some simple features<br>
where 6 lines of text is sufficient info in the blueprint, I don't think that<br>
holds true for the majority.<br></blockquote><div><br></div><div>How many of those blueprints without links or expansive documentation are related to some other blueprint that does have documentation? In ceilometer, we have several blueprint "clusters" where one main blueprint has some documentation and the others are present for assigning and scheduling the work of a multi-part feature, or vice versa. For example, <a href="https://blueprints.launchpad.net/ceilometer/+spec/alarming">https://blueprints.launchpad.net/ceilometer/+spec/alarming</a> has no real doc because it's an "umbrella" blueprint for a lot of other pieces, many of which *do* have documentation.</div>
<div><br></div><div>Doug</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
In addition to helping code reviewers, the blueprints are also arguably a<br>
source of info for QA people testing OpenStack and for the docs teams<br>
documenting new features in each release. I'm not convinced that there is<br>
enough info in many of the blueprints to be of use to QA / docs people.<br>
<br>
<br>
The creation dates of the blueprints are also an interesting data point.<br>
If the design summit is our place for reviewing blueprints and 3 in 4<br>
blueprints in a release are created after the summit, that's alot of<br>
blueprints potentially missing summit discussions. On the other hand many<br>
blueprints will have corresponding discussions on mailing lists too,<br>
which is arguably just as good, or even better than, summit discussions.<br>
<br>
Based on the creation dates though & terseness of design info, I think<br>
there is a valid concern here that blueprints are being created just for<br>
reason of "tickbox" process compliance.<br>
<br>
In theory we have an approval process for blueprints, but are we ever<br>
rejecting code submissions for blueprints which are not yet approved ?<br>
I've only noticed that happen a couple of times in Nova for things that<br>
were pretty clearly controversial.<br>
<br>
I don't intend to suggest that we have strict rules that all blueprints<br>
must be min X lines of text, or be created by date Y. It is important<br>
to keep the flexibility there to avoid development being drowned in<br>
process without benefits.<br>
<br>
I do think we have scope for being more rigourous in our review of<br>
blueprints, asking people to expand on the design info associated with<br>
a blueprint. Perhaps also require that a blueprint is actually approved<br>
by the core team before we go to the trouble of reviewing & approving<br>
the code implementing a blueprint in Gerrit.<br>
<br>
Regards,<br>
Daniel<br>
<br>
[1] <a href="http://berrange.fedorapeople.org/openstack-blueprints/blueprint.py" target="_blank">http://berrange.fedorapeople.org/openstack-blueprints/blueprint.py</a><br>
<span class=""><font color="#888888">--<br>
|: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div><br></div></div>