Open Stack

On Mon, Aug 19, 2013 at 10:47 AM, Daniel P. Berrange <berrange at redhat.com>wrote:

> In this thread about code review:
>
>
> http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html
>
> I mentioned that I thought there were too many blueprints created without
> sufficient supporting design information and were being used for "tickbox"
> process compliance only. I based this assertion on a gut feeling I have
> from experiance in reviewing.
>
> To try and get a handle on whether there is truely a problem, I used the
> launchpadlib API to extract some data on blueprints [1].
>
> In particular I was interested in seeing:
>
>   - What portion of blueprints have an URL containing an associated
>     design doc,
>
>   - How long the descriptive text was in typical blueprints
>
>   - Whether a blueprint was created before or after the dev period
>     started for that major release.
>
>
> The first two items are easy to get data on. On the second point, I redid
> line wrapping on description text to normalize the line count across all
> blueprints. This is because many blueprints had all their text on one
> giant long line, which would skew results. I thus wrapped all blueprints
> at 70 characters.
>
> The blueprint creation date vs release cycle dev start date is a little
> harder. I inferred the start date of each release, by using the end date
> of the previous release. This is probably a little out but hopefully not
> by enough to totally invalidate the usefulness of the stats below. Below,
> "Early" means created before start of devel, "Late" means created after
> the start of devel period.
>
> The data for the last 3 releases is:
>
>   Series: folsom
>     Specs: 178
>     Specs (no URL): 144
>     Specs (w/ URL): 34
>     Specs (Early): 38
>     Specs (Late): 140
>     Average lines: 5
>     Average words: 55
>
>
>   Series: grizzly
>     Specs: 227
>     Specs (no URL): 175
>     Specs (w/ URL): 52
>     Specs (Early): 42
>     Specs (Late): 185
>     Average lines: 5
>     Average words: 56
>
>
>   Series: havana
>     Specs: 415
>     Specs (no URL): 336
>     Specs (w/ URL): 79
>     Specs (Early): 117
>     Specs (Late): 298
>     Average lines: 6
>     Average words: 68
>
>
> Looking at this data there are 4 key take away points
>
>   - We're creating more blueprints in every release.
>
>   - Less than 1 in 4 blueprints has a link to a design document.
>
>   - The description text for blueprints is consistently short
>     (6 lines) across releases.
>
>
Thanks for running the numbers. My instinct told me this was the case, but
the data is especially helpful. Sometimes six lines is enough, but mostly I
rely on the linked spec for writing docs. If those links are at 25% that's
a bad trend.


>   - Less than 1 in 4 blueprints is created before the devel
>     period starts for a release.
>
>
I find this date mismatch especially intriguing, because the Foundation and
member company sponsors spend millions on Design Summits annually and
caters so much to getting people together in person. Yet the blueprints
aren't created in enough detail for discussion before the Summit dates? Is
that really what the data says? Is any one project skewing this (as in,
they haven't been at a Summit or they don't follow integrated release
dates?)

I'll dig in further to the data set below.


>
> You can view the full data set + the script to generate the
> data which you can look at to see if I made any logic mistakes:
>
>   http://berrange.fedorapeople.org/openstack-blueprints/
>
>
>
I wouldn't think to include marconi in the dataset as they've just asked
about incubation in June 2013. I think you excluded keystone. You also want
ceilometer and oslo if you already included heat. Is it fairly easy to
re-run? I'd like to see it re-run with the correct program listings.

Also please rerun with Swift excluded as their release dates are not on the
mark with the other projects. I'd like more info around the actual timing.



> There's only so much you can infer from stats like this, but IMHO think the
> stats show that we ought to think about how well we are using blueprints as
> design / feature approval / planning tools.
>
>
> That 3 in 4 blueprint lack any link to a design doc and have only 6 lines
> of
> text description, is a cause for concern IMHO. The blueprints should be
> giving
> code reviewers useful background on the motivation of the dev work & any
> design planning that took place. While there are no doubt some simple
> features
> where 6 lines of text is sufficient info in the blueprint, I don't think
> that
> holds true for the majority.
>
> In addition to helping code reviewers, the blueprints are also arguably a
> source of info for QA people testing OpenStack and for the docs teams
> documenting new features in each release. I'm not convinced that there is
> enough info in many of the blueprints to be of use to QA / docs people.
>
>
> The creation dates of the blueprints are also an interesting data point.
> If the design summit is our place for reviewing blueprints and 3 in 4
> blueprints in a release are created after the summit, that's alot of
> blueprints potentially missing summit discussions. On the other hand many
> blueprints will have corresponding discussions on mailing lists too,
> which is arguably just as good, or even better than, summit discussions.
>
> Based on the creation dates though & terseness of design info, I think
> there is a valid concern here that blueprints are being created just for
> reason of "tickbox" process compliance.
>

Possibly. Blueprints are created because it's part of our process, but I
also agree that blueprints themselves are not featured enough for the
increasing complexity of integrated projects.


>
> In theory we have an approval process for blueprints, but are we ever
> rejecting code submissions for blueprints which are not yet approved ?
> I've only noticed that happen a couple of times in Nova for things that
> were pretty clearly controversial.
>
> I don't intend to suggest that we have strict rules that all blueprints
> must be min X lines of text, or be created by date Y. It is important
> to keep the flexibility there to avoid development being drowned in
> process without benefits.
>

Agreed.

Anne


>
> I do think we have scope for being more rigourous in our review of
> blueprints, asking people to expand on the design info associated with
> a blueprint. Perhaps also require that a blueprint is actually approved
> by the core team before we go to the trouble of reviewing & approving
> the code implementing a blueprint in Gerrit.
>
> Regards,
> Daniel
>
> [1] http://berrange.fedorapeople.org/openstack-blueprints/blueprint.py
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/:|
> |: http://libvirt.org              -o-             http://virt-manager.org:|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/:|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc:|
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Anne Gentle
annegentle at justwriteclick.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130820/4fdccc0c/attachment.html>