Open Stack

Excerpts from Robert Collins's message of 2015-08-19 11:04:37 +1200:
> On 18 August 2015 at 01:46, Thierry Carrez <thierry at openstack.org> wrote:
> 
> Following on from the IRC discussion about release notes.
> 
> < ttx> * we want consumers of the stable branch (from random commit
> and from tagged versions) to get a valid release notes document
> < ttx> * Some things in that document (like OSSA numbers) we only know
> after the commit is pushed so we need a way to retroactively fix those
> 
> So 'random commit' here is a point in time - I think its reasonable to
> assert that if a commit isn't tagged, its release notes are not final.
> 
> So before I dive into detail, here's the basic design I think we can use.
> 
> 1) release notes are derived from the tree/git history. We might
> eventually use both, for instance.
> 2) commits are needed to change release notes.
> 3) Notes are mutable in that a clone today vs a clone tomorrow might
> have different release notes about the same change.
> 4) Notes are immutable in that for a given git hash/tag the release
> notes will be the same. Tagging a commit will change the version
> description but that is all.
> 
> Design assumptions:
> - We want to avoid merge hell when sheparding in a lot of
> release-note-worthy changes, which we expect to happen on stable
> branches always, and at release times on master branches.
> - We want writing a release note to be straight forward
> - We do not want release notes to be custom ordered within a release.
> As long as the ordering is consistent it is ok.
> - We must be able to entirely remove a release note.
> - We must not make things progressively slow down to a crawl over
> years of usage.

Adding a couple of requirements, based on our discussion yesterday:

- Release note authors shouldn't need to know any special values for
  naming their notes files (i.e., no change id or SHA value that has
  special meaning).
- It would be nice if it was somewhat easy to identify the file
  containing a release note on a particular topic.

> 
> Proposed data structure:
> - create a top level directory in each repo called release-notes
> - within that create a subdirectory called changes.
> - within the release-notes dir we place yaml files containing the
> release note inputs.
> - within the 'changes' subdirectory, the name of the yaml file will be
> the gerrit change id in a canonical form.
>    E.g. I1234abcd.yaml
>    This serves two purposes: it guarantees file name uniqueness (no
> merge conflicts) and lets us
>    determine which release to group it in (the most recent one, in
> case of merge+revert+merge patterns).

We changed this to using a "long enough" random number as a prefix, with
a "slug" value provided by the release note author to help identify what
is in the file.

I think maybe 8 hex digits for the prefix. Or should we go full UUID?

The slug might be something like "bug xxxx" or "removing option foo"
which would be converted to a canonical form, removing whitespace in the
filename. The slug can be changed, to allow for fixing typos, but the
prefix needs to remain the same in order for the note to be recognized
as the same item.

> - we also create files that roll up all the notes within history
> versions - named by the version. e.g. release-notes/1.2.3.yaml.

It's not clear why we need this, can you elaborate?

> 
> Yaml schema:
> prelude: prelude prose
> notes:
>  - note1
>  - note2
>  - note3

Thierry proposed 3 classifications for notes, which would then be
ordered separately: critical, security, and other. So I propose:

  prelude: >
    RST-formatted prelude text
  critical:
    - note 1
    - note 2
  security:
    - note 1
    - note 2
  other:
    - note 1
    - note 2

> 
> Processing:
> 1) determine the revisions we need to generate release notes for. By
> default generate notes for the current minor release. (E.g. if the
> tree version is 1.2.3.dev4 we would generate release notes for 1.2.0,
> 1.2.1, 1.2.2, 1.2.3[which dev4 is leading up to]).

Is there any reason, on a given branch, to not generate the notes for
the entire history of that branch back to the beginning of the project?

> 2) For each version: scan all the commits to determine gerrit change-id's.
>  i) read in all those change ids .yaml files and pull out any notes within them.
>  ii) read in any full version yaml file (and merge in its contained notes)
>  iii) Construct a markdown document as follows:
>   a) Sort any preludes (there should be only one at most, but lets not
> error if there are multiple)

Rather than sorting, which would change the order of notes as new items
are added, what about listing them in an order based on when they were
added in the history? We can track the first time a note file appears,
so we can maintain the order even if a note is modified.

>   b) sort any notes
>   c) for each note transform it into a bullet point by prepending its
> first line with '- ' and every subsequent line with '  ' (two spaces
> to form a hanging indent).
>   d) strip any trailing \n's from everything.
>   e) join the result - '\n'.join(chain(preludes, notes))
>  iv) we output the resulting file to release-notes/$version.md where
> $version is the pbr reported version for the tree (e.g. in the example
> above it would be 1.2.3.dev4, *not* 1.2.3).
> 3) possibly symlink the highest output version to ./RELEASENOTES.md or
> something, so other tooling can look for a constant value.

I'm curious about why we would want separate files for each version.
That's doable, but why not just generate the release notes history in a
single file?

> 
> If we want to put release notes in sdists, we can have pbr do this,
> otherwise it could just be built entirely separately.

We discussed dependency issues with putting this in pbr, so I was going
to start a separate tool that we can then add to our build toolchain.
The other benefit we identified for having it be separate is we can use
the same tool for repositories packaging things not written in python,
and therefore not packaged with pbr.

> 
> I recommend we start with it entirely separate: part of the
> release-management tooling.
> 
> -Rob
>