Open Stack

On 10/21/2015 7:43 AM, Kashyap Chamarthy wrote:
> Background
> ----------
>
> Oslo Guru Meditation (error) Reports (GMR)[*] are a useful debugging
> mechanism that allows one to capture the current state of a Nova
> process/executable (e.g. `nova-compute`, `nova-api`, etc).
>
> The way to generate the error report is to supply the 'User-defined
> signal', SIGUSR1, when killing a Nova process.  E.g.
>
>      $ kill -USR1 `pgrep nova-compute`
>
> which results in GMR being printed to your standard error ('stderr')
> stream, wherever it ends up being redirected to (e.g. to a corresponding
> Nova process-specific log file, otherwise, on systemd-enabled systems,
> to its journal).
>
>
> Change in Mitaka (and above)
> ----------------------------
>
>  From the upcoming Mitaka release onwards, the default expected UNIX
> signal to generate GMR has been changed[1] from USR1 to USR2 (another
> User-defined singal), because the USR1 is reserved by Apache 'mod_wsgi'
> for its own purpose.
>
> So, to generate GMR, from Mitaka release:
>
>      $ kill -USR2 `pgrep nova-compute`
>
> A corresponding Nova documentation change[2] has been submitted to
> reflect this new reality.
>
>
> [1] https://review.openstack.org/#/c/223133/ -- guru_meditation_report:
>      Use SIGUSR2 instead of SIGUSR1
> [2] https://review.openstack.org/#/c/227779/ -- doc: gmr: Update
>      instructions to generate GMR error reports
>
>
> [*] References
> --------------
>
> Related reading:
>
> - http://docs.openstack.org/developer/nova/gmr.html
> - http://docs.openstack.org/developer/oslo.reports/usage.html
> - https://wiki.openstack.org/wiki/GuruMeditationReport
> - https://www.berrange.com/posts/2015/02/19/nova-and-its-use-of-olso-incubator-guru-meditation-reports/
>

Looks like this broke some tooling in the gate job runs where gmr's are 
created at the end of the service logs when the services exit. Here is a 
mitaka change with grenade on the liberty side with the gmr at the end:

http://logs.openstack.org/97/227897/13/check/gate-grenade-dsvm/27723a9/logs/old/screen-n-cpu.txt.gz

And on the new side it's gone:

http://logs.openstack.org/97/227897/13/check/gate-grenade-dsvm/27723a9/logs/new/screen-n-cpu.txt.gz

So obviously an upgrade impact, I'm hoping we get this into the liberty 
release notes as something to change when people move up to oslo.reports 
1.6.0.

We should also get the gate tooling fixed around this, I'm not sure 
where that was configured/triggered though, sdague probably knows.

-- 

Thanks,

Matt Riedemann