<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:798260243;
mso-list-type:hybrid;
mso-list-template-ids:2003477154 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:1732607086;
mso-list-type:hybrid;
mso-list-template-ids:-1543487322 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2
{mso-list-id:1910384072;
mso-list-type:hybrid;
mso-list-template-ids:1512585542 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l2:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">TL;DR: I consider the poor state of log consistency a major impediment for more widespread adoption of OpenStack and would like to volunteer to own this cross-functional process to begin to unify and standardize logging messages and
attributes for Kilo while dealing with the most egregious issues as the community identifies them.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Recap from some mail threads:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">From Sean Dague on Kilo cycle goals:<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in">2. Consistency in southbound interfaces (Logging first)<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoPlainText" style="margin-left:.5in">Logging and notifications are south bound interfaces from OpenStack providing information to people, or machines, about what is going on.<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in">There is also a 3rd proposed south bound with osprofiler.<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoPlainText" style="margin-left:.5in">For Kilo: I think it's reasonable to complete the logging standards and implement them. I expect notifications (which haven't quite kicked off) are going to take 2 cycles.<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoPlainText" style="margin-left:.5in">I'd honestly *really* love to see a unification path for all the the southbound parts, logging, osprofiler, notifications, because there is quite a bit of overlap in the instrumentation/annotation inside the
main code for all of these.<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal">And from Doug Hellmann:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">1. Sean has done a lot of analysis and started a spec on standardizing logging guidelines where he is gathering input from developers, deployers, and operators [1]. Because it is far enough for us to see real progress,
it’s a good place for us to start experimenting with how to drive cross-project initiatives involving code and policy changes from outside of a single project. We have a couple of potentially related specs in Oslo as part of the oslo.log graduation work [2]
[3], but I think most of the work will be within the applications.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[1] <a href="https://review.openstack.org/#/c/91446/">https://review.openstack.org/#/c/91446/</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[2] <a href="https://blueprints.launchpad.net/oslo.log/+spec/app-agnostic-logging-parameters">https://blueprints.launchpad.net/oslo.log/+spec/app-agnostic-logging-parameters</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[3] <a href="https://blueprints.launchpad.net/oslo.log/+spec/remove-context-adapter">https://blueprints.launchpad.net/oslo.log/+spec/remove-context-adapter</a><o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoPlainText">And from James Blair:<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in">1) Improve log correlation and utility<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoPlainText" style="margin-left:.5in">If we're going to improve the stability of OpenStack, we have to be able to understand what's going on when it breaks. That's both true as developers when we're trying to diagnose a failure in an integration
test, and it's true for operators who are all too often diagnosing the same failure in a real deployment. Consistency in logging across projects as well as a cross-project request token would go a long way toward this.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">While I am not currently managing an OpenStack deployment, writing tests or code, or debugging the stack, I have spent many years doing just that. Through QA, Ops and Customer support, I have come to revel in good logging and log messages
and curse the holes and vagaries in many systems.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Defining/refining logs to be useful and usable is a cross-functional effort that needs to include:<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Operators<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>QA<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>End Users<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Community managers<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Tech Pubs<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Translators<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Developers<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>TC (which provides the forum and impetus for all the projects to cooperate on this)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">At the moment, I think this effort may best work under the auspices of Oslo (oslo.log), I’d love to hear other proposals.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the beginnings of my proposal of how to attack and subdue the painful state of logs:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l2 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Post this email to the MLs (dev, ops, enduser) to get feedback, garner support and participants in the process<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.75in">(Done;-)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l2 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>In parallel:<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l2 level2 lfo2">
<![if !supportLists]><span style="font-family:"Courier New""><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Collect up problems, issues, ideas, solutions on an etherpad
<a href="https://etherpad.openstack.org/p/Log-Rationalization">https://etherpad.openstack.org/p/Log-Rationalization</a> where anyone in the communities can post.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l2 level2 lfo2">
<![if !supportLists]><span style="font-family:"Courier New""><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Categorize reported Log issues into classes (already identified classes):<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-.25in;mso-list:l2 level3 lfo2">
<![if !supportLists]><span style="font-family:Wingdings"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Format Consistency across projects<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-.25in;mso-list:l2 level3 lfo2">
<![if !supportLists]><span style="font-family:Wingdings"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Log level definition and categorization across classes<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-.25in;mso-list:l2 level3 lfo2">
<![if !supportLists]><span style="font-family:Wingdings"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Time syncing entries across tens of logfiles<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-.25in;mso-list:l2 level3 lfo2">
<![if !supportLists]><span style="font-family:Wingdings"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Relevancy/usefulness of information provided within messages<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-.25in;mso-list:l2 level3 lfo2">
<![if !supportLists]><span style="font-family:Wingdings"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Etc (missing a lot here, but I’m sure folks will speak up)<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l2 level2 lfo2">
<![if !supportLists]><span style="font-family:"Courier New""><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Analyze existing log message formats, standards across integrated projects<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l2 level2 lfo2">
<![if !supportLists]><span style="font-family:"Courier New""><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>File bugs where issues identified are actual project bugs<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l2 level2 lfo2">
<![if !supportLists]><span style="font-family:"Courier New""><span style="mso-list:Ignore">o<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Build a session outline for F2F working session at the Paris Design Summit<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l2 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>At the Paris Design Summit, use a session and/or pod discussions to set priorities, recruit contributors, start and/or flesh out specs and blueprints<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l2 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Proceed according to priorities, specs, blueprints, contributions and changes as needed as the work progresses.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l2 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Keep an active and open rapport and reporting process for the user community to comment and participate in the processes.<o:p></o:p></p>
<p class="MsoNormal">Measures of success:<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo3"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Log messages provide consistency of format enough for productive mining through operator writable scripts<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo3"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Problem debugging is simplified through the ability to trust timestamps across all OpenStack logs (and use scripts to get to the time you want in any/all of the logfiles)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo3"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Standards for format, content, levels and translations have been proposed and agreed to be adopted across all OpenStack integrated projects<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo3"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>The user communities demonstrate an increased level of trust and decreased level of frustration with OpenStack logging (surveys, bug reports, other measures?)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo3"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>The log team can disband<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I expect that getting the logs in very good shape will take more than just the Kilo timeframe, but once momentum has built, which should happened during Kilo, the process should move very quickly. A lot of this could be handled through
“while you’re in there” or “low hanging fruit” once the standards are established. The bigger win will be if we can ensure what we define/design is extensible over the longer life of OpenStack.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">--Rocky<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>