<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Calibri Light";
panose-1:2 15 3 2 2 2 4 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:Calibri;}
h1
{mso-style-priority:9;
mso-style-link:"Heading 1 Char";
margin-top:24.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:14.0pt;
font-family:"Calibri Light";
color:#2F5496;}
h2
{mso-style-priority:9;
mso-style-link:"Heading 2 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:13.0pt;
font-family:"Calibri Light";
color:#4472C4;}
h3
{mso-style-priority:9;
mso-style-link:"Heading 3 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:11.0pt;
font-family:"Calibri Light";
color:#4472C4;}
h4
{mso-style-priority:9;
mso-style-link:"Heading 4 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:11.0pt;
font-family:"Calibri Light";
color:#4472C4;
font-style:italic;}
h5
{mso-style-priority:9;
mso-style-link:"Heading 5 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:11.0pt;
font-family:"Calibri Light";
color:#1F3763;
font-weight:normal;}
h6
{mso-style-priority:9;
mso-style-link:"Heading 6 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:11.0pt;
font-family:"Calibri Light";
color:#1F3763;
font-weight:normal;
font-style:italic;}
p.MsoHeading7, li.MsoHeading7, div.MsoHeading7
{mso-style-priority:9;
mso-style-link:"Heading 7 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:11.0pt;
font-family:"Calibri Light";
color:#404040;
font-style:italic;}
p.MsoHeading8, li.MsoHeading8, div.MsoHeading8
{mso-style-priority:9;
mso-style-link:"Heading 8 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:10.0pt;
font-family:"Calibri Light";
color:#4472C4;}
p.MsoHeading9, li.MsoHeading9, div.MsoHeading9
{mso-style-priority:9;
mso-style-link:"Heading 9 Char";
margin-top:10.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:10.0pt;
font-family:"Calibri Light";
color:#404040;
font-style:italic;}
p.MsoCaption, li.MsoCaption, div.MsoCaption
{mso-style-priority:35;
margin:0cm;
margin-bottom:.0001pt;
font-size:9.0pt;
font-family:Calibri;
color:#4472C4;
font-weight:bold;}
p.MsoTitle, li.MsoTitle, div.MsoTitle
{mso-style-priority:10;
mso-style-link:"Title Char";
margin-top:0cm;
margin-right:0cm;
margin-bottom:15.0pt;
margin-left:0cm;
mso-add-space:auto;
font-size:26.0pt;
font-family:"Calibri Light";
color:#323E4F;
letter-spacing:.25pt;}
p.MsoTitleCxSpFirst, li.MsoTitleCxSpFirst, div.MsoTitleCxSpFirst
{mso-style-priority:10;
mso-style-link:"Title Char";
mso-style-type:export-only;
margin:0cm;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:26.0pt;
font-family:"Calibri Light";
color:#323E4F;
letter-spacing:.25pt;}
p.MsoTitleCxSpMiddle, li.MsoTitleCxSpMiddle, div.MsoTitleCxSpMiddle
{mso-style-priority:10;
mso-style-link:"Title Char";
mso-style-type:export-only;
margin:0cm;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:26.0pt;
font-family:"Calibri Light";
color:#323E4F;
letter-spacing:.25pt;}
p.MsoTitleCxSpLast, li.MsoTitleCxSpLast, div.MsoTitleCxSpLast
{mso-style-priority:10;
mso-style-link:"Title Char";
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:15.0pt;
margin-left:0cm;
mso-add-space:auto;
font-size:26.0pt;
font-family:"Calibri Light";
color:#323E4F;
letter-spacing:.25pt;}
p.MsoSubtitle, li.MsoSubtitle, div.MsoSubtitle
{mso-style-priority:11;
mso-style-link:"Subtitle Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri Light";
color:#4472C4;
letter-spacing:.75pt;
font-style:italic;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoNoSpacing, li.MsoNoSpacing, div.MsoNoSpacing
{mso-style-priority:1;
mso-style-link:"No Spacing Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Times New Roman";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:11.0pt;
font-family:Calibri;}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst
{mso-style-priority:34;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:11.0pt;
font-family:Calibri;}
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle
{mso-style-priority:34;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:11.0pt;
font-family:Calibri;}
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast
{mso-style-priority:34;
mso-style-type:export-only;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:11.0pt;
font-family:Calibri;}
p.MsoQuote, li.MsoQuote, div.MsoQuote
{mso-style-priority:29;
mso-style-link:"Quote Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:Calibri;
color:black;
font-style:italic;}
p.MsoIntenseQuote, li.MsoIntenseQuote, div.MsoIntenseQuote
{mso-style-priority:30;
mso-style-link:"Intense Quote Char";
margin-top:10.0pt;
margin-right:46.8pt;
margin-bottom:14.0pt;
margin-left:46.8pt;
font-size:11.0pt;
font-family:Calibri;
color:#4472C4;
font-weight:bold;
font-style:italic;}
span.MsoSubtleEmphasis
{mso-style-priority:19;
color:gray;
font-style:italic;}
span.MsoIntenseEmphasis
{mso-style-priority:21;
color:#4472C4;
font-weight:bold;
font-style:italic;}
span.MsoSubtleReference
{mso-style-priority:31;
font-variant:small-caps;
color:#ED7D31;
text-decoration:underline;}
span.MsoIntenseReference
{mso-style-priority:32;
font-variant:small-caps;
color:#ED7D31;
letter-spacing:.25pt;
font-weight:bold;
text-decoration:underline;}
span.MsoBookTitle
{mso-style-priority:33;
font-variant:small-caps;
letter-spacing:.25pt;
font-weight:bold;}
p.MsoTocHeading, li.MsoTocHeading, div.MsoTocHeading
{mso-style-priority:39;
margin-top:24.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:14.0pt;
font-family:"Calibri Light";
color:#2F5496;
font-weight:bold;}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-link:"Heading 1";
font-family:"Calibri Light";
color:#2F5496;
font-weight:bold;}
span.Heading2Char
{mso-style-name:"Heading 2 Char";
mso-style-priority:9;
mso-style-link:"Heading 2";
font-family:"Calibri Light";
color:#4472C4;
font-weight:bold;}
span.Heading3Char
{mso-style-name:"Heading 3 Char";
mso-style-priority:9;
mso-style-link:"Heading 3";
font-family:"Calibri Light";
color:#4472C4;
font-weight:bold;}
span.Heading4Char
{mso-style-name:"Heading 4 Char";
mso-style-priority:9;
mso-style-link:"Heading 4";
font-family:"Calibri Light";
color:#4472C4;
font-weight:bold;
font-style:italic;}
span.Heading5Char
{mso-style-name:"Heading 5 Char";
mso-style-priority:9;
mso-style-link:"Heading 5";
font-family:"Calibri Light";
color:#1F3763;}
span.Heading6Char
{mso-style-name:"Heading 6 Char";
mso-style-priority:9;
mso-style-link:"Heading 6";
font-family:"Calibri Light";
color:#1F3763;
font-style:italic;}
span.Heading7Char
{mso-style-name:"Heading 7 Char";
mso-style-priority:9;
mso-style-link:"Heading 7";
font-family:"Calibri Light";
color:#404040;
font-style:italic;}
span.Heading8Char
{mso-style-name:"Heading 8 Char";
mso-style-priority:9;
mso-style-link:"Heading 8";
font-family:"Calibri Light";
color:#4472C4;}
span.Heading9Char
{mso-style-name:"Heading 9 Char";
mso-style-priority:9;
mso-style-link:"Heading 9";
font-family:"Calibri Light";
color:#404040;
font-style:italic;}
span.TitleChar
{mso-style-name:"Title Char";
mso-style-priority:10;
mso-style-link:Title;
font-family:"Calibri Light";
color:#323E4F;
letter-spacing:.25pt;}
span.SubtitleChar
{mso-style-name:"Subtitle Char";
mso-style-priority:11;
mso-style-link:Subtitle;
font-family:"Calibri Light";
color:#4472C4;
letter-spacing:.75pt;
font-style:italic;}
span.NoSpacingChar
{mso-style-name:"No Spacing Char";
mso-style-priority:1;
mso-style-link:"No Spacing";}
span.QuoteChar
{mso-style-name:"Quote Char";
mso-style-priority:29;
mso-style-link:Quote;
color:black;
font-style:italic;}
span.IntenseQuoteChar
{mso-style-name:"Intense Quote Char";
mso-style-priority:30;
mso-style-link:"Intense Quote";
color:#4472C4;
font-weight:bold;
font-style:italic;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle47
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle48
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle49
{mso-style-type:personal;
font-family:Calibri;
color:#1F497D;}
span.EmailStyle50
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle51
{mso-style-type:personal;
font-family:Calibri;
color:#1F497D;}
span.EmailStyle52
{mso-style-type:personal;
font-family:Calibri;
color:#1F497D;}
span.EmailStyle53
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle54
{mso-style-type:personal;
font-family:Calibri;
color:#1F497D;}
span.EmailStyle55
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:930897167;
mso-list-type:hybrid;
mso-list-template-ids:-1431404656 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:38.4pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:74.4pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:110.4pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:146.4pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:182.4pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:218.4pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:254.4pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:290.4pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:326.4pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1936087305;
mso-list-type:hybrid;
mso-list-template-ids:-929252084 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Paul,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Unfortunately I can’t figure out from the log what went wrong. It seems like the ‘up’ alarms are ignored. Two things that I would try next:<o:p></o:p></p>
<p class="MsoListParagraphCxSpFirst" style="text-indent:-18.0pt;mso-list:l1 level1 lfo3">
<![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span dir="LTR"></span>Try calling an event with ‘status’:’up’ and see if it works. This is working for sure in my environment<o:p></o:p></p>
<p class="MsoListParagraphCxSpLast" style="text-indent:-18.0pt;mso-list:l1 level1 lfo3">
<![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span dir="LTR"></span>I suspect that the problem is somewhere in AlarmDriverBase._filter_and_cache_alarms(). Basically it should search the old alarm in the cache and update it. Try to add many debug messages so we could see
the cache, the new alarm and the old alarm.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Let me know if it helped.<o:p></o:p></p>
<p class="MsoNormal">Ifat.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Paul Vaduva <Paul.Vaduva@enea.com><br>
<b>Date: </b>Wednesday, 21 February 2018 at 19:11<br>
<b>To: </b>"Afek, Ifat (Nokia - IL/Kfar Sava)" <ifat.afek@nokia.com>, "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br>
<b>Cc: </b>Ciprian Barbu <Ciprian.Barbu@enea.com><br>
<b>Subject: </b>RE: [openstack-dev] [vitrage] Vitrage alarm processing behavior</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman""><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D">Hi Ifat,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Link to cuted log version </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><a href="https://hastebin.com/upokifinuq.py">https://hastebin.com/upokifinuq.py</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Plus full graph.log attached plus code for driver.py with Logging modifications
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Paul</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Afek, Ifat (Nokia - IL/Kfar Sava) [mailto:ifat.afek@nokia.com]
<br>
<b>Sent:</b> Wednesday, February 21, 2018 6:18 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org><br>
<b>Cc:</b> Ciprian Barbu <Ciprian.Barbu@enea.com><br>
<b>Subject:</b> Re: [openstack-dev] [vitrage] Vitrage alarm processing behavior<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Hi Paul,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I suggest that you do the following:<o:p></o:p></p>
<p class="MsoListParagraphCxSpFirst" style="margin-left:38.4pt;mso-add-space:auto;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span dir="LTR"></span>Add a LOG message at the end of _get_alarms to print all alarms that are returned by this function<o:p></o:p></p>
<p class="MsoListParagraphCxSpLast" style="margin-left:38.4pt;mso-add-space:auto;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span dir="LTR"></span>Restart vitrage-graph and send me its log. I’d like to see if there is any difference between the alarm that is raised and the alarm that is deleted.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Ifat.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Paul Vaduva <<a href="mailto:Paul.Vaduva@enea.com">Paul.Vaduva@enea.com</a>><br>
<b>Reply-To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Date: </b>Wednesday, 21 February 2018 at 16:30<br>
<b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Cc: </b>Ciprian Barbu <<a href="mailto:Ciprian.Barbu@enea.com">Ciprian.Barbu@enea.com</a>><br>
<b>Subject: </b>Re: [openstack-dev] [vitrage] Vitrage alarm processing behavior</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman""> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D">I attached also the driver.py that I am using.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Paul Vaduva [<a href="mailto:Paul.Vaduva@enea.com">mailto:Paul.Vaduva@enea.com</a>]
<br>
<b>Sent:</b> Wednesday, February 21, 2018 3:22 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Cc:</b> Ciprian Barbu <<a href="mailto:Ciprian.Barbu@enea.com">Ciprian.Barbu@enea.com</a>><br>
<b>Subject:</b> [Attachment removed] Re: [openstack-dev] [vitrage] Vitrage alarm processing behavior<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi Ifat,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Sorry for the late reply. </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">To answer your questions </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I started as an example from the doctor datasource (or a porting of it for the 1.3.0 version of vitrage) but will call it something different so no need to worry about conflicting with present doctor datasource.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I added polling alarms to it but I have a more particular use case:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">* I get compute host down alarm on event</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">* I can't get host up event or it's an intricate sollution to implement</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I tried to see if I can make the following scenario work:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Let's call Scenario I</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">* Get a compute host down event (Raisng an alarm)
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">* Periodically poll for the status of the compute in method "def _get_alarms(self):" of the Driver object
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Both type of Interactions seem to work (polling and event based).
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">However now comes the tricky part. I would need for the alarms (with status up / compute host up) returned by method "def _get_alarms(self):" of this Driver object to cancel/clear the compute host down alarms
raised by event. This unfortunatelly does not happen. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Oddely enough there is a mimic of this scenario that works but is not robust enough for out needs.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Let's call Scenario II:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">* Gettting an event with compute host down(when one of our compute actually goes down)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">* Polling alarm (also compute host down) is raised and somehow overwrites the event based one (I can see the updated time).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">* After a while the actual compute reboots and polling for the alarms returns an alarm with status up that in this case clears the previous (I assume polling type now) alarm.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Now I can't understand why this second scenario works and the first one does not.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">It seems as the same alarm type (compute host down with status down) obtained by polling can overwrite an identical type and status alarm raised by event, but An alarm with an updated status (i. e. up) got by
polling mode cannot overwrite / clear and alarm with status down got by an event.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I am wondering if there is a reason of this behavior and if there is a way to modify it or is it a bug.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">For the event's generation I use modified version of zabbix_vitrage.py script that publishes to rabbitmq
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">vitrage_notifications.info queue. I have attached this python script.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">The code is still experimental But I wanted to know if it's logically posible to create The scenario we need, Scenario I.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Best Regards</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Paul</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Afek, Ifat (Nokia - IL/Kfar Sava) [<a href="mailto:ifat.afek@nokia.com">mailto:ifat.afek@nokia.com</a>]
<br>
<b>Sent:</b> Wednesday, February 7, 2018 7:16 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Cc:</b> Ciprian Barbu <<a href="mailto:Ciprian.Barbu@enea.com">Ciprian.Barbu@enea.com</a>><br>
<b>Subject:</b> Re: [openstack-dev] [vitrage] Vitrage alarm processing behavior<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Hi Paul,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I’m glad that my fix helped. <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Regarding the Doctor datasource: the purpose of this datasource was to be used by the Doctor test scripts. Do you intend to modify it, or to create a new similar datasource that also supports polling? Modifying the existing datasource could
be problematic, since we need to make sure the existing functionality and tests stay the same.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In general, most of our datasources support both polling and notifications. A simple example is the Cinder datasource [1]. For example of an alarm datasource, you can look at Zabbix datasource [2]. You can also go over the documentation
of how to add a new datasource [3].<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">As for your question, it is the responsibility of the datasource to clear the alarms that it created. For the Doctor datasource, you can send an event with “status”:”up” in the details and the datasource will clear the alarm.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[1] <a href="https://url10.mailanyone.net/v1/?m=1ejTL3-0003ZV-4n&i=57e1b682&c=Pe0SmnJrux3qg2aeVKwciP-we0PY0bk3JoTO_20fQHQ70cIoAgpMPXrk8JuN_BWqpqnpygQerGyzW2Snm5KfUQ7Y-INhOKG5eybo-thEBodvAhGSFpyXWQxPXS0Auc9aF0vGy2Ea4hrWfL6eeD0bQycBJN8lTLZnuIQx59ZeULyqstlxVBL34dcnQOFQf-5nS76n_X9owe_iNZrV57fmTrGKDogeMocpOJwlz9vnzzCDaL7RjjqCRLcbAxwkyRas3lujR6oZKt9NK1NBb-hb3uc721qSI6SR8SVN6zZGjQE">
https://github.com/openstack/vitrage/tree/master/vitrage/datasources/cinder/volume</a>
<o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://url10.mailanyone.net/v1/?m=1ejTL3-0003ZV-4n&i=57e1b682&c=uGgIuECLH17WmCqispfyornk-y9i4E2eyyvxC5fH2sepif7vNt0e_Op9ifHIcOuZLWy4fzJMsbItzfWpk5qNeYW2O3iEr5sPuXnguxKSRm6yrD12oGtjjJibDR7oVJnkQSNtu5caCM1BoguJiXBL7WisodfHGVdbYJDe2W2m11dc3ZmARXYI1FlmVWOPQiAGlzNtUgcQ_wpYwHtTJJaur8wiS415nr2oRHwU4C9hawW9HWktVVEH877WI_P1xf3VI1PjGVf75imEW-bHo3lAtCIAv4hWKcrxtHdL48oP7kQ">
https://github.com/openstack/vitrage/tree/master/vitrage/datasources/zabbix</a><o:p></o:p></p>
<p class="MsoNormal">[3] <a href="https://url10.mailanyone.net/v1/?m=1ejTL3-0003ZV-4n&i=57e1b682&c=A08vm8gwOUlRCFuV_ZDNRKrFdo7lGQmqtrZE-ZXEB6yLzcanUHFW1Aue5PnhXvrALgd0apyK5SAU9-PPc5Pqi5uod_I2JAHONug3ILQ9e3RvoKWyoYcuehJzRa3bqH3g_r5GQnKIRRNnYccSg6T4wkA-Wl6PHZ7KXq7cYp9zY7Fhz2jCK_zTUNBGJvLR2W_bqwPdTe2iyetPXPa0N_JrF38KrkUOVppDYgfi4_onM9N6QUUEECArxlYPl-T3xDM5cMSrTf9iE38OJrg_nKG8Fkwr7rAV5L8tAEZ5vGMDQxc">
https://docs.openstack.org/vitrage/latest/contributor/add-new-datasource.html</a>
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Best Regards,<o:p></o:p></p>
<p class="MsoNormal">Ifat.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Paul Vaduva <</span><a href="mailto:Paul.Vaduva@enea.com">Paul.Vaduva@enea.com</a><span style="color:black">><br>
<b>Reply-To: </b>"OpenStack Development Mailing List (not for usage questions)" <</span><a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><span style="color:black">><br>
<b>Date: </b>Wednesday, 7 February 2018 at 15:50<br>
<b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <</span><a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><span style="color:black">><br>
<b>Cc: </b>Ciprian Barbu <</span><a href="mailto:Ciprian.Barbu@enea.com">Ciprian.Barbu@enea.com</a><span style="color:black">><br>
<b>Subject: </b>Re: [openstack-dev] [vitrage] Vitrage alarm processing behavior</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman""> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D">Hi Ifat,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Yes I’ve checked the 1.3.1 refers to a deb package (python-vitrage) version built by us, so the git tag used to build that deb is 1.3.0.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">But I also backported doctor datasource from vitreage git master branch.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I also noticed that when I configure snapshots_interval=10 I also get this exception in
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">/var/log/vitrage/graph.log around the time the alarms disapear.
</span><o:p></o:p></p>
<p class="MsoNormal"><a href="https://url10.mailanyone.net/v1/?m=1ejTL3-0003ZV-4n&i=57e1b682&c=dIFoa_mHWzOpmJ9KV346afu6D9E3lEuyvUD6vwgvXW-hvbG45rR_s7mUjXnZgBFfnmwyP_2yo8TbtBKzX2-NatWbW9ZEbu-UWM9KzGIZ_t9Gd3XlOHgTkzVFIp7EKiMUPgii_AeCSLmrEla5h92sjdmi1Ki6H8V3qOQJ962FXtp5IUPKhIMtDvv8gJSMUeHWOXbhuK21K9PfeHmcf-1-Zpy7sWFV2FP9qVAn5jO9Wm0">https://hastebin.com/ukisajojef.sql</a><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I've cherry picked your before mentioned change and the alarm that came from event is now persistent and the exception is gone.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">So it was a bug. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I understand that for doctor datasources I need to have events for raising the alarm and also for clearing it is that correct?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Best Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Paul</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Afek, Ifat (Nokia - IL/Kfar Sava) [<a href="mailto:ifat.afek@nokia.com">mailto:ifat.afek@nokia.com</a>]
<br>
<b>Sent:</b> Wednesday, February 7, 2018 1:24 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [openstack-dev] [vitrage] Vitrage alarm processing behavior<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Hi Paul,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">It sounds like a bug. Alarms created by a datasource are not supposed to be deleted later on. It might be a bug that was fixed in Queens [1].<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I’m not sure which Vitrage version you are actually using. I failed to find a vitrage version 1.3.1. Could it be that you are referring to a version of python-vitrageclient or vitrage-dashboard?
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In any case, if you are using an older version, I suggest that you try to use the fix that I mentioned [1] and see if it helps.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[1] <a href="https://url10.mailanyone.net/v1/?m=1ejNt4-0001fR-4I&i=57e1b682&c=LqJB68i5VuuaUnZ6iOIMHVhcsHMatfhcTwtLpAT-Rn5UZ3qnX4tq4XOTjYR1XqQIDRQGrqGMwZI31cnT-bEHTFX95wRD-iENXse8JBDHIyv8iJUD7RiwDp74HqNHBFZ-BybLQgQ6-sVcf62n2ogMk31b-Sp0xUJZXxH_0q2Iu-4Hodt4gxhKuFMTT2breh42c7OT5kdHzPJThKClzSEBQ2NWkNTCy112gxlapjMCVxSNQ9nsLg4f0XyJaAVUnAHO">
https://review.openstack.org/#/c/524228</a> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Best Regards,<o:p></o:p></p>
<p class="MsoNormal">Ifat.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">Paul Vaduva <</span><a href="mailto:Paul.Vaduva@enea.com">Paul.Vaduva@enea.com</a><span style="color:black">><br>
<b>Reply-To: </b>"OpenStack Development Mailing List (not for usage questions)" <</span><a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><span style="color:black">><br>
<b>Date: </b>Wednesday, 7 February 2018 at 11:58<br>
<b>To: </b>"</span><a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><span style="color:black">" <</span><a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><span style="color:black">><br>
<b>Subject: </b>[openstack-dev] [vitrage] Vitrage alarm processing behavior</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Times New Roman""> </span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">Hi Vitrage developers, <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">I have a question about vitrage innerworkings, I ported doctor datasource from master branch to an earlier version of vitrage (1.3.1).<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">I noticed some behavior I am wondering if it's ok or it is bug of some sort.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">Here it is:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">1. I am sending some event for rasing an alarm to doctor datasource of vitrage.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">2. I am receiving the event hence the alarm is displayed on vitrage dashboard attached to the affected resource (as expected)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">3. If I have configured snapshot_interval=10 in /etc/vitrage/vitrage.conf The alarm disapears after a while<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">fragment from /etc/vitrage/vitrage.conf<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">***************<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">[datasources]<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">types = nova.host,nova.instance,nova.zone,cinder.volume,neutron.network,neutron.port,doctor<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">snapshots_interval=10<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">***************<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">On the other hand if I comment it out the alarm persists
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">**************<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">[datasources]<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">types = nova.host,nova.instance,nova.zone,cinder.volume,neutron.network,neutron.port,doctor<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">#snapshots_interval=10<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">**************<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">I am interested if this behavior is correct or is this a bug.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">My intention is to create some sort of hybrid datasource starting from the doctor one, that receives events for raising alarms like compute.host.down
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">but uses polling to clear them.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">Best Regards,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">Paul Vaduva<o:p></o:p></p>
</div>
</body>
</html>