<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi. I'm sorry for the late reply.<br></div><div dir="ltr"><br></div><div>my prometheus version : 2.3.2 and alertmanager version : 0.15.2 and I attached files.(vitrage collector,graph logs and apache log and prometheus.yml alertmanager.yml alarm rule file etc..)</div><div>I think the problem that resolved alarm does not disappear is the time stamp problem of the alarm.<br></div><div><div><img src="cid:ii_jn09w7l20" alt="alarm list.JPG" width="562" height="99"><br></div></div><div><div><img src="cid:ii_jn0an0ys1" alt="vitrage-entity_graph.jpg" width="562" height="330"></div><div>-gray alarm info</div><div>severity:PAGE</div><div>vitrage id: c6a94386-3879-499e-9da0-2a5b9d3294b8 , e2c5eae9-dba9-4f64-960b-b964f1c01dfe , 3d3c903e-fe09-4a6f-941f-1a2adb09feca , 8c6e7906-9e66-404f-967f-40037a6afc83 , e291662b-115d-42b5-8863-da8243dd06b4 , 8abd2a2f-c830-453c-a9d0-55db2bf72d46</div><div>----------</div><span class="gmail-m_6618873598713275451gmail-key gmail-m_6618873598713275451gmail-pull-left gmail-m_6618873598713275451ellipsis gmail-m_6618873598713275451gmail-keys-font gmail-m_6618873598713275451gmail-font-title gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span><span class="gmail-m_6618873598713275451gmail-key gmail-m_6618873598713275451gmail-pull-left gmail-m_6618873598713275451ellipsis gmail-m_6618873598713275451gmail-keys-font gmail-m_6618873598713275451gmail-font-title gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span><span class="gmail-m_6618873598713275451gmail-key gmail-m_6618873598713275451gmail-pull-left gmail-m_6618873598713275451ellipsis gmail-m_6618873598713275451gmail-keys-font gmail-m_6618873598713275451gmail-font-title gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span><span class="gmail-m_6618873598713275451gmail-key gmail-m_6618873598713275451gmail-pull-left gmail-m_6618873598713275451ellipsis gmail-m_6618873598713275451gmail-keys-font gmail-m_6618873598713275451gmail-font-title gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span></div><div><br></div><div>The alarms marked with the blue circle are already resolved. However, it does not disappear from the entity graph and alarm list.<br></div><div>There were seven more gray alarms at the top screenshot in active alarms like entity graph. It disappeared by deleting gray alarms from the vitrage-alarms table in the DB or changing the end timestamp value to an earlier time than the current time.<br></div><div><br></div><div>At the log, it seems that the first problem is that the timestamp value from the vitrage comes to 2001-01-01, even though the starting value in the Prometheus alarm information has the correct value. </div><div>When the alarm is solved, the end time stamp value is not updated so alarm does not disappear from the alarm list. <br></div><div><br></div><div>The second problem is that even if the time stamp problem is solved, the entity graph problem will not be solved. Gray alarm information is not in the vitage-collector log but in the vitrage graph and apache log.<br></div><div>I want to know how to forcefully delete entity from a vitage graph.<br></div><div><br></div><div><br></div><div>Regarding the multi nodes, I mean, 1 controll node(pc1) & 1 compute node(pc2). So one openstack. <br></div><div><div><img src="cid:ii_jn2wv7eg2" alt="image.png" width="562" height="296"><br></div></div><div>The test VM in the picture is an instance on compute node that has already been deleted. I waited for hours and checked nova.conf but it was not removed.<br></div><div>This was not the occur in the queens version; in the rocky version, multinode environment, there seems to be a bug in VM creation on multi node.<br></div><div>The same situation occurred in multi-node environments that were configured with different PCs.<br></div><div><br></div><div>thanks,</div><div>Won</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">2018년 10월 4일 (목) 오후 10:46, Ifat Afek <<a href="mailto:ifatafekn@gmail.com" target="_blank">ifatafekn@gmail.com</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:12pt;font-family:Gulim"><span style="font-size:11pt;font-family:Calibri">Can you please give us some more details about your scenario with Prometheus? Please try
and give as many details as possible, so we can try to reproduce the bug.</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:12pt;font-family:Gulim"><br></p><p class="MsoNormal" style="margin:0cm 0cm 0.0001pt"><span style="font-family:Calibri;font-size:11pt">What do you mean by “if the alarm is resolved, the alarm manager makes a
silence, or removes the alarm rule from Prometheus”? these are different cases.
None of them works in your environment?</span><br></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:12pt;font-family:Gulim"><span style="font-size:11pt;font-family:Calibri">Which Prometheus and Alertmanager versions are you using?</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:12pt;font-family:Gulim"><span style="font-size:11pt;font-family:Calibri"> </span><span style="font-family:Calibri;font-size:11pt">Please try to change the Vitrage loglevel to DEBUG (set “debug = true” in
/etc/vitrage/vitrage.conf) and send me the Vitrage collector, graph and api
logs. </span></p><div><br></div><div>Regarding the multi nodes, I'm not sure I understand your configuration. Do you mean there is more than one OpenStack and Nova? more than one host? more than one vm? </div><div><br></div><div>Basically, vms are deleted from Vitrage in two cases:</div><div>1. After each periodic call to get_all of nova.instance datasource. By default this is done once in 10 minutes.</div><div>2. Immediately, if you have the following configuration in /etc/nova/nova.conf:</div><div><span style="font-family:"Times New Roman";font-size:12pt">notification_topics = notifications,vitrage_notifications</span><br></div><div><br></div><div>So, please check your nova.conf and also whether the vms are deleted after 10 minutes.</div><div><br></div><div>Thanks,</div><div>Ifat</div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 4, 2018 at 7:12 AM Won <<a href="mailto:wjstk16@gmail.com" target="_blank">wjstk16@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Thank you for your reply Ifat.</div><div><br></div><div>The alertmanager.yml file already contains 'send_resolved:true'. </div><div>However, the alarm does not disappear from the alarm list and the entity graph even if the alarm is resolved, the alarm manager makes a silence, or removes the alarm rule from Prometheus. </div><div>The only way to remove alarms is to manually remove them from the db. Is there any other way to remove the alarm?</div><div>Entities(vm) that run on multi nodes in the rocky version have similar symptoms. There was a symptom that the Entities created on the multi-node would not disappear from the Entity Graph even after deletion. </div><div>Is this a bug in rocky version?</div><div><br></div><div>Best Regards,</div><div>Won</div></div></div><br>
</blockquote></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div></div></div></div></div></div></div></div>