<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Ifat,</div><div>I'm sorry for the late reply. </div><div dir="ltr"><div><br></div><div>I solved the problem of not updating the Prometheus alarm.</div><div>Alarms with the same Prometheus alarm name are recognized as the same alarm in vitrage.<br></div><div><br></div><div>------- alert.rules.yml</div><div><div>groups:</div><div>- name: alert.rules</div><div> rules:</div><div> - alert: InstanceDown</div><div> expr: up == 0</div><div> for: 60s</div><div> labels:</div><div> severity: warning</div><div> annotations:</div><div> description: '{{ $labels.instance }} of job {{ $labels.job }} has been down</div><div> for more than 30 seconds.'</div><div> summary: Instance {{ $labels.instance }} down</div></div><div>------</div><div>This is the contents of the alert.rules.yml file before I modify it.<br></div><div>This is a yml file that generates an alarm when the cardvisor stops(instance down). Alarm is triggered depending on which instance is down, but all alarms have the same name as 'instance down'. Vitrage recognizes all of these alarms as the same alarm. Thus, until all 'instance down' alarms were cleared, the 'instance down' alarm was recognized as unresolved and the alarm was not extinguished.<br></div><div><br></div><div><br></div><div>------alert.rules.yml(modified)</div><div><div>groups:</div><div>- name: alert.rules</div><div> rules:</div><div> - alert: InstanceDown on Apigateway</div><div> expr: up{instance="<a href="http://192.168.12.164:31121" target="_blank">192.168.12.164:31121</a>"} == 0</div><div> for: 5s</div><div> labels:</div><div> severity: warning</div><div> annotations:</div><div> description: '{{ $labels.instance }} of job {{ $labels.job }} has been down</div><div> for more than 30 seconds.'</div><div> summary: Instance {{ $labels.instance }} down</div><div><br></div><div> - alert: InstanceDown on Signup</div><div> expr: up{instance="<a href="http://192.168.12.164:31122" target="_blank">192.168.12.164:31122</a>"} == 0</div><div> for: 5s</div><div> labels:</div><div> severity: warning</div><div> annotations:</div><div> description: '{{ $labels.instance }} of job {{ $labels.job }} has been down</div><div> for more than 30 seconds.'</div><div> summary: Instance {{ $labels.instance }} down</div></div><div>.</div><div>.</div><div>.</div><div>---------------</div><div>By modifying the rules as above, the problem has been solved.<br></div><div><br></div><div><div style="color:rgb(80,0,80)"> <br></div><blockquote class="gmail_quote" style="color:rgb(80,0,80);margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div></div><div><span style="color:rgb(34,34,34)">Can you please show me where you saw the 2001 timestamp? I didn't find it in the log.</span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div> </div><div> </div><div><img src="cid:ii_jno98j6l0" alt="image.png" style="margin-right: 0px;" width="1133" height="110"><br></div></div><div>The time stamp is recorded well in log(vitrage-graph,collect etc), but in vitrage-dashboard it is marked 2001-01-01.</div><div>However, it seems that the time stamp is recognized well internally because the alarm can be resolved and is recorded well in log.<br></div><div><br></div><div><br></div><div><div class="gmail-m_-3570743932547074883gmail-gs" style="margin:0px;padding:0px 0px 20px;width:1512px"><div class="gmail-m_-3570743932547074883gmail-"><div id="gmail-m_-3570743932547074883gmail-:pm" class="gmail-m_-3570743932547074883gmail-ii gmail-m_-3570743932547074883gmail-gt gmail-m_-3570743932547074883gmail-adO" style="direction:ltr;margin:8px 0px 0px;padding:0px"><div id="gmail-m_-3570743932547074883gmail-:pl" class="gmail-m_-3570743932547074883gmail-a3s gmail-m_-3570743932547074883gmail-aXjCH" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:1.5;outline:none"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="font-family:Arial,Helvetica,sans-serif;font-size:small;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span class="gmail-m_-3570743932547074883gmail-im" style="color:rgb(80,0,80)"><div style="color:rgb(34,34,34)">Let me make sure I understand the problem. </div><div style="color:rgb(34,34,34)">When you create a new vm in Nova, does it immediately appear in the entity graph?</div><div style="color:rgb(34,34,34)">When you delete a vm, it remains? does it remain in a multi-node environment and deleted in a single node environment? </div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"> </div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><div><img src="cid:ii_jnoa1gmi1" alt="image.png" width="876" height="154" style="margin-right: 0px;"><br></div></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small">Host name ubuntu is my main server. I install openstack all in one in this server and i install compute node in host name compute1.</div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small">When i create a new vm in nova(compute1) it immediately appear in the entity graph. But in does not disappear in the entity graph when i delete the vm. No matter how long i wait, it doesn't disappear.</div><div>Afther i execute 'vitrage-purge-data' command and reboot the Openstack(execute reboot command in openstack server(host name ubuntu)), it disappear. Only execute 'vitrage-purge-data' does not work. It need a reboot to disappear.<br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small">When i create a new vm in nova(ubuntu) there is no problem. </div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div>I implemented the web service of the micro service architecture and applied the RCA. Attached file picture shows the structure of the web service I have implemented. I wonder what data I receive and what can i do when I link vitrage with kubernetes.</div><div>As i know, the vitrage graph does not present information about containers or pods inside the vm. If that is correct, I would like to make the information of the pod level appear on the entity graph.<br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small">I follow (<a href="https://docs.openstack.org/vitrage/latest/contributor/k8s_datasource.html">https://docs.openstack.org/vitrage/latest/contributor/k8s_datasource.html</a>) this step. I attached the vitage.conf file and the kubeconfig file. The contents of the Kubeconconfig file are copied from the contents of the admin.conf file on the master node.</div><div>I want to check my settings are right and connected, but I don't know how. It would be very much appreciated if you let me know how.<br></div><div><br></div><div>Br,</div><div>Won</div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small"><br></div><div class="gmail-m_-3570743932547074883gmail-yj6qo gmail-m_-3570743932547074883gmail-ajU" style="font-family:Arial,Helvetica,sans-serif;font-size:small;outline:none;padding:10px 0px;width:22px;margin:2px 0px 0px"><div id="gmail-m_-3570743932547074883gmail-:q1" class="gmail-m_-3570743932547074883gmail-ajR" style="background-color:rgb(232,234,237);border:none;clear:both;line-height:6px;outline:none;width:24px;border-radius:5.5px"><br></div></div></div></div></div></div><div class="gmail-m_-3570743932547074883gmail-hi" style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium;border-bottom-left-radius:1px;border-bottom-right-radius:1px;padding:0px;width:auto;background:rgb(242,242,242);margin:0px"></div></div></div><br class="gmail-m_-3570743932547074883gmail-Apple-interchange-newline"></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">2018년 10월 10일 (수) 오후 11:52, Ifat Afek <<a href="mailto:ifatafekn@gmail.com" target="_blank">ifatafekn@gmail.com</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Won,<div><br></div><div><div class="gmail_quote"><div dir="ltr">On Wed, Oct 10, 2018 at 11:58 AM Won <<a href="mailto:wjstk16@gmail.com" target="_blank">wjstk16@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><div>my prometheus version : 2.3.2 and alertmanager version : 0.15.2 and I attached files.(vitrage collector,graph logs and apache log and prometheus.yml alertmanager.yml alarm rule file etc..)</div><div>I think the problem that resolved alarm does not disappear is the time stamp problem of the alarm.<br></div><div><div><br></div></div><div><div>-gray alarm info</div><div>severity:PAGE</div><div>vitrage id: c6a94386-3879-499e-9da0-2a5b9d3294b8 , e2c5eae9-dba9-4f64-960b-b964f1c01dfe , 3d3c903e-fe09-4a6f-941f-1a2adb09feca , 8c6e7906-9e66-404f-967f-40037a6afc83 , e291662b-115d-42b5-8863-da8243dd06b4 , 8abd2a2f-c830-453c-a9d0-55db2bf72d46</div><div>----------</div><span class="gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-key gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-pull-left gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451ellipsis gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-keys-font gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-font-title gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span><span class="gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-key gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-pull-left gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451ellipsis gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-keys-font gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-font-title gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span><span class="gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-key gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-pull-left gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451ellipsis gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-keys-font gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-font-title gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span><span class="gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-key gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-pull-left gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451ellipsis gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-keys-font gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-font-title gmail-m_-3570743932547074883m_-1732321782488719539m_-2876726172661411467gmail-m_6618873598713275451gmail-ng-binding" title="Vitrage id"></span></div><div><br></div><div>The alarms marked with the blue circle are already resolved. However, it does not disappear from the entity graph and alarm list.<br></div><div>There were seven more gray alarms at the top screenshot in active alarms like entity graph. It disappeared by deleting gray alarms from the vitrage-alarms table in the DB or changing the end timestamp value to an earlier time than the current time.</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>I checked the files that you sent, and it appears that the connection between Prometheus and Vitrage works well. I see in vitrage-graph log that Prometheus notified both on alert firing and on alert resolved statuses. </div><div>I still don't understand why the alarms were not removed from Vitrage, though. Can you please send me the output of 'vitrage topology show' CLI command? </div><div>Also, did you happen to restart vitrage-graph or vitrage-collector during your tests? </div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div></div><div>At the log, it seems that the first problem is that the timestamp value from the vitrage comes to 2001-01-01, even though the starting value in the Prometheus alarm information has the correct value. </div><div>When the alarm is solved, the end time stamp value is not updated so alarm does not disappear from the alarm list. </div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div> </div><div>Can you please show me where you saw the 2001 timestamp? I didn't find it in the log.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div></div><div>The second problem is that even if the time stamp problem is solved, the entity graph problem will not be solved. Gray alarm information is not in the vitage-collector log but in the vitrage graph and apache log.<br></div><div>I want to know how to forcefully delete entity from a vitage graph.<br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>You shouldn't do it :-) there is no API for deleting entities, and messing with the database may cause unexpected results. </div><div>The only thing that you can safely do is to stop all Vitrage services, execute 'vitrage-purge-data' command, and start the services again. This will cause rebuilding of the entity graph.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Regarding the multi nodes, I mean, 1 controll node(pc1) & 1 compute node(pc2). So one openstack. <br></div><div><div><br></div></div><div>The test VM in the picture is an instance on compute node that has already been deleted. I waited for hours and checked nova.conf but it was not removed.<br></div><div>This was not the occur in the queens version; in the rocky version, multinode environment, there seems to be a bug in VM creation on multi node.<br></div><div>The same situation occurred in multi-node environments that were configured with different PCs.</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>Let me make sure I understand the problem. </div><div>When you create a new vm in Nova, does it immediately appear in the entity graph?</div><div>When you delete a vm, it remains? does it remain in a multi-node environment and deleted in a single node environment? </div><div> </div><div>Br,</div><div>Ifat</div><div><br></div></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div></div></div></div></div></div>