[openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

Ifat Afek ifatafekn at gmail.com
Thu Nov 1 17:26:00 UTC 2018


Hi,

On Wed, Oct 31, 2018 at 11:00 AM Won <wjstk16 at gmail.com> wrote:

> Hi,
>
>>
>> This is strange. I would expect your original definition to work as well,
>> since the alarm key in Vitrage is defined by a combination of the alert
>> name and the instance. We will check it again.
>> BTW,  we solved a different bug related to Prometheus alarms not being
>> cleared [1]. Could it be related?
>>
>
> Using the original definition, no matter how different the instances are,
> the alarm names are recognized as the same alarm in vitrage.
> And I tried to install the rocky version and the master version on the new
> server and retest but the problem was not solved. The latest bugfix seems
> irrelevant.
>

Ok. We will check this issue. For now your workaround is ok, right?


> Does the wrong timestamp appear if you run 'vitrage alarm list' cli
>> command? please try running 'vitrage alarm list --debug' and send me the
>> output.
>>
>
> I have attached 'vitrage-alarm-list.txt.'
>

I believe that you attached the wrong file. It seems like another log of
vitrage-graph.


>
>
>> Please send me vitrage-collector.log and vitrage-graph.log from the time
>> that the problematic vm was created and deleted. Please also create and
>> delete a vm on your 'ubuntu' server, so I can check the differences in the
>> log.
>>
>
> I have attached 'vitrage_log_on_compute1.zip' and
> 'vitrage_log_on_ubuntu.zip' files.
> When creating a vm on computer1, a vitrage-collect log occurred, but no
> log occurred when it was removed.
>

Looking at the logs, I see two issues:
1. On ubuntu server, you get a notification about the vm deletion, while on
compute1 you don't get it.
Please make sure that Nova sends notifications to 'vitrage_notifications' -
it should be configured in /etc/nova/nova.conf.

2. Once in 10 minutes (by default) nova.instance datasource queries all
instances. The deleted vm is supposed to be deleted in Vitrage at this
stage, even if the notification was lost.
Please check in your collector log for the a message of "novaclient.v2.client
[-] RESP BODY" before and after the deletion, and send me its content.

Br,
Ifat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20181101/39d4404f/attachment.html>


More information about the OpenStack-dev mailing list