[openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.

Ifat Afek ifatafekn at gmail.com
Thu Nov 15 13:06:25 UTC 2018


On Thu, Nov 15, 2018 at 10:28 AM Won <wjstk16 at gmail.com> wrote:

> Looking at the logs, I see two issues:
>> 1. On ubuntu server, you get a notification about the vm deletion, while
>> on compute1 you don't get it.
>> Please make sure that Nova sends notifications to 'vitrage_notifications'
>> - it should be configured in /etc/nova/nova.conf.
>> 2. Once in 10 minutes (by default) nova.instance datasource queries all
>> instances. The deleted vm is supposed to be deleted in Vitrage at this
>> stage, even if the notification was lost.
>> Please check in your collector log for the a message of "novaclient.v2.client
>> [-] RESP BODY" before and after the deletion, and send me its content.
>
>
>  I attached two log files. I created a VM in computer1 which is a computer
> node and deleted it a few minutes later. Log for 30 minutes from VM
> creation.
> The first is the log of the vitrage-collect that grep instance name.
> The second is the noovaclient.v2.clinet [-] RESP BODY log.
> After I deleted the VM, no log of the instance appeared in the collector
> log no matter how long I waited.
>
> I added the following to Nova.conf on the computer1 node.(attached file
> 'compute_node_local_conf.txt')
> notification_topics = notifications,vitrage_notifications
> notification_driver = messagingv2
> vif_plugging_timeout = 300
> notify_on_state_change = vm_and_task_state
> instance_usage_audit_period = hour
> instance_usage_audit = True
>

Hi,

>From the collector log RESP BODY messages I understand that in the
beginning there were the following servers:
compute1: deltesting
ubuntu: Apigateway, KubeMaster and others

After ~20minutes, there was only Apigateway. Does it make sense? did you
delete the instances on ubuntu, in addition to deltesting?
In any case, I would expect to see the instances deleted from the graph at
this stage, since they were not returned by get_all.
Can you please send me the log of vitrage-graph at the same time (Nov 15,
16:35-17:10)?

There is still the question of why we don't see a notification from Nova,
but let's try to solve the issues one by one.

Thanks,
Ifat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20181115/94fc3d8b/attachment.html>


More information about the OpenStack-dev mailing list