[openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Afek, Ifat (Nokia - IL/Kfar Sava) ifat.afek at nokia.com
Sun Jul 16 09:28:55 UTC 2017


Hi Volodymyr,

According to the vitrage-collector.log, when the alarm is cleared it has a different message:

Raise alarm:
{'vitrage_datasource_action': 'update', 'resource_name': u'qvo818dd156-be', u'severity': u'WARNING', u'plugin': u'ovs_events', 'vitrage_entity_type': 'collectd', u'id': u'd211725834f26fa268016d8b23adf7d7', 'vitrage_sample_date': '2017-07-14 07:31:21.405670+00:00', u'host': u'silpixa00399503', u'time': 1500017481.363748, u'collectd_type': u'gauge', u'plugin_instance': u'qvo818dd156-be', u'type_instance': u'link_status', 'vitrage_event_type': u'collectd.alarm.warning', u'message': u'link state of "qvo818dd156-be" interface has been changed to "DOWN"', 'resource_type': u'neutron.port'}

Clear alarm:
{'vitrage_datasource_action': 'update', 'resource_name': u'qvo818dd156-be', u'severity': u'OK', u'plugin': u'ovs_events', 'vitrage_entity_type': 'collectd', u'id': u'd211725834f26fa268016d8b23adf7d7', 'vitrage_sample_date': '2017-07-14 07:31:35.851112+00:00', u'host': u'silpixa00399503', u'time': 1500017495.841522, u'collectd_type': u'gauge', u'plugin_instance': u'qvo818dd156-be', u'type_instance': u'link_status', 'vitrage_event_type': u'collectd.alarm.ok', u'message': u'link state of "qvo818dd156-be" interface has been changed to "UP"', 'resource_type': u'neutron.port'}

The ‘message’ is converted to the name of the alarm, which is considered part of its unique key. If the message is changed from “DOWN” to “UP”, we don’t recognize that it’s the same alarm.
Any idea how this can be solved? Can you modify the message so it will be the same in both cases? Or is there another field that can uniquely identify the alarm?

Thanks,
Ifat.


From: "Mytnyk, VolodymyrX" <volodymyrx.mytnyk at intel.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
Date: Friday, 14 July 2017 at 10:56
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
Cc: "Tahhan, Maryam" <maryam.tahhan at intel.com>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Ifat,

                Thank you for fixing the issue. The patch works and I’m able to map the alarm to port now. Also, as a workaround, I was able to fix/resolve the issue by creating the static datasource (attached static_port.yaml) and disabling the neutron port datasource in the vitrage.conf.

Another issue that I still observe is the deleting of the alarm from the graph when OK collectd notification is sent (port is becomes up). Currently, it is not removed from the entity graph. Is it an issue in the Vitrage too? Attaching all logs (collected using the fix provided by you).

The 3rd issue is the Vitrage-Mistral integration, but I will describe this as a separate mail thread.

Thanks and Regards,
Volodymyr

From: Afek, Ifat (Nokia - IL/Kfar Sava) [mailto:ifat.afek at nokia.com]
Sent: Thursday, July 13, 2017 5:47 PM
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>
Cc: Tahhan, Maryam <maryam.tahhan at intel.com>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Volodymyr,

I believe that this change[1] will fix your problem.

[1] https://review.openstack.org/#/c/482212/

Best Regards,
Ifat.

From: "Mytnyk, VolodymyrX" <volodymyrx.mytnyk at intel.com<mailto:volodymyrx.mytnyk at intel.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Tuesday, 11 July 2017 at 12:48
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: "Tahhan, Maryam" <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Ifat,

Thank you for investigating the issue.

The port name is unique on the graph.  The ovs port name in collectd ovs_events plugin is identified by the ‘plugin_instance’ notification field.

Thanks and Regards,
Volodymyr

From: Afek, Ifat (Nokia - IL/Kfar Sava) [mailto:ifat.afek at nokia.com]
Sent: Tuesday, July 11, 2017 12:00 PM
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: Tahhan, Maryam <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Volodymyr,

I’m working on this issue.
One question: is the port name, as defined by ‘plugin_instance’, supposed to be unique in the graph? If not, then how do you uniquely identify the port (in collectd)?

Thanks,
Ifat.

From: "Mytnyk, VolodymyrX" <volodymyrx.mytnyk at intel.com<mailto:volodymyrx.mytnyk at intel.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Friday, 7 July 2017 at 13:27
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: "Tahhan, Maryam" <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Ifat,

I’ve tested the template file modified by you with enabled debug for the Vitrage graph. See all Vitrage logs in the attachments.

Thank you!

Best Regards,
Volodymyr

From: Afek, Ifat (Nokia - IL/Kfar Sava) [mailto:ifat.afek at nokia.com]
Sent: Friday, July 7, 2017 12:42 PM
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: Tahhan, Maryam <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Volodymyr,

Can you please enable debug information in vitrage.conf, restart vitrage-graph, and send me the vitrage-graph.log file (in the time where the alarm is raised)? I’ll try to understand why the alarm is not connected to the port. The definitions in collectd_conf.yaml seem correct.

I did find some issues with the template file – in the alarm definition, you specified the name of the resource instead of the name/rawtext of the alarm. Also, the name of the port was missing in the port definition. See the attached template (which I haven’t checked, but I believe should work). In any case, this will not fix the problem with the alarm being connected to the resource; it is relevant only for the next phase after we fix the first problem.

Best Regards,
Ifat.

From: "Mytnyk, VolodymyrX" <volodymyrx.mytnyk at intel.com<mailto:volodymyrx.mytnyk at intel.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Friday, 7 July 2017 at 10:35
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: "Tahhan, Maryam" <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Ifat,

                Sorry, I forgot to attach the topology dump. Attaching it now.

Also, I’ve checked the topology, and looks like there is no relationship between neutron port and the alarm for some reason.

Thanks and Regards,
Volodymyr

From: Afek, Ifat (Nokia - IL/Kfar Sava) [mailto:ifat.afek at nokia.com]
Sent: Friday, July 7, 2017 12:15 AM
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: Tahhan, Maryam <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Volodymyr,

Seems like the problem is that the alarm does not get connected to the port. In your collectd_conf.yaml, you should write:

collectd:
- collectd_host: silpixa00399503/ovs_events/qvo818dd156-be   (collectd resource name)
   type: neutron.port
   name: qvo818dd156      (openstack neutron port name)

By doing this, you cause any Collectd alarm that is raised on the Collectd source named silpixa00399503/ovs_events/qvo818dd156-be to be connected in Vitrage to a resource of type neutron.port with name qvo818dd156.

Try to look in the output of ‘vitrage topology show’ (you did not attach it to the mail) and see the exact details of the port.

Let me know if it helped,
Ifat.

From: "Mytnyk, VolodymyrX" <volodymyrx.mytnyk at intel.com<mailto:volodymyrx.mytnyk at intel.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Thursday, 6 July 2017 at 23:59
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: "Tahhan, Maryam" <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Ifat,

                Thank you for your help. See my response on your questions below:


·         You see the neutron port in your entity graph, and it is connected to the VM
V:  yes, it’s correct.


·         You see the alarm on the alarms view, and its resources is the neutron port
V: I see the alarm, sent by collectd , on the alarm page (or vitrage alarm list).  Not sure how to check the alarm resources?


·         You see the alarm in the entity graph, connected to the neutron port
V: The Alarm is show on the entity graph, but it’s NOT connected to the neutron port.


·         You are asking why the neutron port in the entity graph is green and not red?
V: The neutron port doesn’t change the color at all.

I’ve validated and tried the suggested template but unfortunately it doesn’t help. I still see the alarm on the graph but it is not connected to neutron port. Also, the neutron port desn’t change the state. Please note, that I’m using master branch for the Vitrage in my environment.
Also, attaching the topology dump with raised alarm.

Used configurations :
====================

collectd:
- collectd_host: silpixa00399503/ovs_events/qvo818dd156-be
   type: neutron.port
   name: qvo818dd156                  # What name we should use here in case of neutron.port?

metadata:
name: ovs_interface_down
description: ovs interface down
definitions:
entities:
  - entity:
     category: ALARM
     type: collectd
     name: qvo818dd156
     template_id: collectd_alarm
  - entity:
     category: RESOURCE
     type: neutron.port
     template_id: port
relationships:
  - relationship:
     source: collectd_alarm
     relationship_type: on
     target: port
     template_id: collectd_alarm_on_port
scenarios:
- scenario:
    condition: collectd_alarm_on_port
    actions:
     - action:
        action_type: set_state
        action_target:
         target: port
        properties:
         state: ERROR

Thanks and Regards,
Volodymyr


From: Afek, Ifat (Nokia - IL/Kfar Sava) [mailto:ifat.afek at nokia.com]
Sent: Thursday, July 6, 2017 10:01 PM
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: Tahhan, Maryam <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: Re: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi Volodymyr,

Let me see if I understood the problem correctly. Are my assumptions right?

·         You see the neutron port in your entity graph, and it is connected to the VM

·         You see the alarm on the alarms view, and its resources is the neutron port

·         You see the alarm in the entity graph, connected to the neutron port

·         You are asking why the neutron port in the entity graph is green and not red?

The template that you used as an example is not what you are asking for. This template states that if there is an alarm on the host (in your case, the alarm is on the neutron port), and the host contains an instance -> raise an alarm on the instance and set its state to suboptimal. The template that (I think) you need is more simple: if there is an alarm on a neutron port, set its state to error.

Below is the template that I think you should use. Note that I haven’t tested it locally (because I don’t have the same environment as yours), so there could be errors. As a first step, make sure you are using the correct collectd alarm name (use ‘name’ and not ‘rawtext’ which is Zabbix-specific), and then run ‘vitrage template validate’ to verify its correctness.


metadata:
 name: ovs_interface_down
 description: ovs interface is down
definitions:
 entities:
  - entity:
     category: ALARM
     type: collectd
     name: <put the collectd alarm name here>
     template_id: collectd_alarm
  - entity:
     category: RESOURCE
     type: neutron.port
     template_id: port
 relationships:
  - relationship:
     source: collectd_alarm
     relationship_type: on
     target: port
     template_id : collectd_alarm_on_port
scenarios:
 - scenario:
    condition: collectd_alarm_on_port
     - action:
        action_type: set_state
        action_target:
         target: port
        properties:
         state: ERROR


Let me know if it helped,
Ifat.

From: "Mytnyk, VolodymyrX" <volodymyrx.mytnyk at intel.com<mailto:volodymyrx.mytnyk at intel.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Thursday, 6 July 2017 at 17:18
To: "openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Cc: "Tahhan, Maryam" <maryam.tahhan at intel.com<mailto:maryam.tahhan at intel.com>>
Subject: [openstack-dev] [Vitrage] Collectd notification isn't shown on the Vitrage Graph

Hi All,

                We are trying to configure the collectd and Vitrage service to raise an alarm in the Vitrage and show the alarm on the graph if a VM interface goes down. The collectd & collectd_vitrage plugin have been configured to send the notification if the OvS interface connected to VM  goes down (using OvS events plugin). In our case, when interface goes down, the collectd event is sent to Vitrage and we’re able to see it on the Alarm page. But we would like to show the alarm on the Entity Graph (highlighting the neutron port) but for some reason it doesn’t work. The https://github.com/openstack/vitrage/blob/master/tools/load_generator/templates/vm_0.yaml template has been used as a base for our scenario (changing `zabbix` datasource to `collectd` in the file). Our collectd datasource configuration (collectd_conf.yaml) looks like this (tried both variants):

collectd:
- collectd_host: silpixa00399503/ovs_events/qvo818dd156-be
   type: neutron.port
   name: qvo818dd156-be

collectd:
- collectd_host: silpixa00399503/ovs_events/qvo818dd156-be
   type: nove.host
   name: silpixa00399503

Our setup configuration looks like this:

|HOST| ---->  |VM| ---> |NEUTRON.PORT|---> |NEUTRON.NETWOR|

Any help on the issue would be much appreciated.

Thanks and Regards,
Volodymyr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170716/fcefb9a8/attachment.html>


More information about the OpenStack-dev mailing list