We are stoked to announce the release of: vitrage 5.0.0: The OpenStack RCA Service This release is part of the train release series. The source is available from: https://opendev.org/openstack/vitrage Download the package from: https://tarballs.openstack.org/vitrage/ Please report issues through: https://storyboard.openstack.org/#!/project/openstack/vitrage For more details, please see below. 5.0.0 ^^^^^ New Features * Added a new API to show vitrage status. * Added a new API to list all vitrage template versions supported. * A new "Kapacitor Datasource" was added, to handle alerts coming from Kapacitor. Kapacitor is an alarming engine in the TICK Stack. It is build on an Open Source core, processing metric of host or instance store in InfluxDB to export alerts. * A new "Monasca Datasource" has been introduced to include Monasca alarms in Vitrage Entity Graph. Monasca is Monitoring as a Service solution offering centralized monitoring sink for metrics gathered by Monasca Agents at many infrastructure levels. Moreover it provides alarm management API that enables defining alarms based on collected metrics. This change is the first stage of integration with Monasca. At this point, Monasca entities are extracted using PULL approach, based on periodical snapshot-query to Monasca Alarm API for the current list of alarm entities. In the future, PUSH approach based on Monasca notifications will be implemented. Current implementation requires that the metrics associated with the given alarm contain information about resource type and ID - required for associating alarms with entities in Vitrage Entity Graph. This additional information should be included in the form of metric dimensions, precisely "resource_type" and "resource_id". Dimensions can be defined in Monasca agent configuration. * Added support to overwrite existing template when adding one. * Added support to show and delete template by name. Changes in vitrage 4.3.0..5.0.0 ------------------------------- f11565a3 Use raw strings in all pattern matching strings 1655af02 use the NOVA_NOTIFICATION_FORMAT for notification format 66c608cb Refactor vitrage to use global CONF instead of local one e21043f4 Don't use LIBS_FROM_GIT for vitrage client by default 5e2f6ec9 Make the keycloak middleware thread safe 7c9cc804 update vitrage graph architecture picture d03f350b Add Python 3 Train unit tests c72bf081 Blacklist sphinx 2.1.0 (autodoc bug) 5212c802 Add new api vitrage status 194e547f Bugfix - If graph is empty is_alive returns false. e5998619 Switch python3 versions of test jobs to match Train PTI be44d292 Add new api to list template versions supported 5d00aa0e Add option to overwrite when adding template b6264f71 Update Devstack plugin README 1a4d2bf7 Configure nova to send versioned and unversioned notifications cc889590 Flag to check backend liveliness on api call dfd0cc9d Instantiate logger per process. 6d4d2438 Fix get changes periodic. 57f09941 Support show and delete template by name 6a8b2693 Kapacitor doc have some problem, see in: https://docs.openstack.org/vitrage/latest/contributor/kapacitor-datasource.h... 8daceb81 Add buffer size to uwsgi 0ecc1d62 Added a document with manual tests for Vitrage 8f9bf413 Fix the documentation for Nova versioned notifications 796d4a62 Add Monasca datasource c5b0d78b Make vitrage-api wrap uWSGI bc540942 Add kapacitor datasource 7fdda9f2 Replace git.openstack.org URLs with opendev.org URLs 15c65485 OpenDev Migration Patch 1a789c12 Dropping the py35 testing f296bbb9 Extend default datasources settings 372039a5 Rename cli_type parameter 11131d33 Add project documentation link to README.rst 022af0fc use a rc file for defining tempest version db93737b Bug fix: showing a deleted template. 59c16e8e Updated Stein architecture diagram 259a7c6b fix bug: execute_mistral action always called with the same attribute value 718042c6 Replace openstack.org git:// URLs with https:// bae4fb7b Update master for stable/stein 3a4344de Added a document with entity equivalence use cases Diffstat (except docs and test files) ------------------------------------- .gitreview | 2 +- .zuul.yaml | 48 +- README.rst | 3 +- devstack/README.rst | 12 +- devstack/devstackgaterc | 1 + devstack/gate_hook.sh | 19 +- devstack/gate_hook_mock.sh | 5 +- devstack/override-defaults | 6 + devstack/plugin.sh | 31 +- devstack/settings | 7 +- .../contributor/entity_equivalence_use_cases.rst | 499 ++++++++++++++++ .../images/vitrage_graph_architecture.png | Bin 192537 -> 172861 bytes .../templates/v2_definition_template.yaml | 22 + .../templates/v2_with_invalid_include.yaml | 28 + .../templates/v3_high_mem_consumption.yaml | 31 + .../templates/v3_with_default_params.yaml | 23 + etc/vitrage/datasources_values/kapacitor.yaml | 18 + etc/vitrage/datasources_values/monasca.yaml | 17 + lower-constraints.txt | 4 +- playbooks/legacy/vitrage-dsvm-api-py27/run.yaml | 16 +- .../post.yaml | 0 .../run.yaml | 18 +- .../legacy/vitrage-dsvm-datasources-py27/run.yaml | 16 +- .../post.yaml | 0 .../run.yaml | 18 +- playbooks/legacy/vitrage-dsvm-mock-py27/run.yaml | 16 +- .../post.yaml | 0 .../run.yaml | 18 +- playbooks/legacy/vitrage-grenade/run.yaml | 12 +- .../notes/add_status-c5b13941c3ce978e.yaml | 3 + .../add_template_versions-5476a446705b43d2.yaml | 3 + .../kapacitor_datasource-c0c9563bb52ff1f1.yaml | 6 + .../notes/monasca-datasource-9ca61922ef14c2a8.yaml | 20 + ...rite_when_adding_template-f6795771ae0eff8e.yaml | 3 + ...late_show_and_add_by_name-a73455f462175160.yaml | 3 + releasenotes/source/index.rst | 1 + releasenotes/source/stein.rst | 6 + requirements.txt | 4 +- setup.cfg | 3 +- tools/datasource-scaffold/sample/driver.py | 3 +- tools/datasource-scaffold/sample/transformer.py | 4 +- .../{{cookiecutter.name}}/driver.py | 3 +- .../{{cookiecutter.name}}/transformer.py | 4 +- tools/load_generator/load_generator.py | 15 +- tox.ini | 8 +- vitrage/api/__init__.py | 2 + vitrage/api/app.py | 79 ++- vitrage/api/app.wsgi | 6 +- vitrage/api/controllers/rest.py | 28 +- vitrage/api/controllers/v1/root.py | 2 + vitrage/api/controllers/v1/service.py | 3 +- vitrage/api/controllers/v1/status.py | 38 ++ vitrage/api/controllers/v1/template.py | 117 +++- vitrage/api/controllers/v1/topology.py | 23 +- vitrage/api/hooks.py | 26 +- vitrage/api_handler/apis/base.py | 3 +- vitrage/api_handler/apis/event.py | 5 +- vitrage/api_handler/apis/operational.py | 5 +- vitrage/api_handler/apis/resource.py | 3 - vitrage/api_handler/apis/template.py | 14 + vitrage/api_handler/apis/topology.py | 3 - vitrage/api_handler/apis/webhook.py | 3 +- vitrage/cli/api.py | 6 +- vitrage/cli/graph.py | 14 +- vitrage/cli/machine_learning.py | 6 +- vitrage/cli/notifier.py | 6 +- vitrage/cli/persistor.py | 9 +- vitrage/cli/snmp_parsing.py | 7 +- vitrage/cli/storage.py | 12 +- vitrage/{service.py => common/config.py} | 47 +- vitrage/common/policies/__init__.py | 2 + vitrage/common/policies/status.py | 37 ++ vitrage/coordination/coordination.py | 7 +- vitrage/coordination/service.py | 6 +- vitrage/datasources/alarm_transformer_base.py | 3 - vitrage/datasources/aodh/driver.py | 5 +- vitrage/datasources/aodh/transformer.py | 3 - vitrage/datasources/ceilometer/driver.py | 7 +- vitrage/datasources/ceilometer/transformer.py | 3 - vitrage/datasources/cinder/volume/driver.py | 5 +- vitrage/datasources/cinder/volume/transformer.py | 3 - vitrage/datasources/collectd/driver.py | 11 +- vitrage/datasources/collectd/transformer.py | 3 - vitrage/datasources/consistency/transformer.py | 4 +- vitrage/datasources/doctor/driver.py | 3 +- vitrage/datasources/doctor/transformer.py | 3 - vitrage/datasources/driver_base.py | 3 - vitrage/datasources/heat/stack/driver.py | 15 +- vitrage/datasources/heat/stack/transformer.py | 3 - vitrage/datasources/kapacitor/__init__.py | 42 ++ .../kapacitor/auxiliary/kapacitor_vitrage.py | 109 ++++ vitrage/datasources/kapacitor/config.py | 93 +++ vitrage/datasources/kapacitor/driver.py | 93 +++ vitrage/datasources/kapacitor/properties.py | 32 ++ vitrage/datasources/kapacitor/transformer.py | 107 ++++ vitrage/datasources/kubernetes/driver.py | 13 +- vitrage/datasources/kubernetes/transformer.py | 7 +- vitrage/datasources/monasca/__init__.py | 45 ++ vitrage/datasources/monasca/driver.py | 75 +++ vitrage/datasources/monasca/properties.py | 30 + vitrage/datasources/monasca/transformer.py | 85 +++ vitrage/datasources/nagios/config.py | 6 +- vitrage/datasources/nagios/driver.py | 13 +- vitrage/datasources/nagios/transformer.py | 3 - vitrage/datasources/neutron/base.py | 5 +- vitrage/datasources/neutron/network/transformer.py | 3 - vitrage/datasources/neutron/port/transformer.py | 3 - vitrage/datasources/nova/host/transformer.py | 3 - vitrage/datasources/nova/instance/driver.py | 5 +- vitrage/datasources/nova/instance/transformer.py | 8 +- vitrage/datasources/nova/nova_driver_base.py | 5 +- vitrage/datasources/nova/zone/transformer.py | 3 - vitrage/datasources/prometheus/driver.py | 17 +- vitrage/datasources/prometheus/transformer.py | 3 - vitrage/datasources/resource_transformer_base.py | 4 +- vitrage/datasources/static/driver.py | 7 +- vitrage/datasources/static/transformer.py | 3 - vitrage/datasources/transformer_base.py | 11 +- vitrage/datasources/trove/cluster/driver.py | 4 +- vitrage/datasources/trove/instance/driver.py | 4 +- vitrage/datasources/trove/trove_driver_base.py | 5 +- vitrage/datasources/utils.py | 26 +- vitrage/datasources/zabbix/driver.py | 23 +- vitrage/datasources/zabbix/transformer.py | 3 - vitrage/entity_graph/__init__.py | 6 +- .../consistency/consistency_enforcer.py | 14 +- vitrage/entity_graph/driver_exec.py | 24 +- vitrage/entity_graph/graph_init.py | 29 +- vitrage/entity_graph/graph_persistency.py | 3 +- .../mappings/datasource_info_mapper.py | 13 +- vitrage/entity_graph/processor/notifier.py | 25 +- vitrage/entity_graph/processor/processor.py | 7 +- .../entity_graph/processor/transformer_manager.py | 26 +- vitrage/entity_graph/scheduler.py | 21 +- vitrage/entity_graph/workers.py | 64 +-- vitrage/evaluator/actions/action_executor.py | 4 +- vitrage/evaluator/actions/base.py | 1 + .../actions/evaluator_event_transformer.py | 4 +- vitrage/evaluator/actions/notifier.py | 11 +- vitrage/evaluator/scenario_evaluator.py | 20 +- vitrage/evaluator/scenario_repository.py | 4 +- .../evaluator/template_db/template_repository.py | 12 +- .../template_functions/function_resolver.py | 4 +- .../content/template_content_validator_v3.py | 6 +- vitrage/hacking/checks.py | 4 +- vitrage/keystone_client.py | 21 +- vitrage/machine_learning/plugins/base.py | 3 - .../plugins/jaccard_correlation/alarm_processor.py | 12 +- .../jaccard_correlation/correlation_manager.py | 12 +- vitrage/machine_learning/service.py | 26 +- vitrage/messaging.py | 17 +- vitrage/middleware/keycloak.py | 45 +- vitrage/notifier/plugins/aodh/aodh_notifier.py | 6 +- vitrage/notifier/plugins/base.py | 3 - .../notifier/plugins/mistral/mistral_notifier.py | 7 +- vitrage/notifier/plugins/nova/nova_notifier.py | 6 +- vitrage/notifier/plugins/snmp/base.py | 3 - vitrage/notifier/plugins/snmp/snmp_notifier.py | 8 +- vitrage/notifier/plugins/snmp/snmp_sender.py | 15 +- vitrage/notifier/plugins/webhook/webhook.py | 12 +- vitrage/notifier/plugins/zaqar/zaqar_notifier.py | 12 +- vitrage/notifier/service.py | 29 +- vitrage/opts.py | 7 +- vitrage/os_clients.py | 82 ++- vitrage/persistency/service.py | 18 +- vitrage/snmp_parsing/service.py | 13 +- vitrage/storage/__init__.py | 13 +- vitrage/storage/base.py | 2 +- vitrage/storage/impl_sqlalchemy.py | 19 +- .../functional/api/v1/test_template_versions.py | 49 ++ .../datasources/ceilometer/test_ceilometer.py | 19 +- .../datasources/cinder/test_cinder_volume.py | 20 +- .../datasources/collectd/test_collectd.py | 20 +- .../functional/datasources/heat/test_heat_stack.py | 20 +- .../listener_service/test_listener_service.py | 5 - .../functional/datasources/nagios/test_nagios.py | 20 +- .../datasources/nova/test_nova_datasources.py | 15 +- .../datasources/trove/test_trove_instance.py | 15 +- .../entity_graph/consistency/test_consistency.py | 45 +- .../graph_persistor/test_graph_persistor.py | 22 +- .../entity_graph/processor/test_processor.py | 15 +- .../states/test_datasource_info_mapper.py | 15 +- .../functional/evaluator/test_action_executor.py | 30 +- .../evaluator/test_scenario_evaluator.py | 26 +- .../mocks/mock_graph_datasource/transformer.py | 4 +- .../driver/driver_kapacitor_update_dynamic.json | 12 + .../transformer_kapacitor_update_dynamic.json | 18 + .../unit/datasources/aodh/test_aodh_driver.py | 10 +- .../unit/datasources/aodh/test_aodh_transformer.py | 4 +- .../unit/datasources/ceilometer/mock_driver.py | 3 - .../ceilometer/test_ceilometer_driver.py | 4 +- .../ceilometer/test_ceilometer_transformer.py | 4 +- .../cinder/test_cinder_volume_transformer.py | 4 +- .../datasources/collectd/test_collectd_driver.py | 2 +- .../collectd/test_collectd_transformer.py | 19 +- .../consistency/test_consistency_transformer.py | 2 +- .../unit/datasources/doctor/test_doctor_driver.py | 2 +- .../datasources/doctor/test_doctor_transformer.py | 19 +- .../heat/test_heat_stack_transformer.py | 6 +- .../kapacitor/test_kapacitor_configuration.py | 84 +++ .../datasources/kapacitor/test_kapacitor_driver.py | 113 ++++ .../kapacitor/test_kapacitor_transformer.py | 175 ++++++ .../kubernetes/test_kubernetes_transformer.py | 4 +- .../unit/datasources/nagios/test_nagios_config.py | 13 +- .../unit/datasources/nagios/test_nagios_driver.py | 17 +- .../datasources/nagios/test_nagios_transformer.py | 8 +- .../nova/base_nova_instance_transformer.py | 4 +- ...st_instance_transformer_legacy_notifications.py | 8 +- ...instance_transformer_versioned_notifications.py | 5 - .../datasources/nova/test_nova_host_transformer.py | 4 +- .../datasources/nova/test_nova_instance_driver.py | 21 +- .../datasources/nova/test_nova_zone_transformer.py | 4 +- .../prometheus/test_prometheus_driver.py | 22 +- .../prometheus/test_prometheus_transformer.py | 22 +- .../unit/datasources/static/test_static_driver.py | 6 +- .../datasources/static/test_static_transformer.py | 4 +- .../datasources/test_datasource_update_method.py | 39 +- .../trove/test_trove_cluster_transformer.py | 2 +- .../trove/test_trove_instance_transformer.py | 2 +- .../zabbix/test_zabbix_configuration.py | 11 +- .../unit/datasources/zabbix/test_zabbix_driver.py | 19 +- .../datasources/zabbix/test_zabbix_transformer.py | 8 +- .../entity_graph/processor/test_entity_graph.py | 5 +- .../unit/entity_graph/processor/test_processor.py | 14 +- .../states/test_datasource_info_mapper.py | 94 ++-- .../unit/entity_graph/test_processor_service.py | 2 +- .../unit/entity_graph/test_transformer_manager.py | 39 +- .../template_loading/test_template_loading_v3.py | 9 +- .../test_template_validator_v3.py | 11 +- .../unit/evaluator/test_equivalence_repository.py | 5 +- .../unit/evaluator/test_scenario_repository.py | 91 +-- .../test_jaccard_correlation.py | 46 +- .../test_snmp_sender_with_severity_map.py | 42 +- .../test_snmp_sender_without_severity_map.py | 42 +- 279 files changed, 4860 insertions(+), 1472 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index df41497b..a1de30f3 100644 --- a/requirements.txt +++ b/requirements.txt @@ -17,0 +18 @@ python-openstackclient>=3.12.0 # Apache-2.0 +python-monascaclient>=1.7.1 # Apache-2.0 @@ -22 +23,2 @@ pyzabbix>=0.7.4 # LGPL -networkx>=2.0 # BSD +networkx<2.3,>=2.0;python_version<'3.0' # BSD +networkx>=2.3;python_version>='3.4' # BSD