We are gleeful to announce the release of: vitrage 3.0.0: The OpenStack RCA Service This release is part of the rocky release series. The source is available from: https://git.openstack.org/cgit/openstack/vitrage Download the package from: https://tarballs.openstack.org/vitrage/ For more details, please see below. 3.0.0 ^^^^^ New Features ************ * Add a command line tool used as scaffold for creating new datasource. * Added a new "Mock datasource", which can mock an entire graph and allows testing large scale stability as well as performance. * The collector service was changed to run on demand instead of periodically, hence it can now be run in active-active mode. This is as part of a larger design to improve high availability. * Oslo service was replaced by cotyledon, so Vitrage uses real threads and multiprocessing. This change removes unnecessary complications of using eventlets and timers. * Created a dedicated process for the api handler, for better handling api calls under stress. * Support get_changes in the static datasource * The static datasource now supports changes in existing yaml files, and updates the graph accordingly. Bug Fixes ********* * Many bug fixes related to performance and stability. Changes in vitrage 2.1.0..3.0.0 ------------------------------- e2536fa fix tox python3 overrides 0a1017d Added release notes for Rocky first release d5b4daf Support get_changes in the static datasource 75e121f delete heat resource if in state DELETE_COMPLETE 1c30f8a Enable Hacking H104 rule beb5150 mock driver from config 7d484d9 separate api handler to a dedicated process 35b0d5c items in deleted status should be removed by drivers f1795ba Fix 'tox -e releasenotes' startup 62bd81c Remove duplication from test-requirements.txt eca424b Ignore the 'undefined datasource' severity 047ab2b bugfix - datasource driver single instance 1262783 rpc server executor reverted to blocking f77c72b bugfix - notifier changes the vertex as it notifies 6e22e3e new mock datasource adds data based on static datasource 9cc98e8 Add release note for datasource scaffold 89a0d97 Add tool for datasource scaffold 5455510 create rpc server fix. 91c9662 fix logging and payload parsing dfdece5 make sure graph starts after collector on reboot a182670 stop the vitrage-ml when stopping devstack 6acd8b2 try cotyledon 5c4bd36 Rename Mistral notifier queue d8f1457 rpc collector e3fd88c Update links in README af3ab0f refactoring vitrage processes e3127ce add python version number to db file ebe8a8a Updated from global requirements efe467b add lower-constraints job 683b7a3 Bug fix: filter topology by project_id also when there is a query 7fc2d5e add project id to header in keycloak c90ec5c Updated from global requirements 6bb437c Updated from global requirements 346184a catch uncaught exceptions a043876 Updated from global requirements 7b1a0b6 Improve the consistency between code and document on value aggregation 4fbb152 Replace Nova hosts.list with services.list 0f8029d change kill mode for vitrage-collector 4f00ef8 Updated from global requirements 3b5f1da bugfix for log warnings ae6c873 make tests readable 9d8022e vitrage notifier param change 7ca8df2 Fix the invalid links in docs a228332 undo the removal of persistency tables a6d1d48 move post test script 3239d81 Updated from global requirements 3d37a3e move to stestr 1684ffa Fixed list alarms API example 685accb Updated from global requirements 77df1a9 add process names 352c63a Updated from global requirements 34b747d try to create mistral client without auth_url 86706c7 Updated from global requirements 457c55d adding support for networkx 2.0 0ed2676 remove pre_test 61d3428 tempest cleanup 952c571 graph mock changes 8f8c8ee changes to graph_snapshots table cea206a tempest cleanup bcd9151 remove tempest.conf and other cleanup a510df8 Update reno for stable/queens a3e204b Updated from global requirements Diffstat (except docs and test files) ------------------------------------- .gitignore | 1 + .stestr.conf | 3 + .zuul.yaml | 2 + README.rst | 4 +- devstack/files/tempest/tempest.conf | 82 ----- devstack/gate_hook.sh | 31 +- devstack/plugin.sh | 26 +- devstack/post_test_hook.sh | 45 --- devstack/pre_test_hook.sh | 14 - lower-constraints.txt | 142 ++++++++ playbooks/legacy/vitrage-dsvm-api-py27/run.yaml | 35 +- playbooks/legacy/vitrage-dsvm-api-py35/run.yaml | 35 +- .../legacy/vitrage-dsvm-datasources-py27/run.yaml | 35 +- .../legacy/vitrage-dsvm-datasources-py35/run.yaml | 35 +- .../datasource-scaffold-2f5ee6f0d9f83099.yaml | 4 + .../notes/mock-datasource-1c9b427bc16f4a59.yaml | 4 + .../performance-bug-fixes-a2b5e85ee37bba93.yaml | 3 + ...s-management-improvements-899c011e57002e84.yaml | 10 + ...static-datasource-changes-914f9a16ad7e46ed.yaml | 3 + ...nges-in-static-datasource-02715226f103455d.yaml | 4 + releasenotes/source/conf.py | 3 - releasenotes/source/index.rst | 1 + releasenotes/source/queens.rst | 6 + requirements.txt | 85 ++--- test-requirements.txt | 41 +-- tools/__init__.py | 15 - tools/datasource-scaffold/README | 1 + tools/datasource-scaffold/cookiecutter.json | 3 + tools/datasource-scaffold/requirements.txt | 1 + tools/datasource-scaffold/sample/__init__.py | 48 +++ tools/datasource-scaffold/sample/driver.py | 59 ++++ tools/datasource-scaffold/sample/transformer.py | 69 ++++ .../{{cookiecutter.name}}/__init__.py | 48 +++ .../{{cookiecutter.name}}/driver.py | 59 ++++ .../{{cookiecutter.name}}/transformer.py | 72 ++++ tools/load_generator/__init__.py | 15 - tools/load_generator/load_generator.py | 31 +- tox.ini | 29 +- vitrage/api/controllers/__init__.py | 15 - vitrage/api_handler/__init__.py | 15 - vitrage/api_handler/apis/__init__.py | 15 - vitrage/api_handler/apis/alarm.py | 5 +- vitrage/api_handler/apis/base.py | 28 +- vitrage/api_handler/apis/topology.py | 11 +- vitrage/api_handler/service.py | 75 ---- vitrage/cli/collector.py | 38 +- vitrage/cli/graph.py | 31 +- vitrage/cli/machine_learning.py | 8 +- vitrage/cli/notifier.py | 9 +- vitrage/cli/persistor.py | 8 +- vitrage/cli/snmp_parsing.py | 8 +- vitrage/common/__init__.py | 15 - vitrage/common/utils.py | 8 + vitrage/datasources/alarm_properties.py | 1 - vitrage/datasources/cinder/__init__.py | 15 - .../collectd/collectd_vitrage/__init__.py | 15 - vitrage/datasources/collector_notifier.py | 9 +- vitrage/datasources/doctor/driver.py | 8 +- vitrage/datasources/heat/__init__.py | 15 - vitrage/datasources/heat/stack/driver.py | 5 + vitrage/datasources/launcher.py | 58 ---- vitrage/datasources/listener_service.py | 39 ++- vitrage/datasources/neutron/__init__.py | 15 - vitrage/datasources/nova/__init__.py | 14 - vitrage/datasources/nova/host/driver.py | 12 +- vitrage/datasources/nova/host/transformer.py | 10 +- vitrage/datasources/nova/instance/driver.py | 7 +- vitrage/datasources/rpc_service.py | 110 ++++++ vitrage/datasources/sample | 1 + vitrage/datasources/services.py | 126 ------- vitrage/datasources/static/__init__.py | 2 + vitrage/datasources/static/driver.py | 65 +++- vitrage/datasources/static/transformer.py | 6 +- vitrage/datasources/utils.py | 38 ++ .../datasources/zabbix/auxiliary/zabbix_vitrage.py | 52 ++- .../consistency/consistency_enforcer.py | 9 +- vitrage/entity_graph/consistency/service.py | 58 ---- vitrage/entity_graph/datasource_rpc.py | 68 ++++ vitrage/entity_graph/graph_clone/__init__.py | 14 - vitrage/entity_graph/graph_clone/base.py | 126 ------- vitrage/entity_graph/graph_init.py | 141 ++++++++ vitrage/entity_graph/mappings/__init__.py | 15 - .../mappings/datasource_info_mapper.py | 102 +++--- vitrage/entity_graph/processor/__init__.py | 15 - vitrage/entity_graph/processor/base.py | 4 - vitrage/entity_graph/processor/notifier.py | 15 +- vitrage/entity_graph/processor/processor.py | 27 +- vitrage/entity_graph/processor/processor_utils.py | 8 +- .../entity_graph/processor/transformer_manager.py | 90 +++++ vitrage/entity_graph/scheduler.py | 115 ++++++ vitrage/entity_graph/service.py | 154 --------- vitrage/entity_graph/transformer_manager.py | 90 ----- vitrage/entity_graph/utils.py | 35 -- vitrage/entity_graph/vitrage_init.py | 112 ------ vitrage/entity_graph/workers.py | 385 +++++++++++++++++++++ vitrage/evaluator/actions/__init__.py | 11 +- vitrage/evaluator/actions/notifier.py | 5 +- vitrage/evaluator/actions/recipes/__init__.py | 15 - vitrage/evaluator/evaluator_service.py | 106 ------ vitrage/evaluator/template_db/__init__.py | 15 - vitrage/evaluator/template_functions/__init__.py | 15 - .../evaluator/template_functions/v2/__init__.py | 15 - vitrage/evaluator/template_loader_service.py | 122 ------- vitrage/evaluator/template_loading/__init__.py | 14 - vitrage/evaluator/template_loading/v1/__init__.py | 15 - .../template_validation/content/__init__.py | 15 - .../template_validation/content/v1/__init__.py | 15 - .../template_validation/content/v2/__init__.py | 15 - vitrage/graph/algo_driver/__init__.py | 14 - vitrage/graph/algo_driver/networkx_algorithm.py | 22 +- vitrage/graph/driver/networkx_graph.py | 69 +++- vitrage/machine_learning/plugins/__init__.py | 15 - vitrage/machine_learning/service.py | 15 +- vitrage/messaging.py | 6 +- vitrage/middleware/__init__.py | 15 - vitrage/middleware/keycloak.py | 1 + vitrage/notifier/service.py | 22 +- vitrage/opts.py | 2 + vitrage/os_clients.py | 2 - vitrage/persistency/__init__.py | 5 +- vitrage/persistency/graph_persistor.py | 5 +- vitrage/persistency/service.py | 14 +- vitrage/rpc.py | 13 +- vitrage/snmp_parsing/service.py | 17 +- vitrage/storage/impl_sqlalchemy.py | 9 +- vitrage/storage/sqlalchemy/__init__.py | 15 - vitrage/storage/sqlalchemy/models.py | 24 +- .../functional/datasources/ceilometer/__init__.py | 15 - .../datasources/ceilometer/test_ceilometer.py | 17 +- .../functional/datasources/cinder/__init__.py | 15 - .../datasources/cinder/test_cinder_volume.py | 18 +- .../functional/datasources/collectd/__init__.py | 15 - .../datasources/collectd/test_collectd.py | 25 +- .../functional/datasources/heat/test_heat_stack.py | 22 +- .../datasources/listener_service/__init__.py | 15 - .../listener_service/test_listener_service.py | 4 +- .../functional/datasources/nagios/__init__.py | 15 - .../functional/datasources/nagios/test_nagios.py | 17 +- .../functional/datasources/neutron/__init__.py | 15 - .../datasources/neutron/test_neutron_network.py | 15 - .../datasources/neutron/test_neutron_port.py | 15 - .../datasources/static_physical/__init__.py | 15 - .../static_physical/test_static_physical.py | 17 +- .../entity_graph/consistency/__init__.py | 15 - .../entity_graph/consistency/test_consistency.py | 83 +---- .../entity_graph/graph_persistor/__init__.py | 15 - .../graph_persistor/test_graph_persistor.py | 10 +- .../functional/entity_graph/processor/__init__.py | 15 - .../functional/entity_graph/states/__init__.py | 15 - .../evaluator/test_scenario_evaluator.py | 96 ++--- .../mocks/mock_graph_datasource/transformer.py | 31 ++ .../driver/driver_host_snapshot_dynamic.json | 2 +- .../vertices/cinder.volume.json | 2 +- .../vertices/neutron.network.json | 2 +- .../mock_configurations/vertices/neutron.port.json | 2 +- .../mock_configurations/vertices/nova.host.json | 2 + .../vertices/nova.instance.json | 2 +- .../mock_configurations/vertices/nova.zone.json | 1 + .../vertices/openstack-cluster.json | 1 + .../vertices/tripleo.controller.json | 4 +- .../vertices/vitrage.alarm.json | 1 + .../mock_configurations/vertices/zabbix.json | 4 +- .../added_resources/static.yaml | 48 +++ .../changes_datasources/baseline/static.yaml | 32 ++ .../changed_resources/static.yaml | 32 ++ .../deleted_resources/static.yaml | 18 + .../changes_datasources/mixed_changes/static.yaml | 31 ++ .../unit/datasources/aodh/test_aodh_transformer.py | 5 +- .../ceilometer/test_ceilometer_transformer.py | 5 +- .../cinder/test_cinder_volume_transformer.py | 5 +- .../heat/test_heat_stack_transformer.py | 3 +- .../unit/datasources/nagios/test_nagios_driver.py | 58 ++-- .../datasources/nagios/test_nagios_transformer.py | 3 +- .../unit/datasources/neutron/network/__init__.py | 15 - .../network/test_neutron_network_transformer.py | 15 - .../unit/datasources/neutron/port/__init__.py | 15 - .../neutron/port/test_neutron_port_transformer.py | 15 - .../datasources/nova/test_nova_host_transformer.py | 7 +- .../nova/test_nova_instance_transformer.py | 10 +- .../datasources/nova/test_nova_zone_transformer.py | 3 +- .../unit/datasources/static/test_static_driver.py | 191 +++++++--- .../unit/datasources/static_physical/__init__.py | 15 - .../static_physical/test_static_physical_driver.py | 10 +- .../datasources/test_alarm_transformer_base.py | 4 +- .../datasources/test_datasource_update_method.py | 25 +- .../unit/datasources/zabbix/test_zabbix_driver.py | 48 +-- .../datasources/zabbix/test_zabbix_transformer.py | 3 +- .../entity_graph/processor/test_entity_graph.py | 3 +- .../unit/entity_graph/processor/test_processor.py | 16 +- .../states/test_datasource_info_mapper.py | 141 ++++---- .../unit/entity_graph/test_processor_service.py | 8 +- .../unit/entity_graph/test_transformer_manager.py | 3 +- .../recipes/test_add_causal_relationship_recipe.py | 9 +- .../unit/evaluator/recipes/test_raise_alarm.py | 9 +- .../evaluator/recipes/test_set_state_recipe.py | 12 +- .../unit/evaluator/template_functions/__init__.py | 15 - .../unit/evaluator/template_loading/__init__.py | 15 - .../unit/evaluator/template_loading/v2/__init__.py | 15 - .../unit/evaluator/template_validation/__init__.py | 15 - .../template_validation/content/__init__.py | 15 - .../template_validation/content/v1/__init__.py | 15 - .../template_validation/content/v2/__init__.py | 15 - .../unit/evaluator/test_scenario_repository.py | 30 +- .../jaccard_correlation/__init__.py | 15 - .../test_jaccard_correlation.py | 10 +- .../test_snmp_sender_with_severity_map.py | 7 +- .../test_snmp_sender_without_severity_map.py | 5 +- 265 files changed, 3419 insertions(+), 3917 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index f039cf9..b01eb79 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,12 +5,13 @@ -pbr!=2.1.0,>=2.0.0 # Apache-2.0 -Babel!=2.4.0,>=2.3.4 # BSD -lxml!=3.7.0,>=3.4.1 # BSD -PyMySQL>=0.7.6 # MIT License -aodhclient>=0.9.0 # Apache-2.0 -python-ceilometerclient>=2.5.0 # Apache-2.0 -python-cinderclient>=3.3.0 # Apache-2.0 -python-dateutil>=2.4.2 # BSD -python-keystoneclient>=3.8.0 # Apache-2.0 -python-neutronclient>=6.3.0 # Apache-2.0 -python-novaclient>=9.1.0 # Apache-2.0 -python-heatclient>=1.10.0 # Apache-2.0 +pbr>=3.1.1 # Apache-2.0 +Babel>=2.5.3 # BSD +lxml>=4.1.1 # BSD +PyMySQL>=0.8.0 # MIT License +aodhclient>=1.0.0 # Apache-2.0 +python-ceilometerclient>=2.9.0 # Apache-2.0 +python-cinderclient>=3.5.0 # Apache-2.0 +python-dateutil>=2.7.0 # BSD +python-keystoneclient>=3.15.0 # Apache-2.0 +python-neutronclient>=6.7.0 # Apache-2.0 +python-novaclient>=10.1.0 # Apache-2.0 +python-heatclient>=1.14.0 # Apache-2.0 +python-mistralclient>=3.3.0 # Apache-2.0 @@ -18,30 +19,30 @@ pyzabbix>=0.7.4 # LGPL -networkx<2.0,>=1.10 # BSD -oslo.config>=5.1.0 # Apache-2.0 -oslo.context>=2.19.2 # Apache-2.0 -oslo.db>=4.27.0 # Apache-2.0 -oslo.messaging>=5.29.0 # Apache-2.0 -oslo.middleware>=3.31.0 # Apache-2.0 -oslo.serialization!=2.19.1,>=2.18.0 # Apache-2.0 -oslo.log>=3.36.0 # Apache-2.0 -oslo.policy>=1.30.0 # Apache-2.0 -oslo.service!=1.28.1,>=1.24.0 # Apache-2.0 -oslo.i18n>=3.15.3 # Apache-2.0 -pecan!=1.0.2,!=1.0.3,!=1.0.4,!=1.2,>=1.0.0 # BSD -PasteDeploy>=1.5.0 # MIT -Werkzeug>=0.7 # BSD License -keystonemiddleware>=4.17.0 # Apache-2.0 -stevedore>=1.20.0 # Apache-2.0 -voluptuous>=0.8.9 # BSD License -SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT -sympy>=0.7.6 # BSD -pysnmp>=4.2.3 # BSD -PyJWT>=1.0.1 # MIT -osprofiler>=1.4.0 # Apache-2.0 -keystoneauth1>=3.3.0 # Apache-2.0 -PyYAML>=3.10 # MIT -requests>=2.14.2 # Apache-2.0 -WebOb>=1.7.1 # MIT -eventlet!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2 # MIT -six>=1.10.0 # MIT -debtcollector>=1.2.0 # Apache-2.0 - +networkx>=1.11 # BSD +oslo.config>=5.2.0 # Apache-2.0 +oslo.context>=2.20.0 # Apache-2.0 +oslo.db>=4.35.0 # Apache-2.0 +oslo.messaging>=5.36.0 # Apache-2.0 +oslo.middleware>=3.35.0 # Apache-2.0 +oslo.serialization>=2.25.0 # Apache-2.0 +oslo.log>=3.37.0 # Apache-2.0 +oslo.policy>=1.34.0 # Apache-2.0 +oslo.i18n>=3.20.0 # Apache-2.0 +pecan>=1.2.1 # BSD +PasteDeploy>=1.5.2 # MIT +Werkzeug>=0.14.1 # BSD License +keystonemiddleware>=4.21.0 # Apache-2.0 +stevedore>=1.28.0 # Apache-2.0 +voluptuous>=0.11.1 # BSD License +SQLAlchemy>=1.2.5 # MIT +sympy>=1.1.1 # BSD +pysnmp>=4.4.4 # BSD +PyJWT>=1.6.0 # MIT +osprofiler>=2.0.0 # Apache-2.0 +keystoneauth1>=3.4.0 # Apache-2.0 +PyYAML>=3.12 # MIT +requests>=2.18.4 # Apache-2.0 +WebOb>=1.7.4 # MIT +eventlet!=0.20.1,>=0.20.0 # MIT +six>=1.11.0 # MIT +debtcollector>=1.19.0 # Apache-2.0 +cotyledon>=1.6.8 # Apache-2.0 +futures>=3.0.0;python_version=='2.7' or python_version=='2.6' # BSD diff --git a/test-requirements.txt b/test-requirements.txt index 9bd9443..b2223f2 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -6,32 +6,9 @@ hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 -python-dateutil>=2.4.2 # BSD -coverage!=4.4,>=4.0 # Apache-2.0 -lxml!=3.7.0,>=3.4.1 # BSD -networkx<2.0,>=1.10 # BSD -PyMySQL>=0.7.6 # MIT License -python-ceilometerclient>=2.5.0 # Apache-2.0 -python-cinderclient>=3.3.0 # Apache-2.0 -python-neutronclient>=6.3.0 # Apache-2.0 -python-novaclient>=9.1.0 # Apache-2.0 -python-heatclient>=1.10.0 # Apache-2.0 -python-subunit>=1.0.0 # Apache-2.0/BSD -pyzabbix>=0.7.4 # LGPL -oslo.db>=4.27.0 # Apache-2.0 -oslo.log>=3.36.0 # Apache-2.0 -oslotest>=3.2.0 # Apache-2.0 -oslo.service!=1.28.1,>=1.24.0 # Apache-2.0 -oslo.config>=5.1.0 # Apache-2.0 -oslo.messaging>=5.29.0 # Apache-2.0 -oslo.i18n>=3.15.3 # Apache-2.0 -oslo.policy>=1.30.0 # Apache-2.0 -pecan!=1.0.2,!=1.0.3,!=1.0.4,!=1.2,>=1.0.0 # BSD -requests-mock>=1.1.0 # Apache-2.0 -testrepository>=0.0.18 # Apache-2.0/BSD -testscenarios>=0.4 # Apache-2.0/BSD -testtools>=2.2.0 # MIT -stevedore>=1.20.0 # Apache-2.0 -voluptuous>=0.8.9 # BSD License -sympy>=0.7.6 # BSD -reno>=2.5.0 # Apache-2.0 -pysnmp>=4.2.3 # BSD -osprofiler>=1.4.0 # Apache-2.0 -SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT +coverage>=4.5.1 # Apache-2.0 +python-subunit>=1.2.0 # Apache-2.0/BSD +oslotest>=3.3.0 # Apache-2.0 +requests-mock>=1.4.0 # Apache-2.0 +testrepository>=0.0.20 # Apache-2.0/BSD +testscenarios>=0.5.0 # Apache-2.0/BSD +testtools>=2.3.0 # MIT +stestr>=2.0.0 # Apache-2.0 +reno>=2.7.0 # Apache-2.0