We are excited to announce the release of: vitrage 2.0.0: The OpenStack RCA Service This release is part of the queens release series. The source is available from: https://git.openstack.org/cgit/openstack/vitrage Download the package from: https://tarballs.openstack.org/vitrage/ For more details, please see below. 2.0.0 ^^^^^ Prelude ******* Vitrage Queens release contains many new features and bug fixes. * Major changes were made in Vitrage templates. Version 2 was introduced, and it includes features like: * templates that contain only topology definitions * regular expressions * functions (get_attr) The templates are now stored in a database, and there is a new API for adding and deleting them. * A support was added for webhook registration on Vitrage alarms. The registered webhooks will be notified on every alarm state change. * There are significant performance enhancements, mostly around parallel evaluation of the Vitrage templates. New Features ************ * Created a new API to query alarm counts, with an optional parameter to query for all_tenants. The path for the api is "/v1/alarm/count/". * Support Aodh Gnocchi threshold alarm. * Added the option to import external "definition only" templates (containing only entities and relationships) into a template and use the imported definitions to create scenarios. * Added an Event Persistor service that listens to the RabbitMQ2 (on a different topic) and asynchronously writes the events to a relational database. All events are stored after the filter/enrich phase. * Support querying Vitrage healthcheck status in Vitrage client and displaying it in the console. * Fixed multiple non-working tempest tests and added new ones, increasing the number of running tests. * A "version" field was added to the metadata section of Vitrage templates, to allow future changes that are not backward-compatible. The default version is 1. * Support nested heat stacks in heat datasource. * Parallel evaluation of Vitrage templates. The user can now specify a number of workers to evaluate the templates. Each such worker, holds a clone of the graph and will evaluate a portion of the template scenarios. The number of workers defaults to the number of available cores. * Persisting of the current active actions. Scenarios that execute the same actions are considered overlapping, Vitrage keeps track of these actions, previously in-memory and now it is stored in the DB. This allows for parallel evaluators. * Register default policies in code. Support the community goal for the Queens release. * Refactored the "execute-mistral" action. All input parameters should appear under an "input" section. The change takes effect in template version 2. "execute-mistral" actions from version 1 are automatically converted to the new format. * The "aodh datasource" has been rewritten by using Aodh client, as Ceilometer API is being removed. It also supports "ceilometer datasource" with an older OpenStack version that contains the Ceilometer API. * A new service "SNMP parsing" is added. The service parses alarms reported from SNMP managed systems and sends them to the OpenStack message bus, for further processing by specific alarm datasources. * Integration with Sql Alchemy. Allows for data and state to be kept after restarts. And will also allow a shared data store for multiple processes. * Support functions in Vitrage templates version 2. The first supported function is "get_attr" which allows retrieving attributes from the matched entity in the graph. As the first stage this function is supported only for "execute_mistral" action. * Backend support for alarm show API. Return the alarm properties for a specific alarm. Alarm is fetched according to "vitrage_id" parameter. Path for the api is "/v1/alarm/_id_". * Support "mark_down" action for instances via calling Nova "reset- state" API. * Added a mandatory "type" property to the templates metadata section in version 2. The type Should be one of {standard, definition, equivalence} * Added support for template add and template delete. Templates can now be added/removed by the API while vitrage is running (no restart is required). Templates are stored in the database and remain after restarting vitrage. Adding/removing a template at runtime preforms a live update to the entity graph. * Template fields can now contain regular expressions. * Added support to register webhooks to the database. When these webhooks are added and the webhook notifier option is enabled via the config file, Vitrage will send notifications regarding alarm state changes to the registered webhooks. Changes in vitrage 1.8.0..2.0.0 ------------------------------- 1779d80 read heat templates file from resources dir fc008d6 Fixed the format of Queens prelude a3de888 read static file from resources dir 2a52f51 Updated from global requirements d7a1488 Add template delete and add to api description 56ed1e8 Templates api bug fix 352b7b9 Added a prelude for Vitrage release notes fc11a63 template crud release notes c85fb78 fix log formatting df7a8c7 change the order of installation in gate ccb212f Remove use of unsupported TEMPEST_SERVICES variable 92284a1 Rename 'type' to 'template_type' 7109664 change the kill mode for vitrage-graph service 33a4905 bugfix - subprocesses remain alive after vitrage-graph restart 1f36355 Added tempest tests for v2 template validate 3f7b5a7 Add type property to the template metadata 49bcbb1 allow multiple template deletion 0a7b3a9 suppress ResourceWarning: unclosed socket 9148ec0 template add validate fix backend 150cf5c Updated from global requirements 04adf86 Modify SNMP parsing config file 274c5b7 add support for webhooks 02995dd Add release note for SNMP parsing service bf9ba86 fix policy name for alarm count 812848f add logging 8a35b62 Update the documentation of Vitrage architecture in Queens 4ec7fa8 Remove obsolete tempest-lib 06902fe fixed misspelling of openstack f381a42 support adding and deleting templates without the need to restart vitrage. fc1f0f2 Updated from global requirements 6ad1505 Fixed a typo in the release notes 648a894 bugfix - dict item removed during iteration ede475c Add codes of sending parsed snmp trap to datasource 20600df Support template functions f57d737 graph cloning logic extracted to base class c9db3e0 Updated from global requirements 5b0c432 Remove obsolete code regarding predefined relationship names 9c47a11 Graph Persistor 060bdfd remove V318 hacking a49d23f fix constants iteration e0403bb Cleanup unused function 755a6a2 Update comments for ActiveActionsTracker 042858d Fix error in tests example fe923cf Webhook release notes fe3d05c Add alarm show to vitrage api documentation 26478fe add hacking check for assertEquals(True/False) 65af455 alarm show release notes and logging 387a3d5 add alarm show backend b8cc22c Add parsing of snmp_traps for snmp_parsing service 4576523 Support per-version template loading + change execute_mistral structure a593f3c add database table for templates e5ffee3 revert json field to text in db a4cc107 move test_configuration to functional tests c4f7989 Avoid to overrdie entire policy file ee726d5 dont fail test on rollback just print error 7629494 Add database configuration to unit tests 3e03f50 heat moved to policy in code 9bcb70f Updated from global requirements 9ad7b5f Updated from global requirements 6ee6ac8 TrivialFix: remove redandunt import alias de26e6a Update docs build job for new documentation PTI jobs a6ede3f dont run py27 and py35 in parallel 17a03d1 call super for setUpClass eddcfe8 fix log exception parameters 31ffcb8 use force_delete to delete instances in rollback 2f04427 Updated from global requirements 539042f Revert "Add database configuration to unit tests" 0454e6b fix changing the policy file of heat 90778cf Add snmp_parsing service 6dc1e3a load manager tool fix dbedd6b fix error message when resource not found 0996ffe remove eventlet and monkey patch 01fa841 support json field in db 0748f5f fix log exception b166be3 Add database configuration to unit tests 4d222ed renew the installation document. a6c2621 persistor test fix 4b1f7eb Add py35 jobs to the gate 87d5a2b Update template validation status codes doc fec7f59 Refactor template loading 904ef6e Updated from global requirements 3e81c21 be more verbose when test fails a6008e6 Fix the deprecated usage of "get_transport" 18d8407 remove redundant code 7dc4f24 tempest utility function 72f8164 use messaging instead of multiprocessing queue 561c683 Updated from global requirements 2a27ad3 Remove newline characters and change StandardError to Exception 7a89751 more tempest refactoring fa05c09 don't run dsvm tests for document change 37142fa add releasenotes of support_mark_down_action_for_instance fc9f3ed Added a version to Vitrage templates 90820c2 Release notes - aodh gnocchi threshold alarm 7c5edad Refactor template_content_validator c35c393 consistency should not delete events e5af719 remove neutron.port not related to instances b242333 Added release notes alarm-counts-api blueprint a5f8ee1 Updated from global requirements ca822a9 Release notes for definition templates, template field regex and improved tempest tests 7e76cbb correct the installation procedure 1245df7 add tempest tests 915f97d Apply pep8 check to app.wsgi 8600b13 add note to doc 874c3dc Add releasenote about health check api 286b6a5 Vitrage-dbsync only create non existing tables b07e6af Release notes 6c5b2c7 vitrage persistor service feature note 9e8b16f fix api doc f6c4135 add release notes for nested stacks in heat b2c4491 Added release notes for 'rewrite-aodh-datasource' e2bb7be Added release notes for 'default policies in the code' goal 4bd5028 add default parameters to topology show 3d9ad8f Support different time formats for nagios last_check parameter c708a67 Vitrage Persistor Service b6f0fc9 Tempest tests fix 3bc3706 move the copy of templates to tests 29efbe4 fix the initialization error 885857c new e2e tempest to check actions 4b1ecac support 'mark_down' action for instances 511ec35 Added debug level log messages to Nagios 4f8787d Enhance event api allowing for different doctor alarm types 1c3ed3b add disable log.warn to hacking 8a86ff0 fix the name of 'aodh-api' service c7736ad Updated from global requirements 2a859d6 fic api doc 9ea0438 parallel evaluation fixes 4dd2a0d fix py charm warnings and doc 84deef5 add hacking documentation 19f84d1 fix the exception of nova_notifier log e549583 Removed unnecessary utf-8 encoding 024e9b1 add hacking module b3e72ea xrange not supported in py3 a48dbc9 dont use mutable default parameter d3ae72d Register default policies in code 5d94cc2 Bugfix - Evaluator notifications are not sent to notifier 0b7c37f fix documentation 501c064 Rewrite Aodh datasource 6bdab3e Tempest refactoring 9ecc256 Updated from global requirements c8e4500 enable global index d3c2193 Add make_pickleable_without_end_msg. bb46cd5 support nested stack in heat 99bff12 Set number of evaluator workers to number of CPUs e94e7a0 Support multiprocessing parallel evaluation of Vitrage templates. 386dd79 Zuul: add file extension to playbook path 98d268a Replacing six.iteritems() with .items() c4afc57 Added support and documentation for regular expressions in template entity definitions Implements: blueprint entity-regex-definition 724097c filter out the manager and info section cd72303 Store the Evaluator action tracker cache to db 7891dbc fix for template with definitions taken only from imported definition_template 53fc565 `cardinality_for` should be be chosen from `source`/`target` fe6daf5 dont use raise_for_status 390d99a Updated from global requirements b5f3a6b add templates with illegal yaml to template list f41b74b Move legacy jobs to project 740bbf8 Add exception handling to the DBHook. c73446b Fixed collectd mapper abfa48d add aodh gnocchi threshold alarm 0602dd5 add action_id in scenario repository 1. Identifiable actions allow for fetching specific actions (will be used in a future commit) 2. Solve action tracker bug, it relies on uniqueness of scenario_id instead of action_id bf8a2d4 port driver properties to filter out 335ccab read realm from user info of keycloak 462412c fix tempest cli test d3f7b7d fix the vitrage title be733ea Support the option to include definition template files in Vitrage templates. fd4159b Moving vitrage_id logic to transformer 1ecb33c add title to services 6238601 support regex in collectd config d96aa2c Updated from global requirements 63aa567 Update the documentation link for doc migration cce475a sqlalchemy initial commit 67e347d Fix some reST field lists in docstrings 5090d94 Updated from global requirements 1b4f6d2 add dependency for systemd services fc79650 add some debug information cf966fd Clean up screen and tail_log references 9dc55bf Add default backend for healthcheck 9834acf Remove unnecessary shebang directive from tempest.conf 61fb7b3 Updated from global requirements 34e9e3c misc typo and docstring fix 8f58b2a Fix some reST field lists in docstrings 46f6ca8 fix requirements 15561aa remove duplicate constant ce97f8e py35 doesn't have xrange ba31b77 Updated from global requirements b2adbe1 Add ssl support for keycloak auth middleware b50f2b6 Removes unnecessary utf-8 encoding a9966a5 machine learning rst 0dfee2f Updated from global requirements 741b464 Remove usage of parameter enforce_type dfd9c31 Documentation for template configuration 63bcccc enable listener service to get list of elements in enriched_events 074ea94 Update Vitrage architecture diagram 6ee5738 fix rpc transport 3d63af5 minor changes in gate hook 99cc705 Fix a bug in the profiler serializer. 7cd46bd change the way heat client is created f9636aa Remove rescheduler from snapshot service, use timers and threads. 4ade92d Update profiler configuration with redis backend 43f7ea6 Tempest test for Vitrage integration with Mistral 2b6fc05 make tempest work under py3 ab2cdbe Update reno for stable/pike 17b0322 Implements alarm counts API 2cacde9 Replace six.itervalues with dict.values() Diffstat (except docs and test files) ------------------------------------- .gitignore | 2 + .zuul.yaml | 88 +++ CONTRIBUTING.rst | 4 +- HACKING.rst | 20 +- README.rst | 2 +- devstack/files/tempest/tempest.conf | 1 - devstack/gate_hook.sh | 15 +- devstack/plugin.sh | 79 ++- devstack/post_test_hook.sh | 35 +- devstack/settings | 12 +- .../datasource-snmp-parsing-support.rst | 94 +++ .../images/vitrage_graph_architecture.png | Bin 269090 -> 225491 bytes .../template_validation_status_code.rst | 20 + etc/vitrage/api-paste.ini | 2 + etc/vitrage/policy.json | 15 - .../templates.sample/host_down_scenarios.yaml | 4 + etc/vitrage/vitrage-config-generator.conf | 1 + etc/vitrage/vitrage-policy-generator.conf | 3 + playbooks/legacy/vitrage-dsvm-api-py27/post.yaml | 80 +++ playbooks/legacy/vitrage-dsvm-api-py27/run.yaml | 82 +++ playbooks/legacy/vitrage-dsvm-api-py35/post.yaml | 80 +++ playbooks/legacy/vitrage-dsvm-api-py35/run.yaml | 82 +++ .../legacy/vitrage-dsvm-datasources-py27/post.yaml | 80 +++ .../legacy/vitrage-dsvm-datasources-py27/run.yaml | 82 +++ .../legacy/vitrage-dsvm-datasources-py35/post.yaml | 80 +++ .../legacy/vitrage-dsvm-datasources-py35/run.yaml | 82 +++ .../notes/Vitrage_ID-93737ff7b7ced8e9.yaml | 2 +- .../notes/alarm_count_api-057138c8e0a36443.yaml | 4 + ...aodh_gnocchi_thresh_alarm-a529ec4f50f3cb0c.yaml | 3 + .../definition_templates-6c2c10bb3e6912a1.yaml | 5 + .../notes/event_persistor-1b0b4563cc219915.yaml | 5 + .../notes/health-check-api-19f180a104c7fa29.yaml | 4 + ...ntroduce-template-version-55bca9fc4e7d138f.yaml | 5 + .../notes/nested_stacks-fddbfa1f79b2ef43.yaml | 3 + .../parallel_evaluation-db8660ef3ff5b864.yaml | 7 + ...ist_overlapping_scenarios-d426dffdf3d11094.yaml | 6 + .../notes/policies-in-code-dd3f1490f7f5b3eb.yaml | 4 + .../notes/queens-prelude-a00e64274e8e64be.yaml | 21 + ...or-execute-mistral-action-fc0fac84c07e1784.yaml | 6 + .../rewrite-aodh-datasource-4968c32c1c5b81f1.yaml | 5 + .../notes/snmp_parsing-1a5c4645ce832ee3.yaml | 4 + .../notes/sql_integration-c33342d100492ceb.yaml | 5 + ...upport-template-functions-dcb2d2e1e63e9a5d.yaml | 6 + .../support_alarm_show_api-0b001dd185e2e523.yaml | 5 + ...rt_mark_down_for_instance-179b04c693f17787.yaml | 4 + .../notes/template-type-4587ad6a39130170.yaml | 4 + .../notes/template_crud-91baea2ed68291f0.yaml | 8 + .../notes/template_regex-8cb58f5e4438c11a.yaml | 3 + releasenotes/notes/webhooks-139aeaaab7119302.yaml | 5 + releasenotes/source/conf.py | 1 - releasenotes/source/index.rst | 1 + releasenotes/source/pike.rst | 6 + requirements.txt | 37 +- run_vitrage_tempest.sh | 31 - setup.cfg | 12 + test-requirements.txt | 41 +- tools/load_generator/load_generator.py | 6 +- tox.ini | 15 +- vitrage/api/app.py | 3 +- vitrage/api/app.wsgi | 4 +- vitrage/api/controllers/root.py | 2 +- vitrage/api/controllers/v1/alarm.py | 57 +- vitrage/api/controllers/v1/count.py | 56 ++ vitrage/api/controllers/v1/event.py | 6 +- vitrage/api/controllers/v1/rca.py | 6 +- vitrage/api/controllers/v1/resource.py | 29 +- vitrage/api/controllers/v1/root.py | 2 + vitrage/api/controllers/v1/template.py | 135 ++++- vitrage/api/controllers/v1/topology.py | 15 +- vitrage/api/controllers/v1/webhook.py | 141 +++++ vitrage/api/hooks.py | 17 +- vitrage/api_handler/apis/alarm.py | 60 +- vitrage/api_handler/apis/base.py | 2 - vitrage/api_handler/apis/event.py | 15 +- vitrage/api_handler/apis/rca.py | 14 +- vitrage/api_handler/apis/resource.py | 37 +- vitrage/api_handler/apis/template.py | 182 +++--- vitrage/api_handler/apis/topology.py | 58 +- vitrage/api_handler/apis/webhook.py | 153 +++++ vitrage/api_handler/service.py | 21 +- vitrage/cli/__init__.py | 31 + vitrage/cli/api.py | 3 + vitrage/cli/collector.py | 3 + vitrage/cli/graph.py | 48 +- vitrage/cli/machine_learning.py | 2 + vitrage/cli/notifier.py | 2 + vitrage/cli/persistor.py | 37 ++ vitrage/cli/snmp_parsing.py | 32 + vitrage/cli/storage.py | 23 + vitrage/common/constants.py | 38 +- vitrage/common/exception.py | 4 + vitrage/common/policies/__init__.py | 33 + vitrage/common/policies/alarms.py | 80 +++ vitrage/common/policies/base.py | 14 + vitrage/common/policies/event.py | 34 ++ vitrage/common/policies/rca.py | 45 ++ vitrage/common/policies/resource.py | 58 ++ vitrage/common/policies/template.py | 79 +++ vitrage/common/policies/topology.py | 49 ++ vitrage/common/policies/webhook.py | 82 +++ vitrage/common/utils.py | 29 + vitrage/datasources/aodh/driver.py | 207 ++++--- vitrage/datasources/aodh/properties.py | 13 + vitrage/datasources/aodh/transformer.py | 5 +- vitrage/datasources/ceilometer/__init__.py | 44 ++ vitrage/datasources/ceilometer/driver.py | 385 ++++++++++++ vitrage/datasources/ceilometer/properties.py | 53 ++ vitrage/datasources/ceilometer/transformer.py | 167 +++++ .../collectd/collectd_vitrage/plugin.py | 4 +- .../collectd/collectd_vitrage/vitrageplugin.py | 2 +- vitrage/datasources/collectd/driver.py | 11 +- vitrage/datasources/collectd/mapper.py | 55 ++ vitrage/datasources/collector_notifier.py | 9 +- vitrage/datasources/doctor/driver.py | 5 +- vitrage/datasources/doctor/properties.py | 2 + vitrage/datasources/driver_base.py | 25 +- vitrage/datasources/heat/stack/driver.py | 25 +- vitrage/datasources/listener_service.py | 9 +- vitrage/datasources/nagios/parser.py | 10 + vitrage/datasources/nagios/transformer.py | 2 - vitrage/datasources/neutron/network/driver.py | 5 + vitrage/datasources/neutron/port/driver.py | 9 +- vitrage/datasources/rescheduler.py | 205 ------- vitrage/datasources/services.py | 52 +- vitrage/datasources/transformer_base.py | 52 +- .../datasources/zabbix/auxiliary/zabbix_vitrage.py | 4 +- vitrage/db/__init__.py | 15 - vitrage/entity_graph/__init__.py | 2 + .../consistency/consistency_enforcer.py | 17 +- vitrage/entity_graph/consistency/service.py | 13 +- vitrage/entity_graph/graph_clone/__init__.py | 14 + vitrage/entity_graph/graph_clone/base.py | 126 ++++ vitrage/entity_graph/processor/processor.py | 152 +---- vitrage/entity_graph/processor/processor_utils.py | 10 - vitrage/entity_graph/service.py | 150 +++-- vitrage/entity_graph/transformer_manager.py | 4 - vitrage/entity_graph/vitrage_init.py | 83 ++- vitrage/evaluator/__init__.py | 17 +- vitrage/evaluator/actions/action_executor.py | 17 +- .../actions/evaluator_event_transformer.py | 4 +- vitrage/evaluator/actions/priority_tools.py | 22 +- .../evaluator/actions/recipes/execute_mistral.py | 1 + vitrage/evaluator/base.py | 12 +- vitrage/evaluator/condition.py | 1 - vitrage/evaluator/equivalence_data.py | 63 -- vitrage/evaluator/equivalence_repository.py | 23 +- vitrage/evaluator/evaluator_service.py | 106 ++++ vitrage/evaluator/scenario_evaluator.py | 383 +++++++++--- vitrage/evaluator/scenario_repository.py | 164 +++-- vitrage/evaluator/template_data.py | 361 ++--------- vitrage/evaluator/template_db/__init__.py | 15 + .../evaluator/template_db/template_repository.py | 118 ++++ vitrage/evaluator/template_fields.py | 2 + vitrage/evaluator/template_functions/__init__.py | 15 + .../evaluator/template_functions/v2/__init__.py | 15 + .../evaluator/template_functions/v2/functions.py | 78 +++ vitrage/evaluator/template_loader_service.py | 120 ++++ vitrage/evaluator/template_loading/__init__.py | 14 + .../template_loading/equivalence_loader.py | 63 ++ .../evaluator/template_loading/props_converter.py | 49 ++ .../evaluator/template_loading/scenario_loader.py | 172 ++++++ .../evaluator/template_loading/subgraph_builder.py | 76 +++ .../evaluator/template_loading/template_loader.py | 191 ++++++ vitrage/evaluator/template_loading/v1/__init__.py | 15 + .../evaluator/template_loading/v1/action_loader.py | 37 ++ .../template_loading/v1/execute_mistral_loader.py | 29 + vitrage/evaluator/template_schema_factory.py | 42 ++ vitrage/evaluator/template_schemas.py | 93 +++ vitrage/evaluator/template_validation/__init__.py | 44 ++ vitrage/evaluator/template_validation/base.py | 4 + .../content/add_causal_relationship_validator.py | 73 --- .../evaluator/template_validation/content/base.py | 24 + .../content/execute_mistral_validator.py | 41 -- .../content/mark_down_validator.py | 44 -- .../content/raise_alarm_validator.py | 54 -- .../content/set_state_validator.py | 50 -- .../content/template_content_validator.py | 319 ++-------- .../template_validation/content/v1/__init__.py | 15 + .../v1/add_causal_relationship_validator.py | 73 +++ .../content/v1/definitions_validator.py | 215 +++++++ .../content/v1/execute_mistral_validator.py | 47 ++ .../content/v1/mark_down_validator.py | 44 ++ .../content/v1/metadata_validator.py | 29 + .../content/v1/raise_alarm_validator.py | 54 ++ .../content/v1/scenario_validator.py | 193 ++++++ .../content/v1/set_state_validator.py | 50 ++ .../template_validation/content/v2/__init__.py | 15 + .../content/v2/execute_mistral_validator.py | 58 ++ .../content/v2/metadata_validator.py | 45 ++ .../template_validation/status_messages.py | 26 +- .../template_syntax_validator.py | 142 ++++- vitrage/graph/algo_driver/algorithm.py | 1 - vitrage/graph/algo_driver/networkx_algorithm.py | 13 +- vitrage/graph/driver/elements.py | 3 +- vitrage/graph/driver/graph.py | 5 +- vitrage/graph/driver/networkx_graph.py | 38 +- vitrage/graph/filter.py | 15 +- vitrage/graph/query.py | 6 +- vitrage/graph/utils.py | 24 +- vitrage/hacking/__init__.py | 0 vitrage/hacking/checks.py | 168 ++++++ vitrage/i18n.py | 2 +- vitrage/keystone_client.py | 35 +- vitrage/messaging.py | 21 +- vitrage/middleware/keycloak.py | 93 ++- vitrage/notifier/plugins/aodh/aodh_notifier.py | 8 +- .../notifier/plugins/mistral/mistral_notifier.py | 5 +- vitrage/notifier/plugins/nova/__init__.py | 5 + vitrage/notifier/plugins/nova/nova_notifier.py | 28 +- vitrage/notifier/plugins/snmp/snmp_sender.py | 2 +- vitrage/notifier/plugins/webhook/__init__.py | 27 + vitrage/notifier/plugins/webhook/utils.py | 26 + vitrage/notifier/plugins/webhook/webhook.py | 139 +++++ vitrage/opts.py | 15 +- vitrage/os_clients.py | 34 +- vitrage/persistency/__init__.py | 29 + vitrage/persistency/graph_persistor.py | 59 ++ vitrage/persistency/service.py | 77 +++ vitrage/rpc.py | 25 +- vitrage/service.py | 2 + vitrage/snmp_parsing/__init__.py | 24 + vitrage/snmp_parsing/properties.py | 19 + vitrage/snmp_parsing/service.py | 164 +++++ vitrage/storage/__init__.py | 52 ++ vitrage/storage/base.py | 240 ++++++++ vitrage/storage/impl_sqlalchemy.py | 344 +++++++++++ vitrage/storage/sqlalchemy/__init__.py | 15 + vitrage/storage/sqlalchemy/models.py | 197 ++++++ .../functional/datasources/ceilometer/__init__.py | 15 + .../datasources/ceilometer/test_ceilometer.py | 94 +++ .../datasources/cinder/test_cinder_volume.py | 2 +- .../datasources/collectd/test_collectd.py | 18 +- .../functional/datasources/heat/test_heat_stack.py | 2 +- .../datasources/listener_service/__init__.py | 15 + .../listener_service/test_listener_service.py | 76 +++ .../functional/datasources/nagios/test_nagios.py | 2 +- .../datasources/nova/test_nova_datasources.py | 2 +- .../static_physical/test_static_physical.py | 2 +- .../entity_graph/consistency/test_consistency.py | 33 +- .../entity_graph/graph_persistor/__init__.py | 15 + .../graph_persistor/test_graph_persistor.py | 94 +++ .../entity_graph/processor/test_processor.py | 2 +- .../states/test_datasource_info_mapper.py | 3 +- .../functional/evaluator/test_action_executor.py | 92 ++- .../evaluator/test_scenario_evaluator.py | 90 +-- .../driver/driver_stack_snapshot_dynamic.json | 2 +- .../driver/driver_zabbix_snapshot_dynamic.json | 3 +- .../mock_configurations/edges/attached.json | 5 + .../mock_configurations/edges/contains.json | 5 + .../resources/mock_configurations/edges/on.json | 5 + .../vertices/cinder.volume.json | 19 + .../vertices/neutron.network.json | 15 + .../mock_configurations/vertices/neutron.port.json | 18 + .../mock_configurations/vertices/nova.host.json | 11 + .../vertices/nova.instance.json | 17 + .../mock_configurations/vertices/nova.zone.json | 14 + .../vertices/openstack-cluster.json | 13 + .../vertices/tripleo.controller.json | 13 + .../vertices/vitrage.alarm.json | 18 + .../mock_configurations/vertices/zabbix.json | 18 + .../resources/snmp_parsing/snmp_parsing_conf.yaml | 3 + .../definition_templates/basic_def_template.yaml | 22 + .../definition_templates/large_def_template.yaml | 43 ++ .../definition_templates/single_entity.yaml | 10 + .../definition_templates/with_include.yaml | 22 + .../definition_templates/with_scenarios.yaml | 30 + .../templates/basic_with_include.yaml | 26 + .../basic_with_include_that_doesnt_exist.yaml | 32 + .../templates/basic_with_two_includes.yaml | 34 ++ .../templates/include_with_empty_name.yaml | 25 + .../templates/no_definitions_only_include.yaml | 15 + .../only_using_def_template_definitions.yaml | 36 ++ .../with_conflicting_include_entities.yaml | 16 + .../resources/templates/general/bad_yaml.yaml | 24 + ...t_high_cpu_load_to_instance_cpu_suboptimal.yaml | 4 +- .../resources/templates/regex/basic_regex.yaml | 32 + .../templates/regex/basic_regex_for_fail.yaml | 32 + .../resources/templates/regex/faulty_regex.yaml | 32 + .../templates/regex/regex_for_exact_match.yaml | 32 + .../templates/version/invalid_version.yaml | 30 + .../resources/templates/version/no_version.yaml | 29 + .../templates/version/v1/v1_execute_mistral.yaml | 29 + .../resources/templates/version/v1/version1.yaml | 30 + .../templates/version/v2/v2_execute_mistral.yaml | 31 + .../templates/version/v2/v2_standard.yaml | 32 + .../templates/version/v2/v2_with_func.yaml | 31 + .../datasources/aodh/aodh_transformer_base_test.py | 230 +++---- .../unit/datasources/aodh/test_aodh_driver.py | 597 +++++++++++------- .../unit/datasources/aodh/test_aodh_transformer.py | 315 +++++----- .../ceilometer/ceilometer_transformer_base_test.py | 119 ++++ .../unit/datasources/ceilometer/mock_driver.py | 27 + .../ceilometer/test_ceilometer_driver.py | 390 ++++++++++++ .../ceilometer/test_ceilometer_transformer.py | 165 +++++ .../cinder/test_cinder_volume_transformer.py | 14 +- .../datasources/collectd/test_collectd_driver.py | 1 + .../collectd/test_collectd_transformer.py | 17 +- .../consistency/test_consistency_transformer.py | 1 + .../datasources/doctor/test_doctor_transformer.py | 17 +- .../heat/test_heat_stack_transformer.py | 13 +- .../unit/datasources/nagios/test_nagios_config.py | 1 + .../unit/datasources/nagios/test_nagios_driver.py | 1 + .../datasources/nagios/test_nagios_transformer.py | 40 +- .../datasources/nova/test_nova_host_transformer.py | 21 +- .../nova/test_nova_instance_transformer.py | 30 +- .../datasources/nova/test_nova_zone_transformer.py | 16 +- .../unit/datasources/static/test_static_driver.py | 1 + .../datasources/static/test_static_transformer.py | 6 +- .../static_physical/test_static_physical_driver.py | 1 + .../test_static_physical_transformer.py | 14 +- .../datasources/test_datasource_update_method.py | 1 + .../unit/datasources/test_rescheduler_functions.py | 212 ------- .../zabbix/test_zabbix_configuration.py | 1 + .../unit/datasources/zabbix/test_zabbix_driver.py | 3 +- .../datasources/zabbix/test_zabbix_transformer.py | 20 +- .../unit/entity_graph/processor/test_processor.py | 2 +- .../unit/entity_graph/test_processor_service.py | 72 +++ .../unit/entity_graph/test_transformer_manager.py | 1 + .../recipes/test_add_causal_relationship_recipe.py | 7 +- .../unit/evaluator/recipes/test_execute_mistral.py | 5 +- .../unit/evaluator/recipes/test_raise_alarm.py | 7 +- .../evaluator/recipes/test_set_state_recipe.py | 4 +- .../unit/evaluator/template_functions/__init__.py | 15 + .../template_functions/test_template_functions.py | 59 ++ .../unit/evaluator/template_loading/__init__.py | 15 + .../unit/evaluator/template_loading/v2/__init__.py | 15 + .../template_loading/v2/test_template_loader.py | 51 ++ .../evaluator/template_validation/content/base.py | 7 + .../content/base_test_execute_mistral_validator.py | 137 +++++ .../test_add_causal_relationship_validator.py | 111 ---- .../content/test_execute_mistral_validator.py | 99 --- .../content/test_mark_down_validator.py | 61 -- .../content/test_raise_alarm_validator.py | 95 --- .../content/test_set_state_validator.py | 83 --- .../content/test_template_content_validator.py | 115 +++- .../template_validation/content/v1/__init__.py | 15 + .../v1/test_add_causal_relationship_validator.py | 111 ++++ .../content/v1/test_execute_mistral_validator.py | 93 +++ .../content/v1/test_mark_down_validator.py | 61 ++ .../content/v1/test_metadata_validator.py | 62 ++ .../content/v1/test_raise_alarm_validator.py | 95 +++ .../content/v1/test_set_state_validator.py | 83 +++ .../template_validation/content/v2/__init__.py | 15 + .../content/v2/test_execute_mistral_validator.py | 115 ++++ .../content/v2/test_metadata_validator.py | 57 ++ .../test_def_template_syntax_validator.py | 75 +++ .../test_template_syntax_validator.py | 62 +- .../unit/evaluator/test_equivalence_loader.py | 55 ++ .../unit/evaluator/test_equivalence_repository.py | 16 +- .../unit/evaluator/test_scenario_evaluator.py | 48 ++ .../unit/evaluator/test_scenario_repository.py | 118 +++- .../test_jaccard_correlation.py | 4 +- .../notifier/snmp_notifier/test_snmp_notifier.py | 2 + .../test_snmp_sender_with_severity_map.py | 2 + .../test_snmp_sender_without_severity_map.py | 2 + vitrage/utils/datetime.py | 5 +- vitrage/utils/evaluator.py | 24 + .../templates/api/basic_def_template.yaml | 22 + .../templates/api/basic_equivalence_templates.yaml | 21 + .../templates/api/e2e_test_basic_actions.yaml | 113 ++++ .../api/e2e_test_overlapping_actions.yaml | 131 ++++ .../e2e_test_template_actions.yaml | 66 ++ .../e2e_test_template_actions_2.yaml | 66 ++ .../resources/templates/api/e2e_webhooks.yaml | 29 + .../templates/api/equivalence_template.yaml | 12 + .../api/host_high_memory_usage_scenarios.yaml | 73 +++ .../templates/api/v1_execute_mistral.yaml | 29 + .../templates/api/v2_definition_template.yaml | 30 + .../templates/api/v2_equivalence_template.yaml | 14 + .../templates/api/v2_execute_mistral.yaml | 49 ++ .../templates/api/v2_no_type_template.yaml | 21 + 480 files changed, 19819 insertions(+), 5681 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 75e0ede..6c7857e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7 +7,3 @@ Babel!=2.4.0,>=2.3.4 # BSD -lxml!=3.7.0,>=2.3 # BSD +lxml!=3.7.0,>=3.4.1 # BSD +PyMySQL>=0.7.6 # MIT License +aodhclient>=0.9.0 # Apache-2.0 @@ -9 +11 @@ python-ceilometerclient>=2.5.0 # Apache-2.0 -python-cinderclient>=3.1.0 # Apache-2.0 +python-cinderclient>=3.3.0 # Apache-2.0 @@ -13,2 +15,2 @@ python-neutronclient>=6.3.0 # Apache-2.0 -python-novaclient>=9.0.0 # Apache-2.0 -python-heatclient>=1.6.1 # Apache-2.0 +python-novaclient>=9.1.0 # Apache-2.0 +python-heatclient>=1.10.0 # Apache-2.0 @@ -16,7 +18,11 @@ pyzabbix>=0.7.4 # LGPL -networkx>=1.10 # BSD -oslo.config!=4.3.0,!=4.4.0,>=4.0.0 # Apache-2.0 -oslo.messaging!=5.25.0,>=5.24.2 # Apache-2.0 -oslo.log>=3.22.0 # Apache-2.0 -oslo.policy>=1.23.0 # Apache-2.0 -oslo.service>=1.10.0 # Apache-2.0 -oslo.i18n!=3.15.2,>=2.1.0 # Apache-2.0 +networkx<2.0,>=1.10 # BSD +oslo.config>=5.1.0 # Apache-2.0 +oslo.context>=2.19.2 # Apache-2.0 +oslo.db>=4.27.0 # Apache-2.0 +oslo.messaging>=5.29.0 # Apache-2.0 +oslo.middleware>=3.31.0 # Apache-2.0 +oslo.serialization!=2.19.1,>=2.18.0 # Apache-2.0 +oslo.log>=3.36.0 # Apache-2.0 +oslo.policy>=1.30.0 # Apache-2.0 +oslo.service!=1.28.1,>=1.24.0 # Apache-2.0 +oslo.i18n>=3.15.3 # Apache-2.0 @@ -26 +32 @@ Werkzeug>=0.7 # BSD License -keystonemiddleware>=4.12.0 # Apache-2.0 +keystonemiddleware>=4.17.0 # Apache-2.0 @@ -28,0 +35 @@ voluptuous>=0.8.9 # BSD License +SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT @@ -32,0 +40,6 @@ osprofiler>=1.4.0 # Apache-2.0 +keystoneauth1>=3.3.0 # Apache-2.0 +PyYAML>=3.10 # MIT +requests>=2.14.2 # Apache-2.0 +WebOb>=1.7.1 # MIT +eventlet!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2 # MIT +six>=1.10.0 # MIT diff --git a/test-requirements.txt b/test-requirements.txt index eb17adc..5efa442 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -8,2 +8,3 @@ coverage!=4.4,>=4.0 # Apache-2.0 -lxml!=3.7.0,>=2.3 # BSD -networkx>=1.10 # BSD +lxml!=3.7.0,>=3.4.1 # BSD +networkx<2.0,>=1.10 # BSD +PyMySQL>=0.7.6 # MIT License @@ -11 +12 @@ python-ceilometerclient>=2.5.0 # Apache-2.0 -python-cinderclient>=3.1.0 # Apache-2.0 +python-cinderclient>=3.3.0 # Apache-2.0 @@ -13,3 +14,3 @@ python-neutronclient>=6.3.0 # Apache-2.0 -python-novaclient>=9.0.0 # Apache-2.0 -python-heatclient>=1.6.1 # Apache-2.0 -python-subunit>=0.0.18 # Apache-2.0/BSD +python-novaclient>=9.1.0 # Apache-2.0 +python-heatclient>=1.10.0 # Apache-2.0 +python-subunit>=1.0.0 # Apache-2.0/BSD @@ -17,7 +18,8 @@ pyzabbix>=0.7.4 # LGPL -oslo.log>=3.22.0 # Apache-2.0 -oslotest>=1.10.0 # Apache-2.0 -oslo.service>=1.10.0 # Apache-2.0 -oslo.config!=4.3.0,!=4.4.0,>=4.0.0 # Apache-2.0 -oslo.messaging!=5.25.0,>=5.24.2 # Apache-2.0 -oslo.i18n!=3.15.2,>=2.1.0 # Apache-2.0 -oslo.policy>=1.23.0 # Apache-2.0 +oslo.db>=4.27.0 # Apache-2.0 +oslo.log>=3.36.0 # Apache-2.0 +oslotest>=3.2.0 # Apache-2.0 +oslo.service!=1.28.1,>=1.24.0 # Apache-2.0 +oslo.config>=5.1.0 # Apache-2.0 +oslo.messaging>=5.29.0 # Apache-2.0 +oslo.i18n>=3.15.3 # Apache-2.0 +oslo.policy>=1.30.0 # Apache-2.0 @@ -25,2 +27,2 @@ pecan!=1.0.2,!=1.0.3,!=1.0.4,!=1.2,>=1.0.0 # BSD -requests-mock>=1.1 # Apache-2.0 -tempest-lib>=0.14.0 # Apache-2.0 +requests-mock>=1.1.0 # Apache-2.0 +tempest>=17.1.0 # Apache-2.0 @@ -29 +31 @@ testscenarios>=0.4 # Apache-2.0/BSD -testtools>=1.4.0 # MIT +testtools>=2.2.0 # MIT @@ -33 +35 @@ sympy>=0.7.6 # BSD -reno!=2.3.1,>=1.8.0 # Apache-2.0 +reno>=2.5.0 # Apache-2.0 @@ -36,4 +38 @@ osprofiler>=1.4.0 # Apache-2.0 - -# Doc requirements -openstackdocstheme>=1.16.0 # Apache-2.0 -sphinx>=1.6.2 # BSD +SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT