We joyfully announce the release of:
vitrage 3.1.0: The OpenStack RCA Service
This release is part of the rocky stable release series.
The source is available from:
https://git.openstack.org/cgit/openstack/vitrage
Download the package from:
https://tarballs.openstack.org/vitrage/
For more details, please see below.
3.1.0 ^^^^^
Prelude *******
Vitrage Rocky release contains significant infrastructure changes that bring a lot of value to the end user. The main ones are:
* Graph fast-failover and better HA support.
* High-scale support. The graph was tested to work with over 100,000 entities.
* Alarm and RCA history.
In addition, we added Kubernetes and Prometheus datasources.
New Features ************
* The "Alarm and RCA History" feature allows saving and quering historical alarms and exploring their root cause. New set of parameters in alarm list api and a new history api allows users to query the data saved in Vitrage schema in the DB.
* Add support for more aodh alarm types - composite, gnocchi_aggregation_by_metrics_threshold and gnocchi_aggregation_by_resources_threshold.
* High availability of active standby vitrage-graph is better supported. A fast fail-over is implemented by storing all the required in-memory state data in mysql. Vitrage-graph initializes quickly upon failover without requesting any updates.
* Added a new datasource for Kubernetes cluster as a workload on Openstack. We support kubernetes on top of Nova.
* A new "Prometheus Datasource" was added, to handle alerts coming from Prometheus. Prometheus is an open-source systems monitoring and alerting toolkit, with exporters that exports different metrics to Prometheus and Alertmanager that handles alerts sent by Prometheus server.
* Support for graphs with more than 100,000 vertices has been added and tested. See high-scale configuration document.
Known Issues ************
* As part of Rocky fast-failover support, vitrage-graph is now reloaded from the database. This causes an issue with datasources using caches that can become outdated after vitrage-graph restart, or if more than one vitrage-collector is used. Please avoid running multiple vitrage-collector services.
Bug Fixes *********
* Added support for Networkx version 2.1
Changes in vitrage 3.0.0..3.1.0 -------------------------------
493e5e7 Added k8s and Prometheus to Rocky prelude 2eb2ba3 add operational severity to alarms table 1343da2 change history alarms 'state' to inactive ab6393e Bug fix: calculate md5 on non-unicode strings 36aad15 Add vitrage_datasource_name to mock entities 7cf4432 Fixed the release notes for vitrage-collector limitations 9df7b52 Fix Rocky prelude 20aaced add timezone to alarms queried from db cf842b7 Delete outdated entities by the consistency feae014 alarm and RCA history release notes 1149660 alarm and RCA history support d530492 Prelude for Rocky release notes 9115c83 Collector rpc datasource works at 200k entities. 05fdd3c adding prometheus to VITRAGE_DEFAULT_DATASOURCES in devstack 7b8cb91 remove templates dir we don't use it any more 3d6b340 Document the static physical datasource deprecation 8f1d9b2 expose the user info url as a configuration 454698f determine instance id from instance ip in Prometheus datasource 79bf14a change the deprecate message to queens 034a597 Adjust Vitrage api to support Prometheus datasource c2c273d deprecate the static_physical options 71ec06b release note for Prometheus datasource 98d6401 Release notes for kubernetes datasource 3a39fc1 Add release notes for fast failover and networkx 2 09a7f43 remove enable_persistency from configuration. 7803cc5 Add support for more aodh alarm types: - composite - gnocchi_aggregation_by_metrics_threshold - gnocchi_aggregation_by_resources_threshold fb4088c Graph fast fail-over 83a370f support networkx 2.1 d0da90b Switch to stestr d68438a add support for gnocchi client d5c742f remove pycrypto b85fae2 Drop python-cinderclient usage d1196f5 Add upgrade support for Vitrage 4898883 don't mark deleted if alarm or edge already marked 61dc024 fix typos 798b561 Remove the duplicated word 1345608 modify grammatical error, replace 'a' with 'an' d2606c4 Remove mox from lower-constraints.txt d0af1a0 Add .rst files check to pep8 f47cf92 Modify the value of auth_url 2382591 Prometheus datasource 6484eb0 don't expose exception info on http response 43c2971 Cleanup exception logging ad35fd5 move db deletion from main to graph_init 1cfefbc Add release notes link to README a20c5a5 fix typo 49322bd Updated the API documentation 12113b1 Stopping of vitrage-graph a37c248 changes to graph stop and init 1149e45 K8s datasource configuration doc 77edb87 Kubernetes datasource b73d66e Add a new tempest job with a mock datasource e32752a remove symbolic link 50d7182 Update auth_url in install docs
Diffstat (except docs and test files) -------------------------------------
.gitignore | 1 - .testr.conf | 7 - .zuul.yaml | 60 ++- README.rst | 1 + devstack/apache-vitrage.template | 1 + devstack/gate_hook.sh | 3 - devstack/gate_hook_mock.sh | 67 +++ devstack/plugin.sh | 4 +- devstack/settings | 2 +- devstack/upgrade/settings | 13 + devstack/upgrade/shutdhown.sh | 35 ++ devstack/upgrade/upgrade.sh | 92 ++++ etc/vitrage/api-paste.ini | 10 +- lower-constraints.txt | 9 +- playbooks/legacy/vitrage-dsvm-mock-py27/post.yaml | 80 ++++ playbooks/legacy/vitrage-dsvm-mock-py27/run.yaml | 51 +++ playbooks/legacy/vitrage-dsvm-mock-py35/post.yaml | 80 ++++ playbooks/legacy/vitrage-dsvm-mock-py35/run.yaml | 51 +++ playbooks/legacy/vitrage-grenade/post.yaml | 16 + playbooks/legacy/vitrage-grenade/run.yaml | 58 +++ .../alarm-and-rca-history-0c4c513c98fdc426.yaml | 6 + ...dh_extended_alarm_support-327bb589311da440.yaml | 5 + ...llector-cache-limitations-4eedef1c7664f5b0.yaml | 7 + .../graph-fast-failover-e2d0a62f18ab27f7.yaml | 6 + .../kubernetes-datasource-12af0f9caac1b21b.yaml | 2 + .../prometheus_datasource-65c72bc8e7ec7622.yaml | 6 + .../notes/rocky-prelude-24ec92aedd5d8266.yaml | 10 + ...upport-networkx-version-2-8d47034b6b278137.yaml | 3 + .../notes/support_high_scale-fa1053f06954aed7.yaml | 3 + requirements.txt | 4 +- setup.cfg | 1 + test-requirements.txt | 1 - tools/load_generator/load_generator.py | 4 +- tox.ini | 14 +- vitrage/api/controllers/rest.py | 14 +- vitrage/api/controllers/v1/alarm.py | 60 +-- vitrage/api/controllers/v1/alarm_base.py | 79 ++++ vitrage/api/controllers/v1/count.py | 9 +- vitrage/api/controllers/v1/event.py | 22 +- vitrage/api/controllers/v1/history.py | 32 ++ vitrage/api/controllers/v1/rca.py | 9 +- vitrage/api/controllers/v1/resource.py | 25 +- vitrage/api/controllers/v1/template.py | 71 ++- vitrage/api/controllers/v1/topology.py | 26 +- vitrage/api/controllers/v1/webhook.py | 26 +- vitrage/api_handler/apis/alarm.py | 137 +++--- vitrage/api_handler/apis/base.py | 92 ---- vitrage/api_handler/apis/event.py | 3 +- vitrage/api_handler/apis/rca.py | 139 ++---- vitrage/cli/graph.py | 11 - vitrage/cli/storage.py | 12 + vitrage/common/constants.py | 30 +- vitrage/common/policies/alarms.py | 23 + vitrage/common/utils.py | 29 ++ vitrage/datasources/__init__.py | 3 +- vitrage/datasources/alarm_driver_base.py | 3 +- vitrage/datasources/aodh/driver.py | 94 +++- vitrage/datasources/aodh/properties.py | 11 + vitrage/datasources/ceilometer/driver.py | 9 +- vitrage/datasources/collectd/driver.py | 4 +- vitrage/datasources/collector_notifier.py | 58 --- vitrage/datasources/driver_base.py | 39 +- vitrage/datasources/kubernetes/__init__.py | 44 ++ vitrage/datasources/kubernetes/driver.py | 75 ++++ vitrage/datasources/kubernetes/properties.py | 41 ++ vitrage/datasources/kubernetes/transformer.py | 111 +++++ vitrage/datasources/listener_service.py | 4 - vitrage/datasources/nagios/config.py | 4 +- vitrage/datasources/nagios/driver.py | 4 + vitrage/datasources/nagios/parser.py | 5 +- vitrage/datasources/neutron/network/driver.py | 3 +- vitrage/datasources/neutron/port/driver.py | 3 +- vitrage/datasources/prometheus/__init__.py | 39 ++ vitrage/datasources/prometheus/driver.py | 177 ++++++++ vitrage/datasources/prometheus/properties.py | 57 +++ vitrage/datasources/prometheus/transformer.py | 92 ++++ vitrage/datasources/rpc_service.py | 46 +- vitrage/datasources/sample | 1 - vitrage/datasources/static/driver.py | 4 + vitrage/datasources/static_physical/__init__.py | 31 +- vitrage/datasources/transformer_base.py | 10 +- vitrage/datasources/utils.py | 6 + vitrage/datasources/zabbix/driver.py | 8 +- .../consistency/consistency_enforcer.py | 45 +- vitrage/entity_graph/datasource_rpc.py | 29 +- vitrage/entity_graph/graph_init.py | 94 +++- vitrage/entity_graph/graph_persistency.py | 121 +++++ .../mappings/datasource_info_mapper.py | 48 +- vitrage/entity_graph/processor/notifier.py | 212 ++++++--- vitrage/entity_graph/processor/processor.py | 86 ++-- vitrage/entity_graph/processor/processor_utils.py | 4 + .../entity_graph/processor/transformer_manager.py | 6 +- vitrage/entity_graph/scheduler.py | 38 +- vitrage/entity_graph/workers.py | 37 +- vitrage/evaluator/actions/notifier.py | 7 +- vitrage/evaluator/scenario_evaluator.py | 17 +- vitrage/graph/algo_driver/networkx_algorithm.py | 6 - vitrage/graph/driver/graph.py | 21 +- vitrage/graph/driver/networkx_graph.py | 16 +- vitrage/graph/driver/notifier.py | 17 +- vitrage/graph/utils.py | 10 +- .../accumulation_persistor_utils.py | 8 +- .../jaccard_correlation/correlation_manager.py | 4 +- vitrage/messaging.py | 4 +- vitrage/middleware/basic_and_keystone_auth.py | 138 ++++++ vitrage/middleware/keycloak.py | 11 +- vitrage/notifier/plugins/aodh/aodh_notifier.py | 10 +- vitrage/notifier/plugins/nova/nova_notifier.py | 8 +- vitrage/notifier/plugins/snmp/snmp_notifier.py | 6 +- vitrage/notifier/plugins/webhook/webhook.py | 5 +- vitrage/os_clients.py | 46 +- vitrage/persistency/__init__.py | 14 +- vitrage/persistency/graph_persistor.py | 60 --- vitrage/persistency/service.py | 165 ++++++- vitrage/service.py | 14 + vitrage/storage/__init__.py | 7 + vitrage/storage/base.py | 64 ++- vitrage/storage/history_facade.py | 443 ++++++++++++++++++ vitrage/storage/impl_sqlalchemy.py | 224 ++++++++-- vitrage/storage/sqlalchemy/models.py | 163 ++++++- .../entity_graph/consistency/test_consistency.py | 123 ++++- .../graph_persistor/test_graph_persistor.py | 140 +++--- .../functional/evaluator/test_action_executor.py | 4 +- .../evaluator/test_scenario_evaluator.py | 12 +- .../resources/kubernetes/kubernetes_config.yaml | 19 + .../driver/driver_kubernetes_snapshot_dynamic.json | 11 + .../driver/driver_prometheus_update_dynamic.json | 55 +++ .../transformer_aodh_snapshot_dynamic.json | 1 + .../transformer_aodh_update_dynamic.json | 1 + .../transformer_collectd_update_dynamic.json | 1 + .../transformer_doctor_update_dynamic.json | 1 + .../transformer_inst_snapshot_dynamic.json | 1 + .../transformer_prometheus_update_dynamic.json | 27 ++ .../vertices/cinder.volume.json | 1 + .../vertices/neutron.network.json | 3 +- .../mock_configurations/vertices/neutron.port.json | 1 + .../mock_configurations/vertices/nova.host.json | 5 +- .../vertices/nova.instance.json | 1 + .../mock_configurations/vertices/nova.zone.json | 1 + .../vertices/tripleo.controller.json | 1 + .../mock_configurations/vertices/zabbix.json | 1 + .../templates/basic_with_two_includes.yaml | 2 +- .../unit/datasources/aodh/test_aodh_driver.py | 495 +++++++++++++++++++++ .../kubernetes/test_kubernetes_transformer.py | 162 +++++++ .../nova/test_nova_instance_transformer.py | 2 +- .../prometheus/test_prometheus_driver.py | 67 +++ .../prometheus/test_prometheus_transformer.py | 101 +++++ .../states/test_datasource_info_mapper.py | 4 +- 166 files changed, 5607 insertions(+), 1610 deletions(-)
Requirements updates --------------------
diff --git a/requirements.txt b/requirements.txt index b01eb79..4407e27 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11 +10,0 @@ python-ceilometerclient>=2.9.0 # Apache-2.0 -python-cinderclient>=3.5.0 # Apache-2.0 @@ -17,0 +17,2 @@ python-mistralclient>=3.3.0 # Apache-2.0 +python-openstackclient>=3.12.0 # Apache-2.0 +gnocchiclient>=3.3.1 # Apache-2.0 @@ -48,0 +50 @@ futures>=3.0.0;python_version=='2.7' or python_version=='2.6' # BSD +pytz>=2013.6 # MIT \ No newline at end of file diff --git a/test-requirements.txt b/test-requirements.txt index b2223f2..5a46dd5 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -10 +9,0 @@ requests-mock>=1.4.0 # Apache-2.0 -testrepository>=0.0.20 # Apache-2.0/BSD