We joyfully announce the release of: vitrage 4.0.0: The OpenStack RCA Service This release is part of the stein release series. The source is available from: https://git.openstack.org/cgit/openstack/vitrage Download the package from: https://tarballs.openstack.org/vitrage/ For more details, please see below. 4.0.0 ^^^^^ Prelude ******* Added new tool "vitrage-status upgrade check". New Features ************ * New framework for "vitrage-status upgrade check" command is added. This framework allows adding various checks which can be run before a Vitrage upgrade to ensure if the upgrade can be performed safely. * Collector service removal to simplify and enhance scale performance. vitrage-collector service was removed and vitrage-graph is responsible to execute the drivers. Allowing drivers to take advantage of python yield generators and conserve memory. * Use Nova versioned notifications instead of the legacy, unversioned ones. This bahavior is controlled by the "use_nova_versioned_notifications" configuration option. * Resource count new API with support for queries and group-by. Allows retrieving quick summaries of graph nodes. * Resource list API now supports using a query * A new "Trove Datasource" has been introduced to include Trove entities (database instances and clusters) in Vitrage Entity Graph. Trove is Database as a Service solution offering database lifecycle management (automated provisioning, configuration, backups, clustering etc.). Adding the datasource to Vitrage enables detecting problems at lower levels of infrastructure that may affect functioning of running databases, and react in response to identified issues e.g. scale the database up/out or live-migrate virtual machines from failed compute. This change is the first stage of integration with Trove. At this point, Trove entities are extracted using PULL approach, based on periodical snapshot-query to Trove API for the current list of Trove entities. In the future, PUSH approach based on Trove notifications will be implemented. Upgrade Notes ************* * Operator can now use new CLI tool "vitrage-status upgrade check" to check if Vitrage deployment can be safely upgraded from N-1 to N release. Deprecation Notes ***************** * The static_physical datasource was removed. Please use the static datasource instead. * Resource list GET is deprecated, use POST instead. Changes in vitrage 3.1.0..4.0.0 ------------------------------- 29665f7 Add missing status codes documentation b730d3d add mock dependency to tests ed952f7 Update hacking version to latest d1b5a3e Enhance Vitrage resource APIs d7d4188 Check if instance_id is not None f617f65 Fix topology api crashes e0ccccc Api-worker sync threads - lock graph from changing during API calls. 154b7c0 Add Trove cluster capacity autoscaling example 007f937 Fetch data immediately after starting from a stored snapshot 6599f3a mock driver - generate consistent uuids 149045e Add Trove datasource 81e3d95 Change init fork order 48897a3 use by default uwsgi on devstack 75fb4d1 Consistent timestamp format for all datasources 832522c Use Nova versioned notifications by default a96bc06 Support Nova versioned notifications ed9f7d8 Remove the static physical datasource 3a83fc8 Add framework for vitrage-status upgrade check a21a32c Api performance enhancements f8e3273 force the use of networkx 2.0 and above 5409f4a Change openstack-dev to openstack-discuss a54f579 Bug fix: delete outdated entities for OpenStack datasources 0586105 Better logs for consistency 842f9d6 Complete removal of vitrage-collector service. 4d686c8 Graph service stop should return exit code 0 4bba457 Processor log warnings d417203 Bugfix: return the correct start/end timestamp in Prometheus datasource efb2a03 auth url should never should be None 3c22ee1 Template loading moved from TemplateWorker to EvaluatorWorker 1e894ae Remove setup.py check from pep8 job a73cde4 don't call get_apache_version 37a1870 bugfix: delete resolved prometheus alarms 134f0f2 Update Vitrage architecture diagram for Rocky c8b8df4 Bugfix: Add datasource_name to an entity also on update 513bd5c make user_info_endpoint_url independent of auth_url 269c7fb dont use rabbit_hosts on conf 9afe196 Don't quote {posargs} in tox.ini 5ca6e97 Update high-scale documentation 5953fd5 bugfix: update a placeholder vertex can add properties to a real vertex a9913ff get changes rpc response should be compressed 152fc0d Cleanup .zuul.yaml 9f607dc Using storyboard to collect documentation bugs e069b90 Worker deleted entities removal 85f069b fix port filtering 5b0acfc Add py36 to tox 8a14c37 Add storyboard link into README.rst 67af2c7 bug fix: add prometheus.yaml to datasources_values 0b5ed53 add timezone to created_at field in template list 8d950cc add python 3.6 unit test job a32745d switch documentation job to new PTI 1fe8d73 import zuul job settings from project-config 4756182 bug fix - aodh gnocchi alarm not connected to instance. 4c5d8cb convert iterator to list to prevent change in iterator size during iteration fe2f627 Always load database graph snapshot on restart 2531c72 dont add vitrage_resource_project_id if None 37569b1 remove redundant parenthesis 72d018c Update reno for stable/rocky Diffstat (except docs and test files) ------------------------------------- .zuul.yaml | 21 +- README.rst | 2 +- devstack/README.rst | 4 + devstack/gate_hook.sh | 7 +- devstack/gate_hook_mock.sh | 5 + devstack/plugin.sh | 108 +++---- devstack/settings | 13 +- devstack/upgrade/settings | 6 +- devstack/upgrade/shutdhown.sh | 2 +- devstack/upgrade/upgrade.sh | 8 +- .../images/vitrage_graph_architecture.png | Bin 225491 -> 192537 bytes .../template_validation_status_code.rst | 11 + etc/vitrage/datasources_values/prometheus.yaml | 17 ++ etc/vitrage/datasources_values/trove.cluster.yaml | 31 ++ etc/vitrage/datasources_values/trove.instance.yaml | 47 +++ .../database_cluster_capacity_autoscaling.yaml | 82 +++++ lower-constraints.txt | 4 +- ...d-upgrade-check-framework-166ba1c7e266953e.yaml | 13 + .../notes/collector-removal-fd805c6298d66eb0.yaml | 7 + ...a-versioned-notifications-3c5ff450b9fe69f0.yaml | 5 + ...tatic-physical-datasource-e8e70158f453cff1.yaml | 4 + .../notes/resource_count-1e3184e5f1f413ab.yaml | 4 + .../resource_list_query-b3fbcbe01090f64a.yaml | 5 + .../notes/trove-datasource-2aa7a88ff20aff8c.yaml | 14 + releasenotes/source/index.rst | 1 + releasenotes/source/rocky.rst | 6 + requirements.txt | 7 +- setup.cfg | 4 +- test-requirements.txt | 3 +- tox.ini | 7 +- vitrage/api/app.py | 1 + vitrage/api/controllers/rest.py | 19 +- vitrage/api/controllers/v1/alarm.py | 2 +- vitrage/api/controllers/v1/alarm_base.py | 7 +- vitrage/api/controllers/v1/count.py | 38 ++- vitrage/api/controllers/v1/resource.py | 73 +++-- vitrage/api/controllers/v1/root.py | 4 + vitrage/api/controllers/v1/template.py | 5 + vitrage/api/controllers/v1/topology.py | 5 +- vitrage/api/hooks.py | 10 +- vitrage/api_handler/apis/alarm.py | 16 +- vitrage/api_handler/apis/base.py | 20 +- vitrage/api_handler/apis/event.py | 3 +- vitrage/api_handler/apis/operational.py | 33 ++ vitrage/api_handler/apis/rca.py | 10 +- vitrage/api_handler/apis/resource.py | 58 +++- vitrage/api_handler/apis/topology.py | 124 +++----- vitrage/api_handler/apis/webhook.py | 7 +- vitrage/cli/__init__.py | 19 +- vitrage/cli/collector.py | 53 ---- vitrage/cli/graph.py | 23 +- vitrage/cli/status.py | 53 ++++ vitrage/common/policies/resource.py | 25 ++ vitrage/common/utils.py | 39 +++ vitrage/datasources/__init__.py | 4 - vitrage/datasources/aodh/driver.py | 4 +- vitrage/datasources/cinder/volume/driver.py | 4 + vitrage/datasources/cinder/volume/transformer.py | 3 +- vitrage/datasources/driver_base.py | 2 +- vitrage/datasources/heat/stack/driver.py | 4 + vitrage/datasources/listener_service.py | 103 ------- vitrage/datasources/neutron/network/driver.py | 4 + vitrage/datasources/neutron/port/driver.py | 14 +- vitrage/datasources/nova/host/driver.py | 4 + vitrage/datasources/nova/instance/driver.py | 118 ++++++-- .../datasources/nova/instance/field_extractor.py | 93 ++++++ vitrage/datasources/nova/instance/transformer.py | 79 +++-- vitrage/datasources/prometheus/driver.py | 29 +- vitrage/datasources/prometheus/properties.py | 7 +- vitrage/datasources/rpc_service.py | 92 ------ vitrage/datasources/static/__init__.py | 3 - vitrage/datasources/static_physical/__init__.py | 75 ----- vitrage/datasources/static_physical/driver.py | 167 ----------- vitrage/datasources/static_physical/transformer.py | 128 -------- vitrage/datasources/transformer_base.py | 4 +- vitrage/datasources/trove/__init__.py | 0 vitrage/datasources/trove/cluster/__init__.py | 44 +++ vitrage/datasources/trove/cluster/driver.py | 51 ++++ vitrage/datasources/trove/cluster/transformer.py | 84 ++++++ vitrage/datasources/trove/instance/__init__.py | 44 +++ vitrage/datasources/trove/instance/driver.py | 49 +++ vitrage/datasources/trove/instance/transformer.py | 81 +++++ vitrage/datasources/trove/properties.py | 34 +++ vitrage/datasources/trove/trove_driver_base.py | 104 +++++++ .../consistency/consistency_enforcer.py | 22 +- vitrage/entity_graph/datasource_rpc.py | 69 ----- vitrage/entity_graph/driver_exec.py | 134 +++++++++ vitrage/entity_graph/graph_init.py | 74 +++-- vitrage/entity_graph/graph_persistency.py | 23 +- vitrage/entity_graph/processor/processor.py | 84 +++--- vitrage/entity_graph/processor/processor_utils.py | 8 +- vitrage/entity_graph/scheduler.py | 43 +-- vitrage/entity_graph/workers.py | 201 ++++++------- .../actions/evaluator_event_transformer.py | 2 +- vitrage/evaluator/scenario_evaluator.py | 2 +- vitrage/evaluator/scenario_repository.py | 3 +- vitrage/graph/algo_driver/algorithm.py | 22 -- vitrage/graph/algo_driver/networkx_algorithm.py | 69 +---- vitrage/graph/driver/graph.py | 15 +- vitrage/graph/driver/networkx_graph.py | 37 ++- vitrage/middleware/basic_and_keystone_auth.py | 2 +- vitrage/middleware/keycloak.py | 7 +- vitrage/opts.py | 7 +- vitrage/os_clients.py | 27 +- vitrage/rpc.py | 14 +- vitrage/storage/base.py | 4 +- vitrage/storage/impl_sqlalchemy.py | 6 +- .../listener_service/test_listener_service.py | 11 +- .../datasources/static_physical/__init__.py | 0 .../static_physical/test_static_physical.py | 99 ------ .../datasources/trove/test_trove_instance.py | 92 ++++++ .../entity_graph/consistency/test_consistency.py | 158 +++++----- .../graph_persistor/test_graph_persistor.py | 8 - .../driver/driver_inst_update_dynamic.json | 46 --- .../driver/driver_inst_update_legacy_dynamic.json | 46 +++ .../driver_inst_update_versioned_dynamic.json | 103 +++++++ .../driver/driver_switch_snapshot_dynamic.json | 16 - .../driver_trove_cluster_snapshot_dynamic.json | 19 ++ .../driver_trove_instance_snapshot_dynamic.json | 13 + .../unit/datasources/aodh/test_aodh_driver.py | 4 +- .../nova/base_nova_instance_transformer.py | 246 +++++++++++++++ ...st_instance_transformer_legacy_notifications.py | 62 ++++ .../test_instance_transformer_snapshot_events.py | 130 ++++++++ ...instance_transformer_versioned_notifications.py | 62 ++++ .../datasources/nova/test_nova_instance_driver.py | 98 ++++++ .../nova/test_nova_instance_transformer.py | 332 --------------------- .../unit/datasources/static_physical/__init__.py | 0 .../static_physical/test_static_physical_driver.py | 145 --------- .../test_static_physical_transformer.py | 195 ------------ .../trove/test_trove_cluster_transformer.py | 151 ++++++++++ .../trove/test_trove_instance_transformer.py | 151 ++++++++++ .../unit/entity_graph/processor/test_processor.py | 18 +- .../unit/entity_graph/test_processor_service.py | 2 +- vitrage/utils/datetime.py | 9 + 160 files changed, 3958 insertions(+), 2609 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 4407e27..bee35a9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -17,0 +18 @@ python-openstackclient>=3.12.0 # Apache-2.0 +python-troveclient>=2.2.0 # Apache-2.0 @@ -20 +21 @@ pyzabbix>=0.7.4 # LGPL -networkx>=1.11 # BSD +networkx>=2.0 # BSD @@ -29,0 +31 @@ oslo.i18n>=3.20.0 # Apache-2.0 +oslo.upgradecheck>=0.1.1 # Apache-2.0 @@ -50 +52,2 @@ futures>=3.0.0;python_version=='2.7' or python_version=='2.6' # BSD -pytz>=2013.6 # MIT \ No newline at end of file +pytz>=2013.6 # MIT +tenacity>=4.9.0 diff --git a/test-requirements.txt b/test-requirements.txt index 5a46dd5..1cee643 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -5 +5 @@ -hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 +hacking>=1.1.0,<1.2.0 # Apache-2.0 @@ -13,0 +14 @@ reno>=2.7.0 # Apache-2.0 +mock>=2.0.0 # BSD