We are frolicsome to announce the release of: python-watcher 0.29.0: Watcher takes advantage of CEP and ML algorithms/metaheuristics to improve physical resources usage through better VM placement. Watcher can improve your cloud optimization by reducing energy footprint and increasing profits. This release is part of the newton release series. With source available at: https://github.com/openstack/watcher Please report issues through launchpad: http://bugs.launchpad.net/watcher For more details, please see below. 0.29.0 ^^^^^^ New Features * Added a standard way to both declare and fetch configuration options so that whenever the administrator generates the Watcher configuration sample file, it contains the configuration options of the plugins that are currently available. * Added a generic scoring engine module, which will standarize interactions with scoring engines through the common API. It is possible to use the scoring engine by different Strategies, which improve the code and data model re-use. * Added an in-memory cache of the cluster model built up and kept fresh via notifications from services of interest in addition to periodic syncing logic. * Added a way to add a new action without having to amend the source code of the default planner. * Added a way to create periodic audit to be able to optimize continuously the cloud infrastructure. * Added a way to compare the efficacy of different strategies for a give optimization goal. * Added a way to return the of available goals depending on which strategies have been deployed on the node where the decison engine is running. * Allow decision engine to pass strategy parameters, like optimization threshold, to selected strategy, also strategy to provide parameters info to end user. * Copy all audit templates parameters into audit instead of having a reference to the audit template. * Added a strategy that monitors if there is a higher load on some hosts compared to other hosts in the cluster and re-balances the work across hosts to minimize the standard deviation of the loads in the cluster. * Added a new strategy based on the airflow of servers. This strategy makes decisions to migrate VMs to make the airflow uniform. * Added policies to handle user rights to access Watcher API. * Added a strategy based on the VM workloads of hypervisors. This strategy makes decisions to migrate workloads to make the total VM workloads of each hypervisor balanced, when the total VM workloads of hypervisor reaches threshold. Changes in python-watcher 0.28.0..0.29.0 ---------------------------------------- 6e8dc52 Merge scoring base files 7cce4b9 Add release notes for Newton blueprints deb5cb3 TrivialFix: Remove cfg import unused 720884c TrivialFix: Remove logging import unused 249cd11 Remove unused LOG e4ba59e Update configuration section for notifications f238167 Doc on how to add notification endpoints 77b7fae Notification and CDM partial update 103f541 Remove unreachable line c617938 Added start/end date params on ceilometer queries ecea228 Correct watcher reraising of exception 1fafcc5 Check unspecified parameters create audit 32c13d0 Fix loading of plugin configuration parameters a1cb142 Add Scoring Module implementation da67b24 Add unit tests for continuous.py 6d41c23 Updated from global requirements 7b22834 Fixed flaky tempest test e741728 Remove pot files 35201c8 Updated from global requirements 6be758b Added strategy ID + Action Plan syncing 9450a70 Fixes to get cluster data model 64f45ad Updated from global requirements 4e3593a Fix double self._goal definition 01164b0 Scheduler of decision_engine fix 9933955 Updated from global requirements 17d2d75 Clean imports in code 2e55f4e Modify libvirt_opts="-d -l" to libvirtd_opts="-d -l" ea01031 Rename (pre/post)condition to (pre/post)_condition 6144551 Add unit tests for nova_helper.py 1b2672a Updated from global requirements 31de0e3 Removed unused function in uniform airflow 8145906 Update the home-page info with the developer documentation 9d2d218 Updated from global requirements 31c3734 Refactored the compute model and its elements cc2e805 Use more specific asserts 0a6841f Merged metrics_engine package into decision_engine 4f8591c Updated DE architecture doc + 'period' param 06c6c46 Added DE Background Scheduler w/ model sync jobs b94677c Cluster data model collector plugin documentation 5a2a94f Loadable Cluster Data Model Collectors 61e581f Updated from global requirements 26d84e3 Add scoring engines to database and API layers 52ffc2c Implement goal_id, strategy_id and host_aggregate into Audit api 5dee934 use parameters to set the threshold 0769e53 Updated from global requirements 107bd0b Fixed Basic optim tempest test 926b790 Fix 2 occurrences of typo: "occured" --> "occurred" 7d704db Add hacking checks to watcher 5c08095 Update docs links to docs.openstack.org b579a41 Remove discover from test-requirements 051810d Updated from global requirements 6730202 Fix typos and messages in strategies fd7c41f Remove unused columns parameters in watcher/db/api aef1eba test_context_hook_before_method failed cd60336 Fix dict.keys() PY3 compatible e7a1ba7 There are some spelling errors in the code. 1b0801a Add Python 3.5 classifier and venv ff43753 Add installation from Debian packages section 21d1610 Update executor to eventlet Diffstat (except docs and test files) ------------------------------------- .coveragerc | 4 +- devstack/lib/watcher | 5 + .../plantuml/sequence_architecture_cdmc_sync.txt | 41 + .../sequence_trigger_audit_in_decision_engine.txt | 54 +- .../images/sequence_architecture_cdmc_sync.png | Bin 0 -> 46795 bytes .../sequence_trigger_audit_in_decision_engine.png | Bin 83923 -> 73345 bytes etc/watcher/policy.json | 4 + .../add-plugins-parameters-376eb6b0b8978b44.yaml | 8 + .../notes/add-scoring-module-fa00d013ed2d614e.yaml | 7 + ...ter-model-objects-wrapper-9c799ea262c56a5b.yaml | 6 + ...e-weights-default-planner-3746b33160bc7347.yaml | 4 + ...continuously-optimization-35364f4d2c0b81fc.yaml | 4 + .../notes/efficacy-indicator-95380ad7b84e3be2.yaml | 4 + .../get-goal-from-strategy-396c9b13a38bb650.yaml | 5 + .../optimization-threshold-21ad38f0470d0e1a.yaml | 5 + ...rsistent-audit-parameters-ae41dd7252ba9672.yaml | 6 + ...andard-deviation-strategy-cd1d0c443fdfde9c.yaml | 7 + .../uniform-airflow-strategy-68cdba1419c3f770.yaml | 5 + .../notes/watcher-policies-1e86a30f0f11c6fa.yaml | 4 + ...alance-migration-strategy-a0b05148a57815c0.yaml | 7 + requirements.txt | 23 +- setup.cfg | 14 +- test-requirements.txt | 3 +- tox.ini | 3 +- watcher/api/controllers/v1/__init__.py | 13 + watcher/api/controllers/v1/action.py | 4 +- watcher/api/controllers/v1/action_plan.py | 99 ++- watcher/api/controllers/v1/audit.py | 286 ++++--- watcher/api/controllers/v1/collection.py | 4 +- watcher/api/controllers/v1/goal.py | 4 - watcher/api/controllers/v1/scoring_engine.py | 246 ++++++ watcher/api/controllers/v1/strategy.py | 4 - watcher/api/controllers/v1/types.py | 4 +- watcher/api/middleware/parsable_error.py | 5 +- watcher/applier/action_plan/default.py | 10 +- watcher/applier/actions/base.py | 8 +- .../applier/actions/change_nova_service_state.py | 17 +- watcher/applier/actions/migration.py | 60 +- watcher/applier/actions/nop.py | 8 +- watcher/applier/actions/sleep.py | 10 +- watcher/applier/manager.py | 4 +- watcher/applier/rpcapi.py | 7 +- watcher/applier/workflow_engine/base.py | 3 +- watcher/applier/workflow_engine/default.py | 24 +- watcher/cmd/api.py | 3 +- watcher/cmd/applier.py | 5 +- watcher/cmd/dbmanage.py | 6 +- watcher/cmd/decisionengine.py | 9 +- watcher/cmd/sync.py | 39 + watcher/common/ceilometer_helper.py | 55 +- watcher/common/context.py | 118 ++- watcher/common/exception.py | 37 +- watcher/common/loader/default.py | 2 +- watcher/common/loader/loadable.py | 31 + watcher/common/messaging/messaging_handler.py | 2 +- watcher/common/nova_helper.py | 84 +- watcher/common/rpc.py | 41 +- watcher/common/scheduling.py | 44 + watcher/common/service.py | 88 +- watcher/common/utils.py | 33 +- watcher/db/api.py | 85 +- watcher/db/purge.py | 58 +- watcher/db/sqlalchemy/api.py | 105 ++- watcher/db/sqlalchemy/models.py | 34 +- watcher/decision_engine/audit/base.py | 3 +- watcher/decision_engine/audit/continuous.py | 10 +- watcher/decision_engine/audit/oneshot.py | 4 +- watcher/decision_engine/cluster/__init__.py | 0 .../decision_engine/cluster/history/__init__.py | 0 watcher/decision_engine/cluster/history/base.py | 80 ++ .../decision_engine/cluster/history/ceilometer.py | 45 + watcher/decision_engine/goal/base.py | 4 - watcher/decision_engine/goal/efficacy/base.py | 4 +- .../decision_engine/goal/efficacy/indicators.py | 8 +- watcher/decision_engine/goal/efficacy/specs.py | 6 +- watcher/decision_engine/goal/goals.py | 20 +- watcher/decision_engine/loading/default.py | 20 + watcher/decision_engine/manager.py | 33 +- .../decision_engine/messaging/audit_endpoint.py | 1 + .../decision_engine/model/collector/__init__.py | 0 watcher/decision_engine/model/collector/base.py | 178 ++++ watcher/decision_engine/model/collector/manager.py | 61 ++ watcher/decision_engine/model/collector/nova.py | 113 +++ watcher/decision_engine/model/compute_resource.py | 50 -- watcher/decision_engine/model/disk_info.py | 53 -- watcher/decision_engine/model/element/__init__.py | 38 + watcher/decision_engine/model/element/base.py | 29 + .../model/element/compute_resource.py | 57 ++ watcher/decision_engine/model/element/disk_info.py | 59 ++ watcher/decision_engine/model/element/instance.py | 54 ++ watcher/decision_engine/model/element/node.py | 53 ++ watcher/decision_engine/model/element/resource.py | 61 ++ watcher/decision_engine/model/hypervisor.py | 51 -- watcher/decision_engine/model/hypervisor_state.py | 24 - watcher/decision_engine/model/mapping.py | 118 ++- watcher/decision_engine/model/model_root.py | 151 +++- .../decision_engine/model/notification/__init__.py | 0 watcher/decision_engine/model/notification/base.py | 47 ++ .../model/notification/filtering.py | 88 ++ watcher/decision_engine/model/notification/nova.py | 334 ++++++++ watcher/decision_engine/model/power_state.py | 31 - watcher/decision_engine/model/resource.py | 58 -- watcher/decision_engine/model/sla/__init__.py | 0 watcher/decision_engine/model/vm.py | 31 - watcher/decision_engine/model/vm_state.py | 34 - watcher/decision_engine/planner/default.py | 18 +- watcher/decision_engine/rpcapi.py | 2 + watcher/decision_engine/scheduling.py | 92 +++ watcher/decision_engine/scoring/__init__.py | 0 watcher/decision_engine/scoring/base.py | 126 +++ watcher/decision_engine/scoring/dummy_scorer.py | 169 ++++ .../scoring/dummy_scoring_container.py | 98 +++ watcher/decision_engine/scoring/scoring_factory.py | 106 +++ watcher/decision_engine/strategy/context/base.py | 12 +- .../decision_engine/strategy/context/default.py | 22 +- .../decision_engine/strategy/selection/default.py | 2 - .../strategy/strategies/__init__.py | 5 +- .../decision_engine/strategy/strategies/base.py | 23 +- .../strategy/strategies/basic_consolidation.py | 300 ++++--- .../strategy/strategies/dummy_strategy.py | 8 +- .../strategy/strategies/dummy_with_scorer.py | 166 ++++ .../strategy/strategies/outlet_temp_control.py | 143 ++-- .../strategy/strategies/uniform_airflow.py | 309 +++---- .../strategies/vm_workload_consolidation.py | 390 +++++---- .../strategy/strategies/workload_balance.py | 247 +++--- .../strategy/strategies/workload_stabilization.py | 233 +++--- watcher/decision_engine/sync.py | 258 +++++- watcher/doc.py | 69 -- watcher/hacking/__init__.py | 0 watcher/hacking/checks.py | 309 +++++++ watcher/locale/watcher.pot | 909 --------------------- watcher/metrics_engine/__init__.py | 0 watcher/metrics_engine/cluster_history/__init__.py | 0 watcher/metrics_engine/cluster_history/base.py | 80 -- .../metrics_engine/cluster_history/ceilometer.py | 49 -- .../cluster_model_collector/__init__.py | 0 .../metrics_engine/cluster_model_collector/base.py | 119 --- .../cluster_model_collector/manager.py | 34 - .../metrics_engine/cluster_model_collector/nova.py | 81 -- watcher/objects/__init__.py | 4 +- watcher/objects/action_plan.py | 3 +- watcher/objects/audit.py | 4 +- watcher/objects/base.py | 10 +- watcher/objects/scoring_engine.py | 257 ++++++ watcher/opts.py | 3 + .../action_plan/test_default_action_handler.py | 18 +- .../actions/test_change_nova_service_state.py | 26 +- .../test_default_workflow_engine.py | 4 +- .../common/messaging/test_messaging_handler.py | 1 + .../decision_engine/audit/test_audit_handlers.py | 42 +- .../cluster/test_cluster_data_model_collector.py | 54 ++ .../decision_engine/cluster/test_nova_cdmc.py | 82 ++ .../loading/test_collector_loader.py | 83 ++ .../loading/test_default_planner_loader.py | 31 + .../loading/test_default_strategy_loader.py | 78 ++ .../decision_engine/loading/test_goal_loader.py | 78 ++ .../messaging/test_audit_endpoint.py | 2 +- .../decision_engine/model/notification/__init__.py | 0 .../model/notification/data/instance-create.json | 70 ++ .../notification/data/instance-delete-end.json | 49 ++ .../model/notification/data/instance-update.json | 65 ++ .../data/scenario3_instance-create.json | 70 ++ .../data/scenario3_instance-delete-end.json | 49 ++ .../data/scenario3_instance-update.json | 65 ++ .../data/scenario3_legacy_instance-create-end.json | 62 ++ .../data/scenario3_legacy_instance-delete-end.json | 46 ++ .../data/scenario3_legacy_instance-update.json | 52 ++ ...enario3_legacy_livemigration-post-dest-end.json | 61 ++ .../data/scenario3_service-update.json | 21 + .../model/notification/data/service-update.json | 21 + .../model/notification/fake_managers.py | 53 ++ .../model/notification/test_notifications.py | 106 +++ .../model/notification/test_nova_notifications.py | 450 ++++++++++ .../decision_engine/model/test_named_element.py | 32 - .../planner/test_default_planner.py | 79 +- .../planner/test_default_planner_loader.py | 31 - .../decision_engine/scoring/test_dummy_scorer.py | 54 ++ .../scoring/test_dummy_scoring_container.py | 51 ++ .../scoring/test_scoring_factory.py | 53 ++ .../solution/test_default_solution.py | 11 +- .../strategy/context/test_strategy_context.py | 28 +- .../decision_engine/strategy/loading/__init__.py | 0 .../loading/test_default_strategy_loader.py | 78 -- .../strategy/selector/test_strategy_selector.py | 3 - .../strategies/faker_cluster_and_metrics.py | 268 +++--- .../strategy/strategies/faker_cluster_state.py | 431 +++++----- .../strategy/strategies/faker_metrics_collector.py | 73 +- .../strategies/test_basic_consolidation.py | 120 +-- .../strategy/strategies/test_dummy_strategy.py | 4 +- .../strategy/strategies/test_dummy_with_scorer.py | 61 ++ .../strategies/test_outlet_temp_control.py | 62 +- .../strategy/strategies/test_uniform_airflow.py | 123 +-- .../strategies/test_vm_workload_consolidation.py | 172 ++-- .../strategy/strategies/test_workload_balance.py | 106 +-- .../strategies/test_workload_stabilization.py | 29 +- .../services/infra_optim/v1/json/client.py | 8 +- 261 files changed, 10240 insertions(+), 4919 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 74cb8d3..774b848 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8 +8 @@ jsonpatch>=1.1 # BSD -keystoneauth1>=2.7.0 # Apache-2.0 +keystoneauth1>=2.10.0 # Apache-2.0 @@ -12,3 +12,3 @@ oslo.cache>=1.5.0 # Apache-2.0 -oslo.config>=3.10.0 # Apache-2.0 -oslo.context>=2.4.0 # Apache-2.0 -oslo.db>=4.1.0 # Apache-2.0 +oslo.config>=3.14.0 # Apache-2.0 +oslo.context>=2.9.0 # Apache-2.0 +oslo.db>=4.10.0 # Apache-2.0 @@ -19,0 +20 @@ oslo.reports>=0.6.0 # Apache-2.0 +oslo.serialization>=1.10.0 # Apache-2.0 @@ -21 +22 @@ oslo.service>=1.10.0 # Apache-2.0 -oslo.utils>=3.14.0 # Apache-2.0 +oslo.utils>=3.16.0 # Apache-2.0 @@ -24 +25 @@ pbr>=1.6 # Apache-2.0 -pecan>=1.0.0 # BSD +pecan!=1.0.2,!=1.0.3,!=1.0.4,>=1.0.0 # BSD @@ -27 +28 @@ voluptuous>=0.8.9 # BSD License -python-ceilometerclient>=2.2.1 # Apache-2.0 +python-ceilometerclient>=2.5.0 # Apache-2.0 @@ -29,3 +30,3 @@ python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0 -python-glanceclient>=2.0.0 # Apache-2.0 -python-keystoneclient!=1.8.0,!=2.1.0,>=1.7.0 # Apache-2.0 -python-neutronclient>=4.2.0 # Apache-2.0 +python-glanceclient!=2.4.0,>=2.3.0 # Apache-2.0 +python-keystoneclient!=2.1.0,>=2.0.0 # Apache-2.0 +python-neutronclient>=5.1.0 # Apache-2.0 @@ -36 +37 @@ SQLAlchemy<1.1.0,>=1.0.10 # MIT -stevedore>=1.10.0 # Apache-2.0 +stevedore>=1.16.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 27e4a7f..0ccc117 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -6 +5,0 @@ coverage>=3.6 # Apache-2.0 -discover # BSD @@ -27 +26 @@ reno>=1.8.0 # Apache2 -bandit>=1.0.1 # Apache-2.0 +bandit>=1.1.0 # Apache-2.0