We are stoked to announce the release of: cloudkitty 10.0.0: Rating as a Service component for OpenStack This release is part of the train release series. The source is available from: https://opendev.org/openstack/cloudkitty Download the package from: https://pypi.org/project/cloudkitty Please report issues through: https://bugs.launchpad.net/cloudkitty/+bugs For more details, please see below. 10.0.0 ^^^^^^ New Features ************ * A Prometheus scope fetcher has been added in order to dynamically discover scopes from a Prometheus service using a user defined metric and a scope attribute. It can also filter out the response from Prometheus using metadata filters to have a more fine-grained control over scope discovery. It features HTTP basic auth capabilities and HTTPS configuration options similar to Prometheus collector. * Added a v2 API endpoint allowing to retrieve the state of several scopes. This endpoint is available via a "GET" request on "/v2/scope" and supports filters. Admin privileges are required to use this endpoint. * Added a v2 API endpoint allowing to reset the state of several scopes. This endpoint is available via a "PUT" request on "/v2/scope" and supports filters. Admin privileges are required to use this endpoint. * A v2 API has been bootstrapped. It is compatible with the v2 storage and will be the base for all upcoming API endpoints. It is marked as "EXPERIMENTAL" for now. * It is now possible to force a project_id to retrieve a specific metric from it with the monasca collector. * The processor is now able to run several parallel workers. By default, one worker is spawned for each available CPU. Workers can be limited through the "max_workers" option of the "orchestrator" section. Upgrade Notes ************* * The v1 API is now marked as "CURRENT". The API root is now built with Flask instead of pecan * Metrics are now fetched concurrently with "eventlet" instead of one after another by the orchestrator, leading to a consequent performance improvement. The maximum number of greenthreads to use can be specified through the "max_greenthreads" option of the "orchestrator" section. Security Issues *************** * The default policy for the "/v1/storage/dataframes" endpoint has been changed from "unprotected" (accessible by any unauthenticated used) to "admin_or_owner" (accessible only by admins or members of the project). * Introduce bandit security checks and fix potential security issues detected by bandit linter. Remove unused option where host_ip was a binding to all interfaces. Using of insecure hash function, switch from sha1 to sha512. Bug Fixes ********* * HashMap module field mapping matching has been fixed: Field mapping values are always stored as strings. However, metadatas to match can be floats or integers (eg vcpus or ram). Given that mappings were matched with "==" until now, integers or float metadatas did never match. * "CompileError: Can't resolve label reference for ORDER BY / GROUP BY." errors that were sometimes raised by SQLAlchemy when using the v1 storage backend and grouping on "tenant_id" and "res_type" have been fixed. Changes in cloudkitty 9.0.0.0rc1..10.0.0 ---------------------------------------- 043c793 Do not re-instantiate a StateManager for each request in /v2/scope aed221f Update PUT /v2/scope API reference f044c03 Fix v1 hybrid storage python2 python3 string type comparison in memory d78ba87 Add a v2 API endpoint to reset the state of different scopes b88c937 Add an option for the metric project to the Monasca collector f02196f Add a "delete" method to the v2 storage interface 1797c6c Add quotes to InfluxDb queries a9a56fb Add Monasca collector auth options to the sample config 22e8c93 Bump openstackdocstheme to 1.30.0 acd2fec Add a v2 summary endpoint 1c09de7 Update the "admin/configuration" section of the documentation b0e010a Modify the url of upper_constraints_file 460f12f Blacklist sphinx 2.1.0 (autodoc bug) f34624c Add python 3.7 classifier to setup.cfg 5e716e1 Use openstack-python3-train-jobs for python3 test runtime a88a756 Update the "architecture" section of the documentation 5fef2d6 Fix sqlalchemy grouping on v1 storage bba518c Fix Prometheus fetcher error 870679a Update sphinx dependency 9017472 Use a hash for lock names cb54087 Add a v2 API endpoint to get scope state abb955b Fix bandit job e7247dc Add 'rate:xxx' to gnocchi collector aggregation methods f664432 Remove "group_filters" parameter from v2 storage interface 7b0df70 Addition of catch block to avoid hiding Gnocchi schema validation errors a9f9be8 Updated tooz lock name 3c585ec Updated constraints on storage state b42c3a1 Add Fetcher documentation b70b748 Fix section name in config file generation cc562d4 Add a base resource for v2 API endpoints 8243e18 Replace git.openstack.org URLs with opendev.org URLs 99a3868 OpenDev Migration Patch 07163cb Retrieve metrics in eventlet greenthreads c9889f2 Fix rounding in v2 storage unit tests 46f58fd Add missing import to cloudkitty/common/config 46a54ad Implement Prometheus fetcher 0259fe4 Fix InfluxDB storage's "_point_to_dataframe_entry" method 1e60561 Make cloudkitty-processor run several workers 394177f Fix requirements.txt 19425bd Dropping the py35 testing f315499 Update tox to 2.0 b3c4f18 Update the default policy rule for /v1/storage/dataframes 8691aa6 Add storage backend documentation dc2509b Bootstrap the v2 API 4c69a86 [devstack] Setting [collect]/wait_periods to 0 by default 8ea14cb Fix HashMap field mapping comparison 6074028 Add bandit for security static analysis and fix potential security issues 4cfc06c Update the default metrics.yml file 7c7ff81 Update admin documentation for Prometheus collector df9442c Update master for stable/stein Diffstat (except docs and test files) ------------------------------------- .gitreview | 2 +- .zuul.yaml | 32 +- cloudkitty/api/app.py | 57 ++-- cloudkitty/api/root.py | 191 ++++-------- cloudkitty/api/v1/__init__.py | 57 ++++ cloudkitty/api/{ => v1}/config.py | 2 +- cloudkitty/api/v1/controllers/report.py | 8 +- cloudkitty/api/v1/controllers/storage.py | 9 +- cloudkitty/api/{ => v1}/hooks.py | 0 cloudkitty/api/v2/__init__.py | 64 ++++ cloudkitty/api/v2/base.py | 38 +++ cloudkitty/api/v2/scope/__init__.py | 26 ++ cloudkitty/api/v2/scope/state.py | 139 +++++++++ cloudkitty/api/v2/summary/__init__.py | 26 ++ cloudkitty/api/v2/summary/summary.py | 64 ++++ cloudkitty/api/v2/utils.py | 337 +++++++++++++++++++++ cloudkitty/cli/processor.py | 6 +- cloudkitty/collector/__init__.py | 12 +- cloudkitty/collector/gnocchi.py | 2 +- cloudkitty/collector/monasca.py | 23 +- cloudkitty/collector/prometheus.py | 78 +---- cloudkitty/common/config.py | 9 +- cloudkitty/common/defaults.py | 5 +- cloudkitty/common/policies/__init__.py | 24 +- cloudkitty/common/policies/v1/__init__.py | 0 cloudkitty/common/policies/{ => v1}/collector.py | 0 cloudkitty/common/policies/{ => v1}/info.py | 0 cloudkitty/common/policies/{ => v1}/rating.py | 0 cloudkitty/common/policies/{ => v1}/report.py | 0 cloudkitty/common/policies/{ => v1}/storage.py | 2 +- cloudkitty/common/policies/v2/__init__.py | 0 cloudkitty/common/policies/v2/scope.py | 37 +++ cloudkitty/common/policies/v2/summary.py | 30 ++ cloudkitty/common/prometheus_client.py | 69 +++++ cloudkitty/fetcher/prometheus.py | 142 +++++++++ cloudkitty/messaging.py | 2 +- cloudkitty/orchestrator.py | 230 ++++++++++---- cloudkitty/rating/hash/__init__.py | 3 +- cloudkitty/rating/pyscripts/__init__.py | 2 +- cloudkitty/rating/pyscripts/datamodels/script.py | 4 +- .../75c205f6f1a2_move_from_sha1_to_sha512.py | 43 +++ .../rating/pyscripts/db/sqlalchemy/models.py | 4 +- cloudkitty/storage/__init__.py | 11 +- cloudkitty/storage/v1/hybrid/backends/gnocchi.py | 2 +- cloudkitty/storage/v1/sqlalchemy/__init__.py | 2 +- cloudkitty/storage/v2/__init__.py | 24 +- cloudkitty/storage/v2/influx.py | 54 ++-- cloudkitty/storage_state/__init__.py | 42 +++ ...c50ed2c19204_update_storage_state_constraint.py | 46 +++ cloudkitty/storage_state/models.py | 17 +- .../gabbi/rating/pyscripts/gabbits/pyscripts.yaml | 12 +- cloudkitty/write_orchestrator.py | 4 +- devstack/README.rst | 4 +- devstack/plugin.sh | 3 + .../api-reference/{ => v1}/rating/hashmap.rst | 3 +- .../api-reference/{ => v1}/rating/pyscripts.rst | 3 +- .../v2/api_samples/scope/scope_get.json | 25 ++ .../v2/api_samples/summary/summary_get.json | 45 +++ .../api-reference/v2/scope/scope_parameters.yml | 94 ++++++ .../v2/summary/summary_parameters.yml | 79 +++++ etc/cloudkitty/api_paste.ini | 6 +- etc/cloudkitty/metrics.yml | 19 +- lower-constraints.txt | 7 +- .../add-prometheus-fetcher-be6082f70f279f0e.yaml | 10 + ...age-state-v2-api-endpoint-45a29d0b44e177b8.yaml | 6 + ...age-state-v2-api-endpoint-492d7092e85ed7b1.yaml | 6 + .../notes/added-v2-api-1ef829355c2feea4.yaml | 11 + ...etch-metrics-concurrently-dffffe346bd4900e.yaml | 7 + ...shmap-mapping-value-match-56570510203ce3e5.yaml | 7 + .../fix-v1-storage-groupby-e865d1315bd390cb.yaml | 6 + ...ject-id-monasca-collector-cb30ed073d36d40e.yaml | 5 + .../harden-dataframes-policy-7786286525e52dfb.yaml | 6 + ...ce-bandit-security-linter-592faa26f957a3dd.yaml | 6 + ...essor-run-several-workers-02597b0f77687ef3.yaml | 6 + ...work-prometheus-collector-f9f34a3792888dad.yaml | 2 +- releasenotes/source/index.rst | 1 + releasenotes/source/stein.rst | 6 + requirements.txt | 6 +- setup.cfg | 4 +- test-requirements.txt | 7 +- tox.ini | 15 +- 128 files changed, 4286 insertions(+), 877 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 616826f..7b10a7f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -32 +32,5 @@ voluptuous>=0.11.1 # BSD License -influxdb>=5.1.0,!=5.2.0 # MIT +influxdb>=5.1.0,!=5.2.0,!=5.2.1,!=5.2.2;python_version<'3.0' # MIT +influxdb>=5.1.0;python_version>='3.0' # MIT +Flask>=1.0.2 # BSD +Flask-RESTful>=0.3.5 # BSD +cotyledon>=1.5.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 1d028b4..67ff452 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -15,2 +15,3 @@ mock>=1.2 # BSD -sphinx>=1.6.2,!=1.6.6,!=1.6.7 # BSD -openstackdocstheme>=1.20.0 # Apache-2.0 +sphinx!=1.6.6,!=1.6.7,>=1.6.2,<2.0.0;python_version=='2.7' # BSD +sphinx!=1.6.6,!=1.6.7,!=2.1.0,>=1.6.2;python_version>='3.4' # BSD +openstackdocstheme>=1.30.0 # Apache-2.0 @@ -22,0 +24,2 @@ Pygments>=2.2.0 # BSD license +bandit>=1.1.0 # Apache-2.0 +os-api-ref>=1.0.0 # Apache-2.0