We are stoked to announce the release of: cloudkitty 11.0.0: Rating as a Service component for OpenStack This release is part of the train stable release series. The source is available from: https://opendev.org/openstack/cloudkitty Download the package from: https://pypi.org/project/cloudkitty Please report issues through: https://storyboard.openstack.org/#!/project/openstack/cloudkitty For more details, please see below. 11.0.0 ^^^^^^ New Features ************ * Added a v2 API endpoint allowing to push dataframes into the CloudKitty storage. This endpoint is available via a "POST" request on "/v2/dataframes". Admin privileges are required to use this endpoint. * A v2 storage driver for Elasticsearch has been added. It is marked as "EXPERIMENTAL" for now. * A "force_granularity" option has been added to the gnocchi collector's "extra_args". It allows to force a granularity to use when doing metric aggregations. If not specified or set to 0, the lowest available granularity will be used. * Added a v2 API endpoint allowing to retrieve dataframes from the CloudKitty storage. This endpoint is available via a "GET" request on "/v2/dataframes". Being the owner of the scope or having admin privileges are required to use this endpoint. Upgrade Notes ************* * CloudKitty is now aware of timezones, and the API supports iso8601 formatted timestamps. * Section names that had been deprecated in cloudkitty 9.0.0 have been removed in 11.0.0. These include "gnocchi_collector", "tenant_fetcher", "keystone_fetcher", "source_fetcher" and "hybrid_storage". Deprecation Notes ***************** * Since "eventlet" has been replaced with "futurist", the "[orchestrator]/max_greenthreads" option has been deprecated and replaced with "[orchestrator]/max_threads". Bug Fixes ********* * A validation issue causing the "GET /v2/summary" endpoint to systematically return a 400 error if any of the "begin" or "end" parameters was specified has been fixed. * An issue causing data not to be deleted from the storage backend when resetting a scope's state through the API has been fixed. * It is not required anymore to prefix the url of a resource with a "/" when using "cloudkitty.api.v2.utils.do_init". * Keystone authentication options are now registered with "keystoneauth1" in the monasca collector helper functions, which allows to use the "auth_section" option even when using the "source" fetcher. Other Notes *********** * Data frames/points are now internally represented as objects rather than dicts. * Since data frames are now represented as objects internally, transformers are not used anymore and have been completely removed from the codebase. * The "eventlet" library has been replaced with "futurist". Changes in cloudkitty 10.0.0..11.0.0 ------------------------------------ c8c7965 Add Elasticsearch v2 storage driver configuration documentation 0f6806c Allow missing '/' in api.v2.utils.do_init() 3cab85a Register keystone auth options with keystoneauth1 helper functions 72a7b3f Remove deprecated config section names cb858a9 Replace deprecated devstack authtoken function 78bcc65 Change logging for generic Exceptions d252f59 Add support for Elasticsearch to devstack plugin b7e8323 Add support for PDF doc generation 15f6118 Add an ElasticSearch v2 storage driver a81c01d Add a v2 API endpoint to retrieve DataFrame objects c5faed9 Use cloudkitty.tzutils.diff_seconds() in prometheus collector c6b0411 Use tzutils functions in gnocchi collector b16da52 Replace eventlet with futurist 3a47e50 Add a "force_granularity" option to gnocchi collector's extra_args 294bcb3 Convert timestamps to strs before passing them to gnocchiclient 54ede80 Store collect period in InfluxDB driver datapoints 492ec06 Remove transformers from the codebase e43bf5b Pass 'type' as metric_types in /v2/summary endpoints c841ee8 Fix validation of begin/end in GET /v2/summary endpoint a2afbb4 Update tempest jobs c55515b Fix malformed InfluxDB query (LIMIT and OFFSET inverted) a6a0e8d Added a roadmap to the developer documentation 4acc592 Add lower-constraints job 54c1b98 Update bandit version 6e8efde Add a v2 API endpoint to push DataFrame objects 7ea4dda Fix GET /v1/dataframes endpoint a0ec144 Fix RST markup in v2 API developer documentation 2f4acdc Add DataPoint/DataFrame objects 262de1d Add support for empty or missing "extra_args" in metrics config file e05c00e Introduce validation utils 426075d Updated the documentation's rst markup 58d8596 Fix StateManager.set_state() logic ecc4828 Fix call to storage.delete() in ScopeEndpoint RPC endpoint cb6a653 Use isoformat() instead of isotime() in InfluxDB storage driver 8869de5 Define new 'cloudkitty-tempest-full-ipv6-only' job in gate e552c38 Make cloudkitty timezone-aware 0c1546d Remove usage of unix timestamps 16b6fa5 Removing author identification in all files Diffstat (except docs and test files) ------------------------------------- .zuul.yaml | 95 +++- cloudkitty/api/app.py | 2 - cloudkitty/api/middleware.py | 2 - cloudkitty/api/v1/controllers/__init__.py | 2 - cloudkitty/api/v1/controllers/collector.py | 2 - cloudkitty/api/v1/controllers/info.py | 2 - cloudkitty/api/v1/controllers/rating.py | 2 - cloudkitty/api/v1/controllers/report.py | 7 +- cloudkitty/api/v1/controllers/storage.py | 42 +- cloudkitty/api/v1/datamodels/collector.py | 2 - cloudkitty/api/v1/datamodels/info.py | 2 - cloudkitty/api/v1/datamodels/rating.py | 2 - cloudkitty/api/v1/datamodels/report.py | 2 - cloudkitty/api/v1/datamodels/storage.py | 2 - cloudkitty/api/v1/hooks.py | 2 - cloudkitty/api/v1/types.py | 2 - cloudkitty/api/v2/__init__.py | 1 + cloudkitty/api/v2/dataframes/__init__.py | 26 + cloudkitty/api/v2/dataframes/dataframes.py | 90 ++++ cloudkitty/api/v2/scope/state.py | 19 +- cloudkitty/api/v2/summary/summary.py | 14 +- cloudkitty/api/v2/utils.py | 18 +- cloudkitty/backend/__init__.py | 2 - cloudkitty/backend/file.py | 2 - cloudkitty/cli/dbsync.py | 2 - cloudkitty/cli/processor.py | 2 - cloudkitty/cli/storage.py | 2 - cloudkitty/cli/writer.py | 2 - cloudkitty/collector/__init__.py | 71 +-- cloudkitty/collector/gnocchi.py | 81 ++- cloudkitty/collector/monasca.py | 48 +- cloudkitty/collector/prometheus.py | 23 +- cloudkitty/common/config.py | 6 +- cloudkitty/common/db/alembic/env.py | 2 - cloudkitty/common/db/alembic/migration.py | 2 - cloudkitty/common/db/models.py | 2 - cloudkitty/common/policies/__init__.py | 2 + cloudkitty/common/policies/v2/dataframes.py | 37 ++ cloudkitty/config.py | 2 - cloudkitty/dataframe.py | 279 ++++++++++ cloudkitty/db/__init__.py | 2 - cloudkitty/db/api.py | 2 - cloudkitty/db/sqlalchemy/alembic/env.py | 2 - cloudkitty/db/sqlalchemy/api.py | 2 - cloudkitty/db/sqlalchemy/migration.py | 2 - cloudkitty/db/sqlalchemy/models.py | 2 - cloudkitty/extension_manager.py | 2 - cloudkitty/fetcher/__init__.py | 6 +- cloudkitty/fetcher/gnocchi.py | 2 - cloudkitty/fetcher/keystone.py | 28 +- cloudkitty/fetcher/source.py | 6 - cloudkitty/i18n.py | 2 - cloudkitty/json_utils.py | 4 +- cloudkitty/orchestrator.py | 114 ++-- cloudkitty/rating/__init__.py | 2 - cloudkitty/rating/hash/__init__.py | 39 +- cloudkitty/rating/hash/controllers/field.py | 2 - cloudkitty/rating/hash/controllers/group.py | 2 - cloudkitty/rating/hash/controllers/mapping.py | 2 - cloudkitty/rating/hash/controllers/root.py | 2 - cloudkitty/rating/hash/controllers/service.py | 2 - cloudkitty/rating/hash/controllers/threshold.py | 2 - cloudkitty/rating/hash/datamodels/field.py | 2 - cloudkitty/rating/hash/datamodels/group.py | 2 - cloudkitty/rating/hash/datamodels/mapping.py | 2 - cloudkitty/rating/hash/datamodels/service.py | 2 - cloudkitty/rating/hash/datamodels/threshold.py | 2 - cloudkitty/rating/hash/db/api.py | 2 - .../rating/hash/db/sqlalchemy/alembic/env.py | 2 - .../models/f8c799db4aa0_fix_unnamed_constraints.py | 2 - cloudkitty/rating/hash/db/sqlalchemy/api.py | 2 - cloudkitty/rating/hash/db/sqlalchemy/migration.py | 2 - cloudkitty/rating/hash/db/sqlalchemy/models.py | 2 - cloudkitty/rating/noop.py | 10 - cloudkitty/rating/pyscripts/__init__.py | 4 +- cloudkitty/rating/pyscripts/controllers/root.py | 2 - cloudkitty/rating/pyscripts/controllers/script.py | 2 - cloudkitty/rating/pyscripts/datamodels/script.py | 2 - cloudkitty/rating/pyscripts/db/api.py | 2 - .../rating/pyscripts/db/sqlalchemy/alembic/env.py | 2 - cloudkitty/rating/pyscripts/db/sqlalchemy/api.py | 2 - .../rating/pyscripts/db/sqlalchemy/migration.py | 2 - .../rating/pyscripts/db/sqlalchemy/models.py | 2 - cloudkitty/service.py | 2 - cloudkitty/state.py | 2 - cloudkitty/storage/__init__.py | 47 +- cloudkitty/storage/v1/__init__.py | 33 +- cloudkitty/storage/v1/hybrid/__init__.py | 8 +- cloudkitty/storage/v1/hybrid/alembic/env.py | 2 - .../versions/03da4bb002b9_initial_revision.py | 2 - cloudkitty/storage/v1/hybrid/backends/__init__.py | 2 - cloudkitty/storage/v1/hybrid/backends/gnocchi.py | 2 - cloudkitty/storage/v1/hybrid/migration.py | 2 - cloudkitty/storage/v1/hybrid/models.py | 2 - cloudkitty/storage/v1/sqlalchemy/__init__.py | 12 +- cloudkitty/storage/v1/sqlalchemy/alembic/env.py | 2 - cloudkitty/storage/v1/sqlalchemy/migration.py | 2 - cloudkitty/storage/v1/sqlalchemy/models.py | 9 +- cloudkitty/storage/v2/__init__.py | 35 +- cloudkitty/storage/v2/elasticsearch/__init__.py | 204 +++++++ cloudkitty/storage/v2/elasticsearch/client.py | 399 ++++++++++++++ cloudkitty/storage/v2/elasticsearch/exceptions.py | 32 ++ cloudkitty/storage/v2/influx.py | 142 +++-- cloudkitty/storage_state/__init__.py | 47 +- cloudkitty/storage_state/alembic/env.py | 2 - cloudkitty/storage_state/migration.py | 2 - cloudkitty/storage_state/models.py | 2 - cloudkitty/transformer/__init__.py | 71 --- cloudkitty/transformer/format.py | 43 -- cloudkitty/tzutils.py | 166 ++++++ cloudkitty/utils.py | 41 +- cloudkitty/validation_utils.py | 98 ++++ cloudkitty/write_orchestrator.py | 2 - cloudkitty/writer/__init__.py | 2 - cloudkitty/writer/csv_base.py | 2 - cloudkitty/writer/csv_map.py | 2 - cloudkitty/writer/osrf.py | 2 - contrib/ci/csv_writer.py | 2 - devstack/plugin.sh | 39 +- devstack/settings | 4 + .../v2/api_samples/dataframes/dataframes_get.json | 78 +++ .../v2/api_samples/dataframes/dataframes_post.json | 96 ++++ .../api-reference/v2/dataframes/dataframes.inc | 85 +++ .../v2/dataframes/dataframes_parameters.yml | 55 ++ .../api-reference/v2/dataframes/http_status.yml | 1 + lower-constraints.txt | 4 +- ...taframe-datapoint-objects-a5a4ac3db5289cb6.yaml | 5 + ...ataframes-v2-api-endpoint-601825c344ba0e2d.yaml | 6 + ...-driver-for-elasticsearch-ec41cbb7849e82d3.yaml | 5 + ...orced-granularity-gnocchi-d52e988194197248.yaml | 7 + ...end-validation-v2-summary-52401fb47ef9b5d6.yaml | 6 + ...scope-state-reset-filters-0a1f5ea503bd32a1.yaml | 5 + .../fix-url-building-do-init-7c952afaf6d909cd.yaml | 5 + ...ataframes-v2-api-endpoint-3a4625c6008a5fca.yaml | 7 + ...cloudkitty-timezone-aware-2b65edc42e913d6c.yaml | 5 + ...nctions-monasca-collector-1a539fc8c23e9dbc.yaml | 6 + ...ated-config-section-names-9a125b1af0932c08.yaml | 6 + .../remove-transformers-8d9949ed3088b055.yaml | 5 + ...ce-eventlet-with-futurist-60f1fe6474a5efcf.yaml | 10 + requirements.txt | 2 +- setup.cfg | 4 +- test-requirements.txt | 2 +- tox.ini | 19 +- 205 files changed, 5109 insertions(+), 1491 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 7b10a7f..24b324c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6 +5,0 @@ alembic>=0.8.0 # MIT -eventlet>=0.18.2,!=0.18.3,!=0.20.1,!=0.21.0,!=0.23.0 # MIT @@ -36,0 +36 @@ cotyledon>=1.5.0 # Apache-2.0 +futurist>=1.6.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 67ff452..4d8c22f 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -24 +24 @@ Pygments>=2.2.0 # BSD license -bandit>=1.1.0 # Apache-2.0 +bandit>=1.6.0 # Apache-2.0