We are ecstatic to announce the release of: gnocchi 3.1.0: Metric as a Service Download the package from: https://tarballs.openstack.org/gnocchi/ For more details, please see below. 3.1.0 ^^^^^ New Features ************ * Allow to search for values in metrics by using one or more granularities. * A new archive policy named *bool* is provided by default. It provides a cheap and easy way to store boolean measures (0 and 1). * The REST API authentication mechanism is now pluggable. You can write your own plugin to specify how segregation and policy should be enforced. * Add support to backfill timestamps with missing points in a subset of timeseries when computing aggregation across multiple metrics. User can specify *fill* value with either a float or *null* value. A granularity must be specified in addition to *fill*. * When sending measures in batch for resources, it is now possible to pass *create_metric=true* to the query parameters so missing metrics are created. This only works if an archive policy rule matching those named metrics matches. * Add *resample* parameter to support resampling stored time-series to another granularity not necessarily in existing archive policy. If both resampling and reaggregation parameters are specified, resampling will occur prior to reaggregation. * The *gnocchi-config-generator* program can now generates a default configuration file, usable as a template for custom tweaking. * A healthcheck endpoint is provided by default at /healthcheck. It leverages oslo_middleware healthcheck middleware. It allows to retrieve information about the health of the API service. * gnocchi-metricd now uses the cotyledon/oslo.config helper to handle configuration file reloading. You can dynamically change the number of workers by changing the configuration file and sending SIGHUP to the metricd master process. * When updating a resource attribute, it's now possible to pass the option 'fill' for each attribute to fill existing resources. * required=True is now supported when updating resource type. This requires the option 'fill' to be set. * New storage driver for AWS S3. This new driver works in the same way that the Swift driver, expect that it leverages the Amazon Web Services S3 object storage API. * The storage of new measures that ought to be processed by *metricd* can now be stored using different storage drivers. By default, the driver used is still the regular storage driver configured. See the *[incoming]* section in the configuration file. Known Issues ************ * The conversion mechanism provided by the API to convert non-UUID resource id to UUID is now also based on the user creating/accessing the resource. This makes sure that the conversion generates a unique UUID for the user and that several users can use the same string as *original_resource_id*. Upgrade Notes ************* * The new *auth_type* option specifies which authentication system to use for the REST API. Its default is still *noauth*. * Ceph driver has moved the storage of measures metadata from xattr to omap API. Already created measures are migrated during gnocchi- upgrade run. * The *auth_type* option has a new default value set to "basic". This mode does not do any segregation and uses the standard HTTP *Authorization* header for authentication. The old "noauth" authentication mechanism based on the Keystone headers (*X-User-Id*, *X-Creator-Id* and *X-Roles*) and the Keystone segregation rules, which was the default up to Gnocchi 3.0, is still available. * The api.pecan_debug has been removed. * Since *original_resource_id* is now unique per creator, that means users cannot refer to resource by using the *original_resource_id* if the resource was not created by them. Deprecation Notes ***************** * The *created_by_user_id* and *created_by_project_id* field are now deprecated and being merged into a unique *creator* field. The old fields are still returned and managed by the API for now. Bug Fixes ********* * '/' in resource id and metric name have been accepted by mistake, because they can be POSTed but not GETed/PATCHed/DELETEd. Now this char is forbidden in resource id and metric name, REST api will return 400 if it presents. Metric name and resource id already present with a '/' have their '/' replaced by '_'. Other Notes *********** * ceph driver now uses the rados async api to retrieve measurements to process in parallel. * The default archive policies "low" and "medium" are now storing less data than they used to be. They are only using respectively 1 and 2 definition of archiving policy, which speeds up by 66% and 33% their computing speed. * Gnocchi now leverages microseconds timestamps available since MySQL 5.6.4, meaning it is now the minimum required version of MySQL. * The *noauth* authentication mode now requires that the *X-User-Id* and/or *X-Project-Id* to be present. * The default archive policies list does not contain the 95pct and median aggregation methods by default. These are the least used methods and should make gnocchi-metricd faster by more than 25% in the default scenario. Changes in gnocchi 3.0.0..3.1.0 ------------------------------- 6e41c34 tools: make measure injector works without gnocchiclient 5a8d3a4 paginate ceph report generation 2c0f2b6 indexer: make sure original_resource_id is never NULL ad4b851 rest: string → UUID conversion for resource.id to be unique per user 34b7d30 tests: specify columns to compare in resource list 35ae345 modernise gabbi tests 7dd8a32 supporting disabling reporting process dcb2a6a fix live gabbi test cd8d85e ceph: set return limit 77d02ef tests: do not create a resource with a custom resource type in 2.2 e066d53 tests: factorize measure generation to speed up test fc0e38c mysql: retry on table changes 0d268f5 Revert "indexer: catch another mysql exception" 7d602de carbonara: numpy 1.9 have some bug with ? for bool e87710d tests: use GNU sed where needed e011cdd storage: Read incoming config correctly 2bc624f noauth: force user to be provided 60ae8fa cleanup noauth path c2c8cef doc: move devstack at the end 05ad548 tests: clean upgrade variable 5b974f3 Required some configuration options 5f58016 allow required=True when patching resource type 0a1b374 rest,indexer: handle ResourceUUID conversion in the REST API f27ec58 sqlalchemy: use a list rather than if/elif to convert type in queries a7bc20c Revert "indexer: fix datetime with mysql >= 5.7.17" f97a586 Update Pandas requirements to 0.18 451dbda Add a release note about storage/incoming split ac7ae99 drop gnocchi_<id>_container object 2361401 indexer: catch another mysql exception c4027ce stop validating aggregation on init 2e29652 increase timeout 24d2210 Remove redundant comma 9f3de46 carbonara: resample() with scipy/numpy d2435dc Fix error message 417dd91 carbonara: add tests for each aggregation 34cc642 carbonara: add resample() benchmark 3b99004 utils: use proper timedelta conversion 67807cd tests: increase benchmark timeout for Carbonara e3597ad sqlalchemy: fix compat search on created_by_project_id fdefb41 serialise: simplify array format ed028da carbonara: add merge() benchmark c7b4f40 carbonara: don't use groupby for split ce79e4b Remove non-existent -all suffix in tox targets a5e9cd5 Remove py{27,35} from valid tox targets e1219e2 indexer: fix datetime with mysql >= 5.7.17 71e437b carbonara: Add benchmark for split() c3ab47d carbonara: use numpy for unserialization 9cc3485 carbonara: use numpy for serialization 5596bde carbonara: Don't use clean_ts() ddf69cf carbonara: handle timestamps from struct with numpy eaa1681 carbonara: remove a pandas.iteritems() ff5bbc4 carbonara: prepare datetime for pandas.to_datetime() 7ceed2e add metricd tester for profiling 5005d59 rest: returns orignal resource id 9a4f414 indexer: fix migration script "no_more_slash" b39eead Remove workaround to upgrade from 2.2.0 b817a8b utils: allow ResourceUUID to convert UUID 40b6492 cleanup invalid upgrade errors b8ef321 archive_policy: provide a boolean storage archive policy by default 2aae94a storage: add more debug information to trace behaviour db02e6b opts: list entry points with pkg_resources rather than stevedore ea9ae16 mysql: fix timestamp upgrade fece429 rest: reject / as resource id and metric name b0b8c1a tests: Fix upgrade script ab4d74c ceph: enhance the documentation 9da4e07 Introduce "basic" authentication mechanism 5633edd don't override logging when loading alembic module c03958b upgrade: fix OS_AUTH_TYPE variable name 63e1cd9 Add sem-ver flag so pbr generates correct version 546bd69 ceph: fix data compression when oldest_mutable_timestamp == next(key) 410b164 Add gnocchi-config-generator 1aaf8be Remove broken script e534ebb Fix a typo in estimated sizing per metric under archive policies 39265b4 Merge project and user id in a creator field 020dce5 test: remove unnused method 054b306 Create a base exception for resource type error d55f6f0 ceph: Workaround for oslo.config interpolation bug 178a349 indexer: fix resource type update fc10dba Tests to confirm resources are searchable. da0498e rest: introduce auth_helper to filter resources 98a4961 rest: add auth_mode to pick authentication mode 37db178 Move default policy.json away from etc/ 6964ac2 Ship api-paste.ini out of etc/ 3b1415e rest: make sure 409 is returned when double creating resource with non-UUID c751087 tools: import a small tools to compute size of archive policies 246b787 archive_policy: lighten the default archive policies 6449ed8 run-tests: use case rather than if/elif/else 72fdba7 metricd: move metricd options in metricd group 00d842c storage: remove temporary incoming setup 9376c00 Introduce new storage groups for storage 2cb85a7 devstack: prepare ceph keyring before using it ca43bec Adjust testr group_regex to not group on 'prefix' 9200ae3 storage: split s3 driver 3632911 Fix expected content-type and move CORS tests to gabbi 1dcd2fd [doc] Note lack of constraints is a choice c5dd0d9 gabbi: remove unused variable 4a7a3cc rest: catch create_metric duplicate cd82f8d Revert "add mysql minimum version check" 57bbee6 add mysql minimum version check 85a39e3 All granularity input should be parsed as timespan 5edc39a fill series when aggregating cross metrics 14c58fc carbonara: log a message and do not fail if a file is missing fb31248 Enable H904 hacking check 05c2656 storage: split swift driver 3385186 doc: add a page talking about collectd support 7cff2f7 Enable oslo_middleware healthcheck middleware by default bbe2d37 use datetime when defining series range 9d0192a storage: split ceph driver ecbb5fd storage: split file driver f5794af fix oslo.db 4.15.0 breakage 4bdb88b rest: remove user_id and project_id from metric schema fcf08aa api: use egg entry_point rather than code path dbcfa9c doc: Add reference to gnocchi-nagios tool b6d08c2 fix logging... 77c0aa0 rest: don't ignore measures of created metrics 049116d Modify variable's using method in Log Messages 9055624 Fix two trivial docs errors in architecture.rst e413a93 storage: split the storage interface 14a7cba config: only include oslo.middleware options that are shipped 0b970f0 doc: remove unused links db38938 file: remove tmp configuration 5b833bb storage: remove _pending_measures_to_process_count() 8612ab8 storage: split process_new_measures() b48483a Remove 95pct and median from default archive policies 38c2d32 rest: wait for the thread pool executor result 00b349e indexer: list_metric(), skip sql if names is empty e6d3cee rest: Don't use private webob API 4e5b18d rest: fix batching error handling 516e681 rest: don't fail if the batch measure is not a dict db7ea6a Replace retry with tenacity.retry e2732af ceph: make computed measures read async ff53ff8 utils: do not retry on any exception c5779e4 Remove usage of deprecated operatorPrecedence and remove duplicate operators 6959709 drop pytimeparse requirement 21966a7 support pandas.to_timedelta fd8b81b Bump hacking to 0.12 219fd0d rest: use flatten_dict_to_keypairs instead of recursive_keypairs 0c66859 carbonara: add support for Gnocchi v2 measures format 68a2a35 support resampling on aggregation endpoint 7418017 support resampling abfa908 Use tenacity.TryAgain rather than a own-defined exception f9b19ad rest: allow to create missing metrics when sending measures in batch 04dde36 Interpolate strings using logging own methods 04917de metricd: retry slowly coordination connection failure e4d3b57 rest: don't use is_body_seekable ad5b4fc Allow timespan to be floating values 021a041 rest: empty search query in resource search a8c170d ceph: move out of xattr completely b97a91a support consistent timestamps c95b98f Revert "drop non-I/O threading in upgrade" dd7cda9 rest: register default JSON converter using pecan.jsonify 1e8b6de Don't hang with wsgiref 234b0f0 Add missing granularity to base storage class e1ab93c json: remove outdated comment 2ba63d4 Update doc because default services are all being added to settings 4637aeb accommodate new oslo.config 06fcd34 devstack: stop all gnocchi services, not just api b04b370 Fix incorrect EXTRA_FLAVOR in plugin.sh e7d8faf rest: using ujson to deserialize 839316a doc: add s3 to the list of Carbonara based drivers 37d8fea Fix typo in release note file 6430afb Unify timestamp parsing 537cd29 rest: fix Epoch timestamp parsing 34beae6 test: rewrite test_post_unix_timestamp in Gabbi 4bc5170 Remove pecan_debug option 0e453f2 test: allow to pass OS_DEBUG 16a05de Remove unused requests dependency 9a95873 carbonara: fix SplitKey with datetime greater than 32bits value 201dc6d Revert "Remove the file named MANIFEST.in" 2ac7a20 Add helper for utcnow to epoch nano 2b1ed8d Add http_proxy_to_wsgi to api-paste 510d799 Use Cotyledon oslo config glue 6baf05b Add a S3 based storage driver 5186bcc tox: only install all storage drivers in py-$index or py-#index-all 50c6141 Remove Python 3.4 support 93bb23d doc,tests: fix reaggregate/reaggregation mispelling b36b66e Fix oslo.log minimum requirement bb3cd40 Remove the file named MANIFEST.in 811a8fd Stop adding ServiceAvailable group option bbc821d sqlalchemy: use precise DATETIME rather than DECIMAL in MySQL bbbe67e enable threading of batched writes 83f0299 switch to stop validating batch individually c34b246 Don't mock ceph 0a06089 upgrade tests: don't patch gnocchi 3.0 1157cac tests: Cover resource-type modification 0198b5b Use xx=None instead of xx=[] to initialize the default value 35c4a08 Fix some gabbi tests 15c0c71 Add STRICT_RESPONSE_HEADERS check to gabbi tests 984a976 Add upgrade targets for Gnocchi 3.0 e26bf90 tox: shorter envdir name for upgrade target 21845a8 remove the pandas module in test test_carbonara.py 07034a1 doc: include stable/3.0 release notes b5106eb Fix typos in tests/gabbi/gabbits/resource.yaml b5ad717 Add granularity in searching for values in metrics b8c0c23 Modify api startup parameters in devstack plugin 723557f Replace retrying with tenacity 31cdcb5 ceph: fix python3 issue bd14276 Add simple upgrade tests 7a37e4e compute new first_block_timestamp once 3a8f315 Allow to pass search filter in the query string Diffstat (except docs and test files) ------------------------------------- .testr.conf | 2 +- bindep.txt | 2 + devstack/gate/post_test_hook.sh | 1 - devstack/plugin.sh | 31 +- devstack/settings | 1 - etc/gnocchi/api-paste.ini | 31 -- etc/gnocchi/gnocchi-config-generator.conf | 9 - etc/gnocchi/policy.json | 42 -- gnocchi/aggregates/moving_stats.py | 9 +- gnocchi/archive_policy.py | 32 +- gnocchi/carbonara.py | 482 +++++++++++++------ gnocchi/cli.py | 92 +++- gnocchi/genconfig.py | 13 +- gnocchi/gendoc.py | 8 +- gnocchi/gnocchi-config-generator.conf | 11 + gnocchi/indexer/__init__.py | 12 +- gnocchi/indexer/alembic/alembic.ini | 34 -- gnocchi/indexer/alembic/env.py | 5 - .../1e1a63d3d186_original_resource_id_not_null.py | 39 ++ .../397987e38570_no_more_slash_and_reencode.py | 185 +++++++ .../5c4f93e5bb4_mysql_float_to_timestamp.py | 77 +++ .../aba5a217ca9b_merge_created_in_creator.py | 53 ++ gnocchi/indexer/sqlalchemy.py | 142 ++++-- gnocchi/indexer/sqlalchemy_base.py | 78 ++- gnocchi/indexer/sqlalchemy_extension.py | 26 +- gnocchi/json.py | 14 +- gnocchi/opts.py | 82 +++- gnocchi/resource_type.py | 71 ++- gnocchi/rest/__init__.py | 532 +++++++++++++-------- gnocchi/rest/api-paste.ini | 46 ++ gnocchi/rest/app.py | 29 +- gnocchi/rest/auth_helper.py | 125 +++++ gnocchi/rest/policy.json | 42 ++ gnocchi/service.py | 18 +- gnocchi/statsd.py | 41 +- gnocchi/storage/__init__.py | 79 +-- gnocchi/storage/_carbonara.py | 410 ++++++++-------- gnocchi/storage/ceph.py | 286 +++-------- gnocchi/storage/common/__init__.py | 0 gnocchi/storage/common/ceph.py | 66 +++ gnocchi/storage/common/s3.py | 81 ++++ gnocchi/storage/common/swift.py | 63 +++ gnocchi/storage/file.py | 133 +----- gnocchi/storage/incoming/__init__.py | 58 +++ gnocchi/storage/incoming/_carbonara.py | 85 ++++ gnocchi/storage/incoming/ceph.py | 209 ++++++++ gnocchi/storage/incoming/file.py | 124 +++++ gnocchi/storage/incoming/s3.py | 151 ++++++ gnocchi/storage/incoming/swift.py | 98 ++++ gnocchi/storage/s3.py | 200 ++++++++ gnocchi/storage/swift.py | 118 +---- gnocchi/tempest/config.py | 13 +- gnocchi/tempest/plugin.py | 13 +- .../gabbi/gabbits/metric-timestamp-format.yaml | 60 +++ gnocchi/utils.py | 169 +++++-- ...add-parameter-granularity-7f22c677dc1b1238.yaml | 4 + .../archive_policy_bool-9313cae7122c4a2f.yaml | 5 + .../notes/auth_type_option-c335b219afba5569.yaml | 5 + .../auth_type_pluggable-76a3c73cac8eec6a.yaml | 5 + ...ackfill-cross-aggregation-2de54c7c30b2eb67.yaml | 6 + ...e_measures_create_metrics-f73790a8475ad628.yaml | 5 + releasenotes/notes/ceph-omap-34e069dfb3df764d.yaml | 5 + .../notes/ceph-read-async-ca2f7512c6842adb.yaml | 4 + .../notes/creator_field-6b715c917f6afc93.yaml | 6 + .../notes/dynamic-resampling-b5e545b1485c152f.yaml | 6 + .../notes/forbid-slash-b3ec2bc77cc34b49.yaml | 7 + .../gnocchi_config_generator-0fc337ba8e3afd5f.yaml | 5 + .../healthcheck-middleware-81c2f0d02ebdb5cc.yaml | 5 + ...-default-archive-policies-455561c027edf4ad.yaml | 5 + .../mysql_precise_datetime-57f868f3f42302e2.yaml | 4 + .../noauth-force-headers-dda926ce83f810e8.yaml | 5 + .../noauth-keystone-compat-e8f760591d593f07.yaml | 9 + .../pecan-debug-removed-1a9dbc4a0a6ad581.yaml | 3 + releasenotes/notes/reloading-734a639a667c93ee.yaml | 6 + ...fault-aggregation-methods-2f5ec059855e17f9.yaml | 5 + ...-type-required-attributes-f446c220d54c8eb7.yaml | 6 + releasenotes/notes/s3_driver-4b30122bdbe0385d.yaml | 5 + .../notes/storage-incoming-586b3e81de8deb4f.yaml | 6 + .../notes/uuid5-change-8a8c467d2b2d4c85.yaml | 12 + requirements.txt | 18 +- setup.cfg | 30 +- tools/gnocchi-archive-policy-size.py | 49 ++ tools/measures_injector.py | 26 +- tox.ini | 83 +++- 128 files changed, 6499 insertions(+), 2617 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 23cf70d..ea3d96a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2 +2 @@ pbr -numpy +numpy>=1.9.0 @@ -5 +5 @@ oslo.config>=2.6.0 -oslo.log>=1.0.0 +oslo.log>=2.3.0 @@ -8,3 +8,4 @@ oslo.serialization>=1.4.0 -oslo.utils>=3.3.0 -oslo.middleware>=3.11.0 -pandas>=0.17.0 +oslo.utils>=3.18.0 +oslo.middleware>=3.22.0 +pandas>=0.18.0 +scipy>=0.18.1 # BSD @@ -12 +12,0 @@ pecan>=0.9 -pytimeparse>=1.1.5 @@ -15,2 +15 @@ jsonpatch -cotyledon>=1.2.2 -requests +cotyledon>=1.5.0 @@ -18,0 +18 @@ stevedore +ujson @@ -22 +22 @@ trollius; python_version < '3.4' -retrying +tenacity>=3.1.0 # Apache-2.0