[release-announce] [telemetry] gnocchi 3.1.0
no-reply at openstack.org
no-reply at openstack.org
Thu Feb 2 18:52:10 UTC 2017
We are ecstatic to announce the release of:
gnocchi 3.1.0: Metric as a Service
Download the package from:
https://tarballs.openstack.org/gnocchi/
For more details, please see below.
3.1.0
^^^^^
New Features
************
* Allow to search for values in metrics by using one or more
granularities.
* A new archive policy named *bool* is provided by default. It
provides a cheap and easy way to store boolean measures (0 and 1).
* The REST API authentication mechanism is now pluggable. You can
write your own plugin to specify how segregation and policy should
be enforced.
* Add support to backfill timestamps with missing points in a subset
of timeseries when computing aggregation across multiple metrics.
User can specify *fill* value with either a float or *null* value. A
granularity must be specified in addition to *fill*.
* When sending measures in batch for resources, it is now possible
to pass *create_metric=true* to the query parameters so missing
metrics are created. This only works if an archive policy rule
matching those named metrics matches.
* Add *resample* parameter to support resampling stored time-series
to another granularity not necessarily in existing archive policy.
If both resampling and reaggregation parameters are specified,
resampling will occur prior to reaggregation.
* The *gnocchi-config-generator* program can now generates a default
configuration file, usable as a template for custom tweaking.
* A healthcheck endpoint is provided by default at /healthcheck. It
leverages oslo_middleware healthcheck middleware. It allows to
retrieve information about the health of the API service.
* gnocchi-metricd now uses the cotyledon/oslo.config helper to
handle configuration file reloading. You can dynamically change the
number of workers by changing the configuration file and sending
SIGHUP to the metricd master process.
* When updating a resource attribute, it's now possible to pass the
option 'fill' for each attribute to fill existing resources.
* required=True is now supported when updating resource type. This
requires the option 'fill' to be set.
* New storage driver for AWS S3. This new driver works in the same
way that the Swift driver, expect that it leverages the Amazon Web
Services S3 object storage API.
* The storage of new measures that ought to be processed by
*metricd* can now be stored using different storage drivers. By
default, the driver used is still the regular storage driver
configured. See the *[incoming]* section in the configuration file.
Known Issues
************
* The conversion mechanism provided by the API to convert non-UUID
resource id to UUID is now also based on the user creating/accessing
the resource. This makes sure that the conversion generates a unique
UUID for the user and that several users can use the same string as
*original_resource_id*.
Upgrade Notes
*************
* The new *auth_type* option specifies which authentication system
to use for the REST API. Its default is still *noauth*.
* Ceph driver has moved the storage of measures metadata from xattr
to omap API. Already created measures are migrated during gnocchi-
upgrade run.
* The *auth_type* option has a new default value set to "basic".
This mode does not do any segregation and uses the standard HTTP
*Authorization* header for authentication. The old "noauth"
authentication mechanism based on the Keystone headers (*X-User-Id*,
*X-Creator-Id* and *X-Roles*) and the Keystone segregation rules,
which was the default up to Gnocchi 3.0, is still available.
* The api.pecan_debug has been removed.
* Since *original_resource_id* is now unique per creator, that means
users cannot refer to resource by using the *original_resource_id*
if the resource was not created by them.
Deprecation Notes
*****************
* The *created_by_user_id* and *created_by_project_id* field are now
deprecated and being merged into a unique *creator* field. The old
fields are still returned and managed by the API for now.
Bug Fixes
*********
* '/' in resource id and metric name have been accepted by mistake,
because they can be POSTed but not GETed/PATCHed/DELETEd. Now this
char is forbidden in resource id and metric name, REST api will
return 400 if it presents. Metric name and resource id already
present with a '/' have their '/' replaced by '_'.
Other Notes
***********
* ceph driver now uses the rados async api to retrieve measurements
to process in parallel.
* The default archive policies "low" and "medium" are now storing
less data than they used to be. They are only using respectively 1
and 2 definition of archiving policy, which speeds up by 66% and 33%
their computing speed.
* Gnocchi now leverages microseconds timestamps available since
MySQL 5.6.4, meaning it is now the minimum required version of
MySQL.
* The *noauth* authentication mode now requires that the *X-User-Id*
and/or *X-Project-Id* to be present.
* The default archive policies list does not contain the 95pct and
median aggregation methods by default. These are the least used
methods and should make gnocchi-metricd faster by more than 25% in
the default scenario.
Changes in gnocchi 3.0.0..3.1.0
-------------------------------
6e41c34 tools: make measure injector works without gnocchiclient
5a8d3a4 paginate ceph report generation
2c0f2b6 indexer: make sure original_resource_id is never NULL
ad4b851 rest: string → UUID conversion for resource.id to be unique per user
34b7d30 tests: specify columns to compare in resource list
35ae345 modernise gabbi tests
7dd8a32 supporting disabling reporting process
dcb2a6a fix live gabbi test
cd8d85e ceph: set return limit
77d02ef tests: do not create a resource with a custom resource type in 2.2
e066d53 tests: factorize measure generation to speed up test
fc0e38c mysql: retry on table changes
0d268f5 Revert "indexer: catch another mysql exception"
7d602de carbonara: numpy 1.9 have some bug with ? for bool
e87710d tests: use GNU sed where needed
e011cdd storage: Read incoming config correctly
2bc624f noauth: force user to be provided
60ae8fa cleanup noauth path
c2c8cef doc: move devstack at the end
05ad548 tests: clean upgrade variable
5b974f3 Required some configuration options
5f58016 allow required=True when patching resource type
0a1b374 rest,indexer: handle ResourceUUID conversion in the REST API
f27ec58 sqlalchemy: use a list rather than if/elif to convert type in queries
a7bc20c Revert "indexer: fix datetime with mysql >= 5.7.17"
f97a586 Update Pandas requirements to 0.18
451dbda Add a release note about storage/incoming split
ac7ae99 drop gnocchi_<id>_container object
2361401 indexer: catch another mysql exception
c4027ce stop validating aggregation on init
2e29652 increase timeout
24d2210 Remove redundant comma
9f3de46 carbonara: resample() with scipy/numpy
d2435dc Fix error message
417dd91 carbonara: add tests for each aggregation
34cc642 carbonara: add resample() benchmark
3b99004 utils: use proper timedelta conversion
67807cd tests: increase benchmark timeout for Carbonara
e3597ad sqlalchemy: fix compat search on created_by_project_id
fdefb41 serialise: simplify array format
ed028da carbonara: add merge() benchmark
c7b4f40 carbonara: don't use groupby for split
ce79e4b Remove non-existent -all suffix in tox targets
a5e9cd5 Remove py{27,35} from valid tox targets
e1219e2 indexer: fix datetime with mysql >= 5.7.17
71e437b carbonara: Add benchmark for split()
c3ab47d carbonara: use numpy for unserialization
9cc3485 carbonara: use numpy for serialization
5596bde carbonara: Don't use clean_ts()
ddf69cf carbonara: handle timestamps from struct with numpy
eaa1681 carbonara: remove a pandas.iteritems()
ff5bbc4 carbonara: prepare datetime for pandas.to_datetime()
7ceed2e add metricd tester for profiling
5005d59 rest: returns orignal resource id
9a4f414 indexer: fix migration script "no_more_slash"
b39eead Remove workaround to upgrade from 2.2.0
b817a8b utils: allow ResourceUUID to convert UUID
40b6492 cleanup invalid upgrade errors
b8ef321 archive_policy: provide a boolean storage archive policy by default
2aae94a storage: add more debug information to trace behaviour
db02e6b opts: list entry points with pkg_resources rather than stevedore
ea9ae16 mysql: fix timestamp upgrade
fece429 rest: reject / as resource id and metric name
b0b8c1a tests: Fix upgrade script
ab4d74c ceph: enhance the documentation
9da4e07 Introduce "basic" authentication mechanism
5633edd don't override logging when loading alembic module
c03958b upgrade: fix OS_AUTH_TYPE variable name
63e1cd9 Add sem-ver flag so pbr generates correct version
546bd69 ceph: fix data compression when oldest_mutable_timestamp == next(key)
410b164 Add gnocchi-config-generator
1aaf8be Remove broken script
e534ebb Fix a typo in estimated sizing per metric under archive policies
39265b4 Merge project and user id in a creator field
020dce5 test: remove unnused method
054b306 Create a base exception for resource type error
d55f6f0 ceph: Workaround for oslo.config interpolation bug
178a349 indexer: fix resource type update
fc10dba Tests to confirm resources are searchable.
da0498e rest: introduce auth_helper to filter resources
98a4961 rest: add auth_mode to pick authentication mode
37db178 Move default policy.json away from etc/
6964ac2 Ship api-paste.ini out of etc/
3b1415e rest: make sure 409 is returned when double creating resource with non-UUID
c751087 tools: import a small tools to compute size of archive policies
246b787 archive_policy: lighten the default archive policies
6449ed8 run-tests: use case rather than if/elif/else
72fdba7 metricd: move metricd options in metricd group
00d842c storage: remove temporary incoming setup
9376c00 Introduce new storage groups for storage
2cb85a7 devstack: prepare ceph keyring before using it
ca43bec Adjust testr group_regex to not group on 'prefix'
9200ae3 storage: split s3 driver
3632911 Fix expected content-type and move CORS tests to gabbi
1dcd2fd [doc] Note lack of constraints is a choice
c5dd0d9 gabbi: remove unused variable
4a7a3cc rest: catch create_metric duplicate
cd82f8d Revert "add mysql minimum version check"
57bbee6 add mysql minimum version check
85a39e3 All granularity input should be parsed as timespan
5edc39a fill series when aggregating cross metrics
14c58fc carbonara: log a message and do not fail if a file is missing
fb31248 Enable H904 hacking check
05c2656 storage: split swift driver
3385186 doc: add a page talking about collectd support
7cff2f7 Enable oslo_middleware healthcheck middleware by default
bbe2d37 use datetime when defining series range
9d0192a storage: split ceph driver
ecbb5fd storage: split file driver
f5794af fix oslo.db 4.15.0 breakage
4bdb88b rest: remove user_id and project_id from metric schema
fcf08aa api: use egg entry_point rather than code path
dbcfa9c doc: Add reference to gnocchi-nagios tool
b6d08c2 fix logging...
77c0aa0 rest: don't ignore measures of created metrics
049116d Modify variable's using method in Log Messages
9055624 Fix two trivial docs errors in architecture.rst
e413a93 storage: split the storage interface
14a7cba config: only include oslo.middleware options that are shipped
0b970f0 doc: remove unused links
db38938 file: remove tmp configuration
5b833bb storage: remove _pending_measures_to_process_count()
8612ab8 storage: split process_new_measures()
b48483a Remove 95pct and median from default archive policies
38c2d32 rest: wait for the thread pool executor result
00b349e indexer: list_metric(), skip sql if names is empty
e6d3cee rest: Don't use private webob API
4e5b18d rest: fix batching error handling
516e681 rest: don't fail if the batch measure is not a dict
db7ea6a Replace retry with tenacity.retry
e2732af ceph: make computed measures read async
ff53ff8 utils: do not retry on any exception
c5779e4 Remove usage of deprecated operatorPrecedence and remove duplicate operators
6959709 drop pytimeparse requirement
21966a7 support pandas.to_timedelta
fd8b81b Bump hacking to 0.12
219fd0d rest: use flatten_dict_to_keypairs instead of recursive_keypairs
0c66859 carbonara: add support for Gnocchi v2 measures format
68a2a35 support resampling on aggregation endpoint
7418017 support resampling
abfa908 Use tenacity.TryAgain rather than a own-defined exception
f9b19ad rest: allow to create missing metrics when sending measures in batch
04dde36 Interpolate strings using logging own methods
04917de metricd: retry slowly coordination connection failure
e4d3b57 rest: don't use is_body_seekable
ad5b4fc Allow timespan to be floating values
021a041 rest: empty search query in resource search
a8c170d ceph: move out of xattr completely
b97a91a support consistent timestamps
c95b98f Revert "drop non-I/O threading in upgrade"
dd7cda9 rest: register default JSON converter using pecan.jsonify
1e8b6de Don't hang with wsgiref
234b0f0 Add missing granularity to base storage class
e1ab93c json: remove outdated comment
2ba63d4 Update doc because default services are all being added to settings
4637aeb accommodate new oslo.config
06fcd34 devstack: stop all gnocchi services, not just api
b04b370 Fix incorrect EXTRA_FLAVOR in plugin.sh
e7d8faf rest: using ujson to deserialize
839316a doc: add s3 to the list of Carbonara based drivers
37d8fea Fix typo in release note file
6430afb Unify timestamp parsing
537cd29 rest: fix Epoch timestamp parsing
34beae6 test: rewrite test_post_unix_timestamp in Gabbi
4bc5170 Remove pecan_debug option
0e453f2 test: allow to pass OS_DEBUG
16a05de Remove unused requests dependency
9a95873 carbonara: fix SplitKey with datetime greater than 32bits value
201dc6d Revert "Remove the file named MANIFEST.in"
2ac7a20 Add helper for utcnow to epoch nano
2b1ed8d Add http_proxy_to_wsgi to api-paste
510d799 Use Cotyledon oslo config glue
6baf05b Add a S3 based storage driver
5186bcc tox: only install all storage drivers in py-$index or py-#index-all
50c6141 Remove Python 3.4 support
93bb23d doc,tests: fix reaggregate/reaggregation mispelling
b36b66e Fix oslo.log minimum requirement
bb3cd40 Remove the file named MANIFEST.in
811a8fd Stop adding ServiceAvailable group option
bbc821d sqlalchemy: use precise DATETIME rather than DECIMAL in MySQL
bbbe67e enable threading of batched writes
83f0299 switch to stop validating batch individually
c34b246 Don't mock ceph
0a06089 upgrade tests: don't patch gnocchi 3.0
1157cac tests: Cover resource-type modification
0198b5b Use xx=None instead of xx=[] to initialize the default value
35c4a08 Fix some gabbi tests
15c0c71 Add STRICT_RESPONSE_HEADERS check to gabbi tests
984a976 Add upgrade targets for Gnocchi 3.0
e26bf90 tox: shorter envdir name for upgrade target
21845a8 remove the pandas module in test test_carbonara.py
07034a1 doc: include stable/3.0 release notes
b5106eb Fix typos in tests/gabbi/gabbits/resource.yaml
b5ad717 Add granularity in searching for values in metrics
b8c0c23 Modify api startup parameters in devstack plugin
723557f Replace retrying with tenacity
31cdcb5 ceph: fix python3 issue
bd14276 Add simple upgrade tests
7a37e4e compute new first_block_timestamp once
3a8f315 Allow to pass search filter in the query string
Diffstat (except docs and test files)
-------------------------------------
.testr.conf | 2 +-
bindep.txt | 2 +
devstack/gate/post_test_hook.sh | 1 -
devstack/plugin.sh | 31 +-
devstack/settings | 1 -
etc/gnocchi/api-paste.ini | 31 --
etc/gnocchi/gnocchi-config-generator.conf | 9 -
etc/gnocchi/policy.json | 42 --
gnocchi/aggregates/moving_stats.py | 9 +-
gnocchi/archive_policy.py | 32 +-
gnocchi/carbonara.py | 482 +++++++++++++------
gnocchi/cli.py | 92 +++-
gnocchi/genconfig.py | 13 +-
gnocchi/gendoc.py | 8 +-
gnocchi/gnocchi-config-generator.conf | 11 +
gnocchi/indexer/__init__.py | 12 +-
gnocchi/indexer/alembic/alembic.ini | 34 --
gnocchi/indexer/alembic/env.py | 5 -
.../1e1a63d3d186_original_resource_id_not_null.py | 39 ++
.../397987e38570_no_more_slash_and_reencode.py | 185 +++++++
.../5c4f93e5bb4_mysql_float_to_timestamp.py | 77 +++
.../aba5a217ca9b_merge_created_in_creator.py | 53 ++
gnocchi/indexer/sqlalchemy.py | 142 ++++--
gnocchi/indexer/sqlalchemy_base.py | 78 ++-
gnocchi/indexer/sqlalchemy_extension.py | 26 +-
gnocchi/json.py | 14 +-
gnocchi/opts.py | 82 +++-
gnocchi/resource_type.py | 71 ++-
gnocchi/rest/__init__.py | 532 +++++++++++++--------
gnocchi/rest/api-paste.ini | 46 ++
gnocchi/rest/app.py | 29 +-
gnocchi/rest/auth_helper.py | 125 +++++
gnocchi/rest/policy.json | 42 ++
gnocchi/service.py | 18 +-
gnocchi/statsd.py | 41 +-
gnocchi/storage/__init__.py | 79 +--
gnocchi/storage/_carbonara.py | 410 ++++++++--------
gnocchi/storage/ceph.py | 286 +++--------
gnocchi/storage/common/__init__.py | 0
gnocchi/storage/common/ceph.py | 66 +++
gnocchi/storage/common/s3.py | 81 ++++
gnocchi/storage/common/swift.py | 63 +++
gnocchi/storage/file.py | 133 +-----
gnocchi/storage/incoming/__init__.py | 58 +++
gnocchi/storage/incoming/_carbonara.py | 85 ++++
gnocchi/storage/incoming/ceph.py | 209 ++++++++
gnocchi/storage/incoming/file.py | 124 +++++
gnocchi/storage/incoming/s3.py | 151 ++++++
gnocchi/storage/incoming/swift.py | 98 ++++
gnocchi/storage/s3.py | 200 ++++++++
gnocchi/storage/swift.py | 118 +----
gnocchi/tempest/config.py | 13 +-
gnocchi/tempest/plugin.py | 13 +-
.../gabbi/gabbits/metric-timestamp-format.yaml | 60 +++
gnocchi/utils.py | 169 +++++--
...add-parameter-granularity-7f22c677dc1b1238.yaml | 4 +
.../archive_policy_bool-9313cae7122c4a2f.yaml | 5 +
.../notes/auth_type_option-c335b219afba5569.yaml | 5 +
.../auth_type_pluggable-76a3c73cac8eec6a.yaml | 5 +
...ackfill-cross-aggregation-2de54c7c30b2eb67.yaml | 6 +
...e_measures_create_metrics-f73790a8475ad628.yaml | 5 +
releasenotes/notes/ceph-omap-34e069dfb3df764d.yaml | 5 +
.../notes/ceph-read-async-ca2f7512c6842adb.yaml | 4 +
.../notes/creator_field-6b715c917f6afc93.yaml | 6 +
.../notes/dynamic-resampling-b5e545b1485c152f.yaml | 6 +
.../notes/forbid-slash-b3ec2bc77cc34b49.yaml | 7 +
.../gnocchi_config_generator-0fc337ba8e3afd5f.yaml | 5 +
.../healthcheck-middleware-81c2f0d02ebdb5cc.yaml | 5 +
...-default-archive-policies-455561c027edf4ad.yaml | 5 +
.../mysql_precise_datetime-57f868f3f42302e2.yaml | 4 +
.../noauth-force-headers-dda926ce83f810e8.yaml | 5 +
.../noauth-keystone-compat-e8f760591d593f07.yaml | 9 +
.../pecan-debug-removed-1a9dbc4a0a6ad581.yaml | 3 +
releasenotes/notes/reloading-734a639a667c93ee.yaml | 6 +
...fault-aggregation-methods-2f5ec059855e17f9.yaml | 5 +
...-type-required-attributes-f446c220d54c8eb7.yaml | 6 +
releasenotes/notes/s3_driver-4b30122bdbe0385d.yaml | 5 +
.../notes/storage-incoming-586b3e81de8deb4f.yaml | 6 +
.../notes/uuid5-change-8a8c467d2b2d4c85.yaml | 12 +
requirements.txt | 18 +-
setup.cfg | 30 +-
tools/gnocchi-archive-policy-size.py | 49 ++
tools/measures_injector.py | 26 +-
tox.ini | 83 +++-
128 files changed, 6499 insertions(+), 2617 deletions(-)
Requirements updates
--------------------
diff --git a/requirements.txt b/requirements.txt
index 23cf70d..ea3d96a 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -2 +2 @@ pbr
-numpy
+numpy>=1.9.0
@@ -5 +5 @@ oslo.config>=2.6.0
-oslo.log>=1.0.0
+oslo.log>=2.3.0
@@ -8,3 +8,4 @@ oslo.serialization>=1.4.0
-oslo.utils>=3.3.0
-oslo.middleware>=3.11.0
-pandas>=0.17.0
+oslo.utils>=3.18.0
+oslo.middleware>=3.22.0
+pandas>=0.18.0
+scipy>=0.18.1 # BSD
@@ -12 +12,0 @@ pecan>=0.9
-pytimeparse>=1.1.5
@@ -15,2 +15 @@ jsonpatch
-cotyledon>=1.2.2
-requests
+cotyledon>=1.5.0
@@ -18,0 +18 @@ stevedore
+ujson
@@ -22 +22 @@ trollius; python_version < '3.4'
-retrying
+tenacity>=3.1.0 # Apache-2.0
More information about the Release-announce
mailing list