Open Stack

We are jazzed to announce the release of:

monasca_transform 0.9.0: Data aggregation for metrics

This is the first release of monasca_transform. This release is part
of the ocata release series.

Download the package from:

    https://tarballs.openstack.org/monasca-transform/

For more details, please see below.

Changes in monasca_transform 52504f802da029399c0133c0aa1be895f5b65b67..0.9.0
----------------------------------------------------------------------------

8b1819b Enable monasca transform functional tests
8feb6db Monasca Transform devstack rotate spark logs in /var/run/spark
5c15a99 Enhanced refresh monasca transform script
2da3904 Hourly aggregation account for early arrving metrics
83650f7 Add Constraints support
47235b4 Updated from global requirements
f99a3fa Fix development environment and functional tests
738a53f Optimize the link address
5fe82bf Replaces uuid.uuid4 with uuidutils.generate_uuid()
8a77c4f Updated from global requirements
d78ac09 Updated from global requirements
9f9082d Delete doc/build/html/.root-marker
81b239c Updated from global requirements
8aef98d Use to kafka_lib library in monasca common
a64f124 Change Spark version to 1.6.3
d8f283c Started adding kubernetes metrics aggregation
8d3f0b9 [Fix gate]Update test requirement
1579d8b Reuse existing spark sql context
f8f5667 Remove unused logging import
00b4797 Corrected catch up aggregation logic
6cba31c Replaced e.message with str(e)
45a989e Updated from global requirements
41df30d Update monasca-transform git path in README
f341768 Adjust proxy port parsing to only return number.
c189fee Delete hourly offsets from offsets table
d725b41 Updated from global requirements
88e04a3 Fix ci
903c9fe Addressed proxy issues for maven in vagrant
5998f16 Added tools/vagrant/.vagrant file to .gitignore
014c2b5 Delete deprecated Hacking in tox.ini
51001d2 Manual sync with openstack/requirements
ecde33a Show team and repo badges on README
0a73dc9 Refactored vagrant environment for xenial
043eec2 Added blank test hooks for ci
26e5333 Add configurable amnesty period for late metrics
c3fcd61 Remove unique metric count aggregation
fd6a79c Fixed vagrant provision script
d76af6a Corrected test environment
8a6e619 Populate the project id for kafka publish
30b9540 Fixed the refresh script
501297a Enable cleanup for spark-worker
8ac3250 Remove metric filters from transform-spec
6cc6f94 Fixed minor issues in unstack
698856b Use database password from variable
1c65ca0 Validate metrics before publishing to kafka
0ea79c0 Added aggregation results to application log
87a8960 Changed hourly storage.objects.size_agg operation
4edad02 Eliminated ceiling function for utilization metrics
3cdb0d1 Made corrections such that swiftlm.diskusage.rate_agg is now correctly based on swiftlm.diskusage.host.val.avail (instead of incorrectly being based on swiftlm.diskusage.host.val.size).
bf2e42b Made changes to prevent multiple metrics in the same batch.
eff0e74 Eliminated processing_meta from hourly metrics.
d4f791e Added aggregation of storage-objects.size.
0365bfa Modifications to include processing_meta in pre-hourly metrics.
615e52d Modifications to make rate calculations work with two-stage aggregation.
accbacb Reintroduced some aggregations which were apparently lost when we transitioned to the upstream Monasca-Transform OpenStack repo. Specifically, the missing aggregations were those for the nova.vm.cpu.total_allocated and nova.vm.mem.total_allocated_mb source metrics. This set of changes also includes the resolution of a couple pre-existing pep8 errors.
442b7fb add test_fetch_quantity_util_agg
3d9e1d9 Removed version from monasca-common dependency
6b40f66 add test cases for first and second stage of host and vm cpu usage processing
a977550 Removed 'device' as an expected dimension on incoming Swift metrics (since it is no longer included in them).
bb83b30 Add TLS/SSL capability to database connection
acba178 add test cases for fetch_quantity_prehourly_instance_usage
c498564 add 'string' to all firstrecord_timestamp and lastrecord_timestamp to fix bug
c7128b0 Added filter capability for transform specs.
bbf977a timestamp parsed into utc instead of localtime(which was default) Closes Bug:#160531
90b20bf Change to monasca-common simport
b3f85e1 Change jdbc driver to drizzle
1c3a798 Added some bulletproofing to catch invalid configuration entries for caching levels.  Also changed the calculate_rate component to use values from instance usage if available (rather than using 'all').
82fe3b9 Update kafka-python version to 0.9.5
1afd082 fix no_proxy bug in tools/vagrant/Vagrantfile
9c95206 Corrected log file name
00b874a Two stage transformation
d8e73f3 Added several Swift aggregations (including a new usage component for calculating rate changes).  Also fixed some pep8 issues.
02b2374 Removed spark-events from config
633aff4 Vagrant to use local branch for devstack
72f240b Removed patch for monasca-api plugin
c7aabb6 Added aggregation for vm.mem.used_mb and swiftlm.diskusage.host.val.size.  Also renamed disk.allocation to vm.disk.allocation and resolved a problem with resource_id not being found for certain aggregations.
11e8bac Added aggregation for cpu.total_logical_cores and cpu.utilized_logical_cores by host.
05c36ab Allow configurable SPARK_HOME
e4ade60 Implemented aggregation for disk.allocation.  Also set the apache download source to use the archive site to ensure that the dependency package does not disappear.  Also brought the vagrant environment inline with monasca-api (i.e., use the same values for private network, add substitution for kafka brokers ip address to the conf).  Also parameterised dependency sources (i.e., added settings to parameterise the maven and apache repositories for the devstack plugin).
8f61dd9 monasca-transform initial commit




Requirements updates
--------------------

diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..eb18bd6
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,13 @@
+# The order of packages is significant, because pip processes them in the order
+# of appearance. Changing the order has an impact on the overall integration
+# process, which may cause wedges in the gate later.
+pbr!=2.1.0,>=2.0.0 # Apache-2.0
+PyMySQL>=0.7.6 # MIT License
+six>=1.9.0 # MIT
+SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT
+stevedore>=1.20.0 # Apache-2.0
+monasca-common>=1.4.0 # Apache-2.0
+oslo.config>=3.22.0 # Apache-2.0
+oslo.log>=3.22.0 # Apache-2.0
+oslo.service>=1.10.0 # Apache-2.0
+tooz>=1.47.0 # Apache-2.0
diff --git a/test-requirements.txt b/test-requirements.txt
new file mode 100644
index 0000000..920e5eb
--- /dev/null
+++ b/test-requirements.txt
@@ -0,0 +1,16 @@
+# The order of packages is significant, because pip processes them in the order
+# of appearance. Changing the order has an impact on the overall integration
+# process, which may cause wedges in the gate later.
+# mock object framework
+hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0
+flake8<2.6.0,>=2.5.4 # MIT
+nose # LGPL
+mock>=2.0 # BSD
+fixtures>=3.0.0 # Apache-2.0/BSD
+os-testr>=0.8.0 # Apache-2.0
+# required to build documentation
+sphinx>=1.5.1 # BSD
+#oslosphinx>=4.7.0 # Apache-2.0
+testrepository>=0.0.18 # Apache-2.0/BSD
+# computes code coverage percentages
+coverage>=4.0 # Apache-2.0