We are jazzed to announce the release of: monasca_transform 0.9.0: Data aggregation for metrics This is the first release of monasca_transform. This release is part of the ocata release series. Download the package from: https://tarballs.openstack.org/monasca-transform/ For more details, please see below. Changes in monasca_transform 52504f802da029399c0133c0aa1be895f5b65b67..0.9.0 ---------------------------------------------------------------------------- 8b1819b Enable monasca transform functional tests 8feb6db Monasca Transform devstack rotate spark logs in /var/run/spark 5c15a99 Enhanced refresh monasca transform script 2da3904 Hourly aggregation account for early arrving metrics 83650f7 Add Constraints support 47235b4 Updated from global requirements f99a3fa Fix development environment and functional tests 738a53f Optimize the link address 5fe82bf Replaces uuid.uuid4 with uuidutils.generate_uuid() 8a77c4f Updated from global requirements d78ac09 Updated from global requirements 9f9082d Delete doc/build/html/.root-marker 81b239c Updated from global requirements 8aef98d Use to kafka_lib library in monasca common a64f124 Change Spark version to 1.6.3 d8f283c Started adding kubernetes metrics aggregation 8d3f0b9 [Fix gate]Update test requirement 1579d8b Reuse existing spark sql context f8f5667 Remove unused logging import 00b4797 Corrected catch up aggregation logic 6cba31c Replaced e.message with str(e) 45a989e Updated from global requirements 41df30d Update monasca-transform git path in README f341768 Adjust proxy port parsing to only return number. c189fee Delete hourly offsets from offsets table d725b41 Updated from global requirements 88e04a3 Fix ci 903c9fe Addressed proxy issues for maven in vagrant 5998f16 Added tools/vagrant/.vagrant file to .gitignore 014c2b5 Delete deprecated Hacking in tox.ini 51001d2 Manual sync with openstack/requirements ecde33a Show team and repo badges on README 0a73dc9 Refactored vagrant environment for xenial 043eec2 Added blank test hooks for ci 26e5333 Add configurable amnesty period for late metrics c3fcd61 Remove unique metric count aggregation fd6a79c Fixed vagrant provision script d76af6a Corrected test environment 8a6e619 Populate the project id for kafka publish 30b9540 Fixed the refresh script 501297a Enable cleanup for spark-worker 8ac3250 Remove metric filters from transform-spec 6cc6f94 Fixed minor issues in unstack 698856b Use database password from variable 1c65ca0 Validate metrics before publishing to kafka 0ea79c0 Added aggregation results to application log 87a8960 Changed hourly storage.objects.size_agg operation 4edad02 Eliminated ceiling function for utilization metrics 3cdb0d1 Made corrections such that swiftlm.diskusage.rate_agg is now correctly based on swiftlm.diskusage.host.val.avail (instead of incorrectly being based on swiftlm.diskusage.host.val.size). bf2e42b Made changes to prevent multiple metrics in the same batch. eff0e74 Eliminated processing_meta from hourly metrics. d4f791e Added aggregation of storage-objects.size. 0365bfa Modifications to include processing_meta in pre-hourly metrics. 615e52d Modifications to make rate calculations work with two-stage aggregation. accbacb Reintroduced some aggregations which were apparently lost when we transitioned to the upstream Monasca-Transform OpenStack repo. Specifically, the missing aggregations were those for the nova.vm.cpu.total_allocated and nova.vm.mem.total_allocated_mb source metrics. This set of changes also includes the resolution of a couple pre-existing pep8 errors. 442b7fb add test_fetch_quantity_util_agg 3d9e1d9 Removed version from monasca-common dependency 6b40f66 add test cases for first and second stage of host and vm cpu usage processing a977550 Removed 'device' as an expected dimension on incoming Swift metrics (since it is no longer included in them). bb83b30 Add TLS/SSL capability to database connection acba178 add test cases for fetch_quantity_prehourly_instance_usage c498564 add 'string' to all firstrecord_timestamp and lastrecord_timestamp to fix bug c7128b0 Added filter capability for transform specs. bbf977a timestamp parsed into utc instead of localtime(which was default) Closes Bug:#160531 90b20bf Change to monasca-common simport b3f85e1 Change jdbc driver to drizzle 1c3a798 Added some bulletproofing to catch invalid configuration entries for caching levels. Also changed the calculate_rate component to use values from instance usage if available (rather than using 'all'). 82fe3b9 Update kafka-python version to 0.9.5 1afd082 fix no_proxy bug in tools/vagrant/Vagrantfile 9c95206 Corrected log file name 00b874a Two stage transformation d8e73f3 Added several Swift aggregations (including a new usage component for calculating rate changes). Also fixed some pep8 issues. 02b2374 Removed spark-events from config 633aff4 Vagrant to use local branch for devstack 72f240b Removed patch for monasca-api plugin c7aabb6 Added aggregation for vm.mem.used_mb and swiftlm.diskusage.host.val.size. Also renamed disk.allocation to vm.disk.allocation and resolved a problem with resource_id not being found for certain aggregations. 11e8bac Added aggregation for cpu.total_logical_cores and cpu.utilized_logical_cores by host. 05c36ab Allow configurable SPARK_HOME e4ade60 Implemented aggregation for disk.allocation. Also set the apache download source to use the archive site to ensure that the dependency package does not disappear. Also brought the vagrant environment inline with monasca-api (i.e., use the same values for private network, add substitution for kafka brokers ip address to the conf). Also parameterised dependency sources (i.e., added settings to parameterise the maven and apache repositories for the devstack plugin). 8f61dd9 monasca-transform initial commit Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..eb18bd6 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,13 @@ +# The order of packages is significant, because pip processes them in the order +# of appearance. Changing the order has an impact on the overall integration +# process, which may cause wedges in the gate later. +pbr!=2.1.0,>=2.0.0 # Apache-2.0 +PyMySQL>=0.7.6 # MIT License +six>=1.9.0 # MIT +SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT +stevedore>=1.20.0 # Apache-2.0 +monasca-common>=1.4.0 # Apache-2.0 +oslo.config>=3.22.0 # Apache-2.0 +oslo.log>=3.22.0 # Apache-2.0 +oslo.service>=1.10.0 # Apache-2.0 +tooz>=1.47.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt new file mode 100644 index 0000000..920e5eb --- /dev/null +++ b/test-requirements.txt @@ -0,0 +1,16 @@ +# The order of packages is significant, because pip processes them in the order +# of appearance. Changing the order has an impact on the overall integration +# process, which may cause wedges in the gate later. +# mock object framework +hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 +flake8<2.6.0,>=2.5.4 # MIT +nose # LGPL +mock>=2.0 # BSD +fixtures>=3.0.0 # Apache-2.0/BSD +os-testr>=0.8.0 # Apache-2.0 +# required to build documentation +sphinx>=1.5.1 # BSD +#oslosphinx>=4.7.0 # Apache-2.0 +testrepository>=0.0.18 # Apache-2.0/BSD +# computes code coverage percentages +coverage>=4.0 # Apache-2.0