We are excited to announce the release of: sahara-plugin-spark 1.0.0: Spark Plugin for Sahara Project This release is part of the stein release series. The source is available from: https://git.openstack.org/cgit/openstack/sahara-plugin-spark Download the package from: https://tarballs.openstack.org/sahara-plugin-spark/ Please report issues through: https://storyboard.openstack.org/#!/project/1104 For more details, please see below. Changes in sahara-plugin-spark 927379388c422e58c995f2067c18953198530205..1.0.0 ------------------------------------------------------------------------------ fcdf058 Add the buildimages job to the check queue b85e306 Adding string to int conversion eb6333c add python 3.7 unit test job 3a1da9f Adding Spark to sahara-image-pack 9431924 Reduce the dependencies, add more common Zuul jobs 3ac6ec0 Sync the tox.ini files with the other plugins 84c8877 Update mailinglist from dev to discuss a66b3ff Migrate away from oslo_i18n.enable_lazy() 1d1feec Fix translations: add the babel.cfg file f6876f9 Post-import fixes: name, license, doc, translations ee84691 Updating plugin documentation and release notes 5b5dcbe Add .gitreview and basic Zuul jobs fbc2044 Plugins splitted from sahara core b7b39a2 Add framework for sahara-status upgrade check d103751 doc: restructure the image building documentation 5ea6463 Cleanup tox.ini constraint handling 33ebe62 doc: update distro information and cloud-init users d2d6a9f Imported Translations from Zanata e29af39 Update reno for stable/rocky ce808d0 Imported Translations from Zanata 8121952 S3 data source aaf3c14 Switch the coverage tox target to stestr 6d63cfe Updating Spark versions dd61a64 Switch ostestr to stestr 06959eb Bump Flask version according requirements c4b88af Remove any reference to pre-built images 4fb2459 Updating plugins status for Rocky 1e1f0f6 fix tox python3 overrides b8db88b doc: add the redirect for a file recently renamed 82238db Remove the (now obsolete) pip-missing-reqs tox target 46b60ce uncap eventlet 731358c Follow the new PTI for document build c218995 Updated from global requirements 8443768 add lower-constraints job 6f62ba2 Migration to Storyboard ba8c306 Updated from global requirements 7b1b2b7 Updated from global requirements eff1c9a Updated from global requirements ccdaf42 Updated from global requirements c91a41b Updated from global requirements be7371a Imported Translations from Zanata 7fde9d0 Updated from global requirements ce08416 Updated from global requirements 4a8d85d Imported Translations from Zanata e9c06bf Update reno for stable/queens 70cbf65 Replace chinese quotes 897867d Enable hacking-extensions H204, H205 3f48078 Updated from global requirements 9a6e638 Updated from global requirements 7fd59c2 Updated from global requirements 64fd743 Updated from global requirements 1c02167 Updated from global requirements b48477c S3 job binary and binary retriever 90047ba Updated from global requirements 201e148 Updated from global requirements b6eb439 Updated from global requirements 715a734 Upgrading Spark to version 2.2 6f66d34 Updated from global requirements d41e6ad Updated from global requirements 889be66 Updated from global requirements 8a6e314 Updated from global requirements 7679805 Remove setting of version/release from releasenotes ceb4882 Updated from global requirements 4c4f718 Updated from global requirements f6bf49a Incorrect indent Sahara Installation Guide in sahara 5df78cf Updated from global requirements c697c03 Policy in code for Sahara 4926011 TrivialFix: Redundant alias in import statement 869c7ca Updated from global requirements a0800d8 Updated from global requirements 247202c Add default configuration files to data_files c5f065e Updated from global requirements 738841e Updated from global requirements 2a828dc Updated from global requirements fd96518 [ut] replace .testr.conf with .stestr.conf fc36c97 Updated from global requirements a0f8b2d Updated from global requirements 87b1b04 Updated from global requirements 7fc86fa Updated from global requirements aec8dd0 doc: point to the main git repository and update links d77cab7 Updated from global requirements a8dad03 Updated from global requirements c21ff76 Imported Translations from Zanata e95be6c Update reno for stable/pike a0447f6 Updated from global requirements 50d34e3 Restructure the documentation according the new spec 7e03d8e Deprecate Spark 1.3.1 ec13d22 Enable some off-by-default checks a9cc6fa Updated from global requirements b619f94 Updated from global requirements 9233721 Updated from global requirements c688e89 Updated from global requirements 55a5b13 Updated from global requirements d85c3ff Updated from global requirements 639311d Update the documentation link for doc migration 3bdcf1c Update Documention link 57d7299 Updated from global requirements 676ebc5 Enable warnings as errors for doc building 56ef27f Enable H904 check 0260ea1 doc: update the configuration of the theme b2d90a1 Updated from global requirements 1e6541e doc: switch to openstackdocstheme and add metadata a2b5f1d Updated from global requirements 072aa6b Updated from global requirements e3cba9c Updated from global requirements 8696083 Updated from global requirements eb52742 Updated from global requirements 72dd23c Updated from global requirements 59f48a7 Updated from global requirements 4850f9c Raise better exception for Spark master validation f727b40 Updated from global requirements f8482aa Basic script for pack-based build image a574de6 Remove usage of parameter enforce_type 13766ad Updated from global requirements 30b7223 Updated from global requirements c8399e5 Remove log translations b4fcbb7 Updated from global requirements bf1e6e8 Remove log translations 7bc9b7f Remove log translations 8d75437 Remove log translations 36cc624 Upgrading Spark version to 2.1.0 05d000b Updated from global requirements e73f730 Apply monkeypatching from eventlet before the tests starts 9e9ab41 install saharaclient from pypi if not from source 443887e Fix some reST field lists in docstrings a64d49f Add ability to install with Apache in devstack 846b42d Support Job binary pluggability 72e1db5 Updated from global requirements 55ef414 Updated from global requirements 116b4c2 Support Data Source pluggability e420f83 Indicating the location tests directory in oslo_debug_helper 396fb88 Fix api-ref build cf909f3 Updated from global requirements b53c840 [Fix gate]Update test requirement 67f96cc Updated from global requirements 3221709 Updated from global requirements 17a7615 Updated from global requirements 95fc4dd Updated from global requirements 0db8711 Remove support for py34 5e01f8d Update reno for stable/ocata 0165c71 Updated from global requirements 1e16478 Updated from global requirements c9dbf95 Updated from global requirements ace688b Updated from global requirements 4f18e91 Updated from global requirements 1014cb2 Updated from global requirements f1d3659 Updated from global requirements 46e9e5b Updated from global requirements b15804b Updated from global requirements 4395089 Updated from global requirements 765230e Updated from global requirements 0c51e15 Updated from global requirements 886678c Updated from global requirements a630fb9 Updated from global requirements 85ec4d0 Updated from global requirements a1f4847 spelling fixed. 7629148 Updated from global requirements f696833 Updated from global requirements 38f5aa3 Remove enable_notifications option 9a51ed0 Updated from global requirements 9ba22bc Updated from global requirements 27f915c Updated from global requirements 018a603 Updated from global requirements b73c3da Updated from global requirements e141245 Updated from global requirements 41ac6a2 Constraints are ready to be used for tox.ini 9c271ce Enable release notes translation e4f9882 Updated from global requirements 911ae33 Updated from global requirements ab26826 Updated from global requirements 8592866 Updated from global requirements 3e6f622 Updated from global requirements fe3fd65 Updated from global requirements a3a355c Updated from global requirements Diffstat (except docs and test files) ------------------------------------- .gitignore | 1 + .gitreview | 4 + .stestr.conf | 3 + .zuul.yaml | 14 ++ LICENSE | 175 +++++++++++++++ README.rst | 38 ++++ babel.cfg | 1 + lower-constraints.txt | 162 ++++++++++++++ .../spark-on-image-pack-f5609daf38c45b6f.yaml | 4 + releasenotes/source/conf.py | 40 ++-- releasenotes/source/index.rst | 9 +- releasenotes/source/liberty.rst | 6 - releasenotes/source/mitaka.rst | 6 - releasenotes/source/newton.rst | 6 - requirements.txt | 49 +---- {sahara => sahara_plugin_spark}/__init__.py | 0 {sahara => sahara_plugin_spark}/i18n.py | 15 +- .../plugins/__init__.py | 0 .../plugins/spark/__init__.py | 0 .../plugins/spark/config_helper.py | 64 +++--- .../plugins/spark/edp_engine.py | 14 +- sahara_plugin_spark/plugins/spark/images.py | 44 ++++ .../plugins/spark/plugin.py | 107 +++++---- .../plugins/spark/resources/README.rst | 0 .../plugins/spark/resources/core-default.xml | 0 .../plugins/spark/resources/hdfs-default.xml | 0 .../resources/images/centos/turn_off_services | 7 + .../spark/resources/images/centos/wget_cdh_repo | 43 ++++ .../plugins/spark/resources/images/common/add_jar | 20 ++ .../spark/resources/images/common/install_extjs | 30 +++ .../spark/resources/images/common/install_spark | 41 ++++ .../spark/resources/images/common/manipulate_s3 | 12 ++ .../plugins/spark/resources/images/image.yaml | 61 ++++++ .../spark/resources/images/ubuntu/config_spark | 12 ++ .../resources/images/ubuntu/turn_off_services | 7 + .../spark/resources/images/ubuntu/wget_cdh_repo | 36 ++++ .../plugins/spark/resources/spark-cleanup.cron | 0 .../plugins/spark/resources/spark-env.sh.template | 0 .../spark/resources/tmp-cleanup.sh.template | 0 .../plugins/spark/resources/topology.sh | 0 .../plugins/spark/run_scripts.py | 10 +- .../plugins/spark/scaling.py | 25 +-- .../plugins/spark/shell_engine.py | 6 +- {sahara => sahara_plugin_spark}/utils/__init__.py | 0 {sahara => sahara_plugin_spark}/utils/patches.py | 0 setup.cfg | 74 ++----- setup.py | 2 +- test-requirements.txt | 24 +-- tox.ini | 113 ++++------ 63 files changed, 1566 insertions(+), 533 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 488089d..ff6e585 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5 +5 @@ -pbr>=1.6 # Apache-2.0 +pbr!=2.1.0,>=2.0.0 # Apache-2.0 @@ -7,38 +7,9 @@ pbr>=1.6 # Apache-2.0 -alembic>=0.8.4 # MIT -Babel>=2.3.4 # BSD -castellan>=0.4.0 # Apache-2.0 -eventlet!=0.18.3,>=0.18.2 # MIT -Flask!=0.11,<1.0,>=0.10 # BSD -iso8601>=0.1.11 # MIT -Jinja2>=2.8 # BSD License (3 clause) -jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT -keystoneauth1>=2.10.0 # Apache-2.0 -keystonemiddleware!=4.1.0,!=4.5.0,>=4.0.0 # Apache-2.0 -oslo.config>=3.14.0 # Apache-2.0 -oslo.concurrency>=3.8.0 # Apache-2.0 -oslo.context>=2.9.0 # Apache-2.0 -oslo.db!=4.13.1,!=4.13.2,>=4.10.0 # Apache-2.0 -oslo.i18n>=2.1.0 # Apache-2.0 -oslo.log>=1.14.0 # Apache-2.0 -oslo.messaging>=5.2.0 # Apache-2.0 -oslo.middleware>=3.0.0 # Apache-2.0 -oslo.policy>=1.9.0 # Apache-2.0 -oslo.rootwrap>=5.0.0 # Apache-2.0 -oslo.serialization>=1.10.0 # Apache-2.0 -oslo.service>=1.10.0 # Apache-2.0 -oslo.utils>=3.16.0 # Apache-2.0 -paramiko>=2.0 # LGPLv2.1+ -requests>=2.10.0 # Apache-2.0 -python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0 -python-keystoneclient!=2.1.0,>=2.0.0 # Apache-2.0 -python-manilaclient>=1.10.0 # Apache-2.0 -python-novaclient!=2.33.0,>=2.29.0 # Apache-2.0 -python-swiftclient>=2.2.0 # Apache-2.0 -python-neutronclient>=5.1.0 # Apache-2.0 -python-heatclient>=1.4.0 # Apache-2.0 -python-glanceclient!=2.4.0,>=2.3.0 # Apache-2.0 -six>=1.9.0 # MIT -stevedore>=1.16.0 # Apache-2.0 -SQLAlchemy<1.1.0,>=1.0.10 # MIT -tooz>=1.28.0 # Apache-2.0 -WebOb>=1.2.3 # MIT +Babel!=2.4.0,>=2.3.4 # BSD +eventlet!=0.18.3,!=0.20.1,>=0.18.2 # MIT +oslo.i18n>=3.15.3 # Apache-2.0 +oslo.log>=3.36.0 # Apache-2.0 +oslo.serialization!=2.19.1,>=2.18.0 # Apache-2.0 +oslo.utils>=3.33.0 # Apache-2.0 +requests>=2.14.2 # Apache-2.0 +sahara>=10.0.0.0b1 +six>=1.10.0 # MIT diff --git a/test-requirements.txt b/test-requirements.txt index a456df3..9cffbb1 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -5 +5 @@ -hacking<0.11,>=0.10.0 +hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 @@ -7 +6,0 @@ hacking<0.11,>=0.10.0 -PyMySQL!=0.7.7,>=0.6.2 # MIT License @@ -9,3 +8,3 @@ bandit>=1.1.0 # Apache-2.0 -bashate>=0.2 # Apache-2.0 -coverage>=3.6 # Apache-2.0 -doc8 # Apache-2.0 +bashate>=0.5.1 # Apache-2.0 +coverage!=4.4,>=4.0 # Apache-2.0 +doc8>=0.6.0 # Apache-2.0 @@ -13,6 +12,3 @@ fixtures>=3.0.0 # Apache-2.0/BSD -mock>=2.0 # BSD -oslosphinx!=3.4.0,>=2.5.0 # Apache-2.0 -oslotest>=1.10.0 # Apache-2.0 -os-api-ref>=1.0.0 # Apache-2.0 -os-testr>=0.7.0 # Apache-2.0 -psycopg2>=2.5 # LGPL/ZPL +mock>=2.0.0 # BSD +oslotest>=3.2.0 # Apache-2.0 +stestr>=1.0.0 # Apache-2.0 @@ -20,4 +15,0 @@ pylint==1.4.5 # GPLv2 -reno>=1.8.0 # Apache2 -sphinx!=1.3b1,<1.3,>=1.2.1 # BSD -sphinxcontrib-httpdomain # BSD -testresources>=0.2.4 # Apache-2.0/BSD @@ -25 +17 @@ testscenarios>=0.4 # Apache-2.0/BSD -testtools>=1.4.0 # MIT +testtools>=2.2.0 # MIT