We jubilantly announce the release of: sahara 9.0.0: Sahara project This release is part of the rocky release series. The source is available from: https://git.openstack.org/cgit/openstack/sahara Download the package from: https://tarballs.openstack.org/sahara/ For more details, please see below. Changes in sahara 8.0.0..9.0.0 ------------------------------ 7d9483b Fixing anti-affinity for Sahara fa007bc import zuul job settings from project-config 0a88f08 Update UPPER_CONSTRAINTS_FILE for stable/rocky 6f61053 Update .gitreview for stable/rocky 5eab743 Imported Translations from Zanata 141a67e Add some S3 doc 7f55880 Enable also ambari by default in devstack bb7c039 Another small fix for cluster creation on APIv2 a37dfac S3 data source URL format change fee5d17 Sets correct permission for /etc/hosts 7654940 Fixing cluster creation on APIv2 bf68afb Allow overriding of /etc/hosts entries 094138d Enable mutable config in sahara 15317a6 Adding Ambari 2.6 to image pack b6504ee Adding Storm 1.2.0 and 1.2.1 8ecbeff Unversioned endpoint recommendation 8b830df api-ref: move to a v1.1 sub-folder bb60fbc Trivial: Update Zuul Status Page to correct URL 9aab4b5 Switch make_json_error back to being a function cb0e5ad Final fixup to APIv2 responses de5a4cf Deprecate sahara-all 4b4effa Switch hive_enable_db_notification's default value a449558 S3 data source 6f66893 Switch the coverage tox target to stestr e1a36ee Updating Spark versions f5bf9ff Fixing extjs check on cdh and mapr 1e14e8d Switch ostestr to stestr dd1c265 Bump Flask version according requirements 2ebcc90 Fix flask.request.content_length is None 72b493a Use register_error_handler to register make_json_error f172235 Boot from volume 203c796 Remove any reference to pre-built images 17509c3 Updating plugins status for Rocky 28eb4ce Adding CDH 5.13 1d3b02e Replace the deleted keypair in clusters for API v2 391bfba Better default value for domain in swift config ed59cfe Improve force delete d0d1a1d Updated oozie version aae9253 Fix the code repository for clone action 9988c7e add release notes to readme.rst 455b456 doc: light cleanup of the ironic-integration page 39e7c2d doc: external link helper for other projects' doc b817420 Update the command to change the hostname ebda885 fix tox python3 overrides aae8a86 Check node processes earlier f96709a [APIv2]Consolidate cluster creation endpoints 4c5fab8 Add support to deploy hadoop 2.7.5 522627e Restore Ambari with newer JDK security policies d0833f3 Fixing java version for Ambari 9f6b2bf Switch from sahara-file to tarballs.o.o for artifacts 3ff85ab Deploy using wsgi by default e8bcce9 Fix: really install extjs in CDH images at build time 03587a0 doc: add the redirect for a file recently renamed 9abcec3 Fix the detection of scala version (now https) 6edd220 Fix the installation of Swift Hadoop connector (Ambari) 8cc8768 Fix the installation of the Swift Hadoop connector (CDH) 892db4a fix a typo: s/avaliable/available 775f6e4 Remove the (now obsolete) pip-missing-reqs tox target d8ab151 Replace Chinese punctuation with English punctuation 7ed8ebf Fix the openstack endpoint create failed e789f9a Fix: always use kafka 2.2 for CDH 5.11 d9c9fe4 Adding Ambari missing versions ee42f01 Extend config-grabbing magic to new oslo.config 2a1d92c Adding ntpdate and Scala to mapr image f757422 Change doc registering-image image message 7553ae8 Remove step upload package to oozie/sharelib 755a808 uncap eventlet b3c8d75 Fix MapR dependency on mysql on RHEL db7cc2d correct lower-constraints 3be693e Support of HDP 2.6 edc7097 Follow the new PTI for document build ab3903a Updated from global requirements ae0f1af add lower-constraints job 68b447d File copy timesout when file is too big fee369b Preload soci-mysql and soci on RHEL7 images 3734ef7 Migration to Storyboard 809f937 Updated from global requirements 7c2ce6b Updated from global requirements 9868888 Updated from global requirements 33335b4 Adding support for RHEL images 0e6ab1d Remove unused module 12b4d11 change python-libguestfs to python-guestfs for ubuntu acec53b Updated from global requirements 23f36bf Imported Translations from Zanata c8c4273 Updated from global requirements 4e35c08 Update mysql connection in configuration-guide.rst e622fd6 Imported Translations from Zanata 779cade Fix Spark EDP job failed in vanilla 2.8.2 fd67730 Fix documents title format error 8ae30c0 Migrate the artifact link to sahara-extra, use https eeee06a Updated from global requirements 7c14eba Updated from global requirements e97e0ab Adding Ambari 2.4.2.0 to image gen 6fc0f25 Native Zuul v3 jobs (almost all of them) e0a7d03 Change some parameters to be required in api-ref 6861df5 Fix the parameter in api-ref 90602fc Imported Translations from Zanata ae51614 Update reno for stable/queens Diffstat (except docs and test files) ------------------------------------- .gitreview | 1 + .stestr.conf | 2 +- .testr.conf | 8 - .zuul.yaml | 62 + CONTRIBUTING.rst | 4 +- README.rst | 6 +- api-ref/source/cluster-templates.inc | 253 - api-ref/source/clusters.inc | 335 -- api-ref/source/conf.py | 2 +- api-ref/source/data-sources.inc | 212 - api-ref/source/event-log.inc | 42 - api-ref/source/image-registry.inc | 249 - api-ref/source/index.rst | 26 +- api-ref/source/job-binaries.inc | 266 - api-ref/source/job-binary-internals.inc | 258 - api-ref/source/job-executions.inc | 325 -- api-ref/source/job-types.inc | 61 - api-ref/source/jobs.inc | 265 - api-ref/source/node-group-templates.inc | 269 - api-ref/source/parameters.yaml | 1159 ---- api-ref/source/plugins.inc | 187 - .../cluster-template-create-request.json | 17 - .../cluster-template-create-response.json | 82 - .../cluster-template-show-response.json | 82 - .../cluster-template-update-request.json | 11 - .../cluster-template-update-response.json | 67 - .../cluster-templates-list-response.json | 140 - .../samples/clusters/cluster-create-request.json | 9 - .../samples/clusters/cluster-create-response.json | 128 - .../samples/clusters/cluster-scale-request.json | 15 - .../samples/clusters/cluster-scale-response.json | 370 -- .../samples/clusters/cluster-show-response.json | 128 - .../samples/clusters/cluster-update-request.json | 4 - .../samples/clusters/cluster-update-response.json | 128 - .../samples/clusters/clusters-list-response.json | 327 -- .../clusters/multiple-clusters-create-request.json | 11 - .../multiple-clusters-create-response.json | 6 - .../data-source-register-hdfs-request.json | 6 - .../data-source-register-hdfs-response.json | 14 - .../data-source-register-swift-request.json | 10 - .../data-source-register-swift-response.json | 14 - .../data-sources/data-source-show-response.json | 14 - .../data-sources/data-source-update-request.json | 4 - .../data-sources/data-source-update-response.json | 14 - .../data-sources/data-sources-list-response.json | 28 - .../event-log/cluster-progress-response.json | 72 - .../image-registry/image-register-request.json | 4 - .../image-registry/image-register-response.json | 25 - .../image-registry/image-show-response.json | 24 - .../image-registry/image-tags-add-request.json | 7 - .../image-registry/image-tags-add-response.json | 27 - .../image-registry/image-tags-delete-request.json | 5 - .../image-registry/image-tags-delete-response.json | 25 - .../image-registry/images-list-response.json | 48 - .../samples/job-binaries/create-request.json | 9 - .../samples/job-binaries/create-response.json | 13 - .../source/samples/job-binaries/list-response.json | 37 - .../source/samples/job-binaries/show-data-response | 3 - .../source/samples/job-binaries/show-response.json | 13 - .../samples/job-binaries/update-request.json | 5 - .../samples/job-binaries/update-response.json | 13 - .../job-binary-internals/create-response.json | 12 - .../job-binary-internals/list-response.json | 24 - .../job-binary-internals/show-data-response | 3 - .../job-binary-internals/show-response.json | 12 - .../job-binary-internals/update-request.json | 4 - .../job-binary-internals/update-response.json | 12 - .../samples/job-executions/cancel-response.json | 120 - .../samples/job-executions/job-ex-response.json | 120 - .../job-executions/job-ex-update-request.json | 3 - .../job-executions/job-ex-update-response.json | 120 - .../samples/job-executions/list-response.json | 122 - .../samples/job-types/job-types-list-response.json | 209 - .../source/samples/jobs/job-create-request.json | 11 - .../source/samples/jobs/job-create-response.json | 35 - .../source/samples/jobs/job-execute-request.json | 19 - .../source/samples/jobs/job-execute-response.json | 30 - api-ref/source/samples/jobs/job-show-response.json | 26 - .../source/samples/jobs/job-update-request.json | 4 - .../source/samples/jobs/job-update-response.json | 26 - .../source/samples/jobs/jobs-list-response.json | 66 - .../node-group-template-create-request.json | 13 - .../node-group-template-create-response.json | 33 - .../node-group-template-show-response.json | 37 - .../node-group-template-update-request.json | 10 - .../node-group-template-update-response.json | 30 - .../node-group-templates-list-response.json | 76 - .../samples/plugins/plugin-show-response.json | 12 - .../samples/plugins/plugin-update-request.json | 7 - .../samples/plugins/plugin-update-response.json | 32 - .../plugins/plugin-version-show-response.json | 92 - .../samples/plugins/plugins-list-response.json | 41 - api-ref/source/v1.1/cluster-templates.inc | 253 + api-ref/source/v1.1/clusters.inc | 335 ++ api-ref/source/v1.1/data-sources.inc | 212 + api-ref/source/v1.1/event-log.inc | 42 + api-ref/source/v1.1/image-registry.inc | 249 + api-ref/source/v1.1/index.rst | 20 + api-ref/source/v1.1/job-binaries.inc | 266 + api-ref/source/v1.1/job-binary-internals.inc | 258 + api-ref/source/v1.1/job-executions.inc | 325 ++ api-ref/source/v1.1/job-types.inc | 61 + api-ref/source/v1.1/jobs.inc | 265 + api-ref/source/v1.1/node-group-templates.inc | 269 + api-ref/source/v1.1/parameters.yaml | 1159 ++++ api-ref/source/v1.1/plugins.inc | 187 + .../cluster-template-create-request.json | 17 + .../cluster-template-create-response.json | 82 + .../cluster-template-show-response.json | 82 + .../cluster-template-update-request.json | 11 + .../cluster-template-update-response.json | 67 + .../cluster-templates-list-response.json | 140 + .../samples/clusters/cluster-create-request.json | 9 + .../samples/clusters/cluster-create-response.json | 128 + .../samples/clusters/cluster-scale-request.json | 15 + .../samples/clusters/cluster-scale-response.json | 370 ++ .../samples/clusters/cluster-show-response.json | 128 + .../samples/clusters/cluster-update-request.json | 4 + .../samples/clusters/cluster-update-response.json | 128 + .../samples/clusters/clusters-list-response.json | 327 ++ .../clusters/multiple-clusters-create-request.json | 11 + .../multiple-clusters-create-response.json | 6 + .../data-source-register-hdfs-request.json | 6 + .../data-source-register-hdfs-response.json | 14 + .../data-source-register-swift-request.json | 10 + .../data-source-register-swift-response.json | 14 + .../data-sources/data-source-show-response.json | 14 + .../data-sources/data-source-update-request.json | 4 + .../data-sources/data-source-update-response.json | 14 + .../data-sources/data-sources-list-response.json | 28 + .../event-log/cluster-progress-response.json | 72 + .../image-registry/image-register-request.json | 4 + .../image-registry/image-register-response.json | 25 + .../image-registry/image-show-response.json | 24 + .../image-registry/image-tags-add-request.json | 7 + .../image-registry/image-tags-add-response.json | 27 + .../image-registry/image-tags-delete-request.json | 5 + .../image-registry/image-tags-delete-response.json | 25 + .../image-registry/images-list-response.json | 48 + .../v1.1/samples/job-binaries/create-request.json | 9 + .../v1.1/samples/job-binaries/create-response.json | 13 + .../v1.1/samples/job-binaries/list-response.json | 37 + .../v1.1/samples/job-binaries/show-data-response | 3 + .../v1.1/samples/job-binaries/show-response.json | 13 + .../v1.1/samples/job-binaries/update-request.json | 5 + .../v1.1/samples/job-binaries/update-response.json | 13 + .../job-binary-internals/create-response.json | 12 + .../job-binary-internals/list-response.json | 24 + .../job-binary-internals/show-data-response | 3 + .../job-binary-internals/show-response.json | 12 + .../job-binary-internals/update-request.json | 4 + .../job-binary-internals/update-response.json | 12 + .../samples/job-executions/cancel-response.json | 120 + .../samples/job-executions/job-ex-response.json | 120 + .../job-executions/job-ex-update-request.json | 3 + .../job-executions/job-ex-update-response.json | 120 + .../v1.1/samples/job-executions/list-response.json | 122 + .../samples/job-types/job-types-list-response.json | 209 + .../v1.1/samples/jobs/job-create-request.json | 11 + .../v1.1/samples/jobs/job-create-response.json | 35 + .../v1.1/samples/jobs/job-execute-request.json | 19 + .../v1.1/samples/jobs/job-execute-response.json | 30 + .../v1.1/samples/jobs/job-show-response.json | 26 + .../v1.1/samples/jobs/job-update-request.json | 4 + .../v1.1/samples/jobs/job-update-response.json | 26 + .../v1.1/samples/jobs/jobs-list-response.json | 66 + .../node-group-template-create-request.json | 13 + .../node-group-template-create-response.json | 33 + .../node-group-template-show-response.json | 37 + .../node-group-template-update-request.json | 10 + .../node-group-template-update-response.json | 30 + .../node-group-templates-list-response.json | 76 + .../v1.1/samples/plugins/plugin-show-response.json | 12 + .../samples/plugins/plugin-update-request.json | 7 + .../samples/plugins/plugin-update-response.json | 32 + .../plugins/plugin-version-show-response.json | 92 + .../samples/plugins/plugins-list-response.json | 41 + devstack/settings | 4 +- .../dashboard-dev-environment-guide.rst | 4 +- lower-constraints.txt | 161 + playbooks/buildimages/run.yaml | 6 + .../ambari26-image-pack-88c9aad59bf635b2.yaml | 3 + .../apiv2-payload-tweaks-b73c20a35263d958.yaml | 9 + .../notes/boot-from-volume-e7078452fac1a4a0.yaml | 3 + releasenotes/notes/cdh-513-bdce0d5d269d8f20.yaml | 3 + ...te-cluster-creation-apiv2-5d5aceeb2e97c702.yaml | 5 + ...te-sahara-all-entry-point-1446a00dab643b7b.yaml | 4 + ...ble-mutable-configuration-2dd6b7a0e0fe4437.yaml | 6 + .../force-delete-changes-2e0881a99742c339.yaml | 6 + .../hadoop-swift-domain-fix-c1dfdf6c52b5aa25.yaml | 5 + releasenotes/notes/hdp26-5a406d7066706bf1.yaml | 3 + .../keypair-replacement-0c0cc3db0551c112.yaml | 5 + ...lib-step-in-vanilla-2.8.2-546b2026e2f5d557.yaml | 6 + .../s3-datasource-protocol-d3abd0b22f653b3b.yaml | 4 + ...ndpoint-version-discovery-826e9f31093cb10f.yaml | 8 + releasenotes/notes/spark-2.3-0277fe9feae6668a.yaml | 3 + releasenotes/notes/storm-1.2-af75fedb413de56a.yaml | 4 + .../support-s3-data-source-a912e2cdf4cd51fb.yaml | 3 + ...updating-plugins-versions-b8d27764178c3cdd.yaml | 11 + .../vanilla-2.7.5-support-ffeeb88fc4be34b4.yaml | 4 + releasenotes/source/conf.py | 2 +- releasenotes/source/index.rst | 1 + .../locale/en_GB/LC_MESSAGES/releasenotes.po | 812 +++ .../source/locale/fr/LC_MESSAGES/releasenotes.po | 63 - releasenotes/source/queens.rst | 6 + requirements.txt | 14 +- roles/build-sahara-images-cli/README.rst | 13 + roles/build-sahara-images-cli/defaults/main.yaml | 4 + roles/build-sahara-images-cli/tasks/main.yaml | 4 + sahara/api/middleware/sahara_middleware.py | 25 +- sahara/api/v2/cluster_templates.py | 16 +- sahara/api/v2/clusters.py | 48 +- sahara/api/v2/job_templates.py | 8 +- sahara/api/v2/jobs.py | 22 +- sahara/api/v2/node_group_templates.py | 15 +- sahara/cli/sahara_all.py | 13 + sahara/conductor/manager.py | 35 +- sahara/conductor/objects.py | 3 + sahara/conductor/resource.py | 6 + sahara/config.py | 16 + .../versions/034_boot_from_volume.py | 40 + sahara/db/sqlalchemy/models.py | 3 + sahara/locale/de/LC_MESSAGES/sahara.po | 2239 ++++++++ sahara/main.py | 2 +- sahara/plugins/ambari/deploy.py | 4 + sahara/plugins/ambari/plugin.py | 7 +- sahara/plugins/ambari/resources/configs-2.6.json | 2008 +++++++ .../images/centos/disable_certificate_check | 12 + .../images/centos/unlimited_security_artifacts | 4 +- .../plugins/ambari/resources/images/common/add_jar | 2 +- .../resources/images/common/fix_tls_ambari_agent | 17 + .../images/common/mysql_connector_java_link | 14 + sahara/plugins/ambari/resources/images/image.yaml | 26 +- sahara/plugins/cdh/cloudera_utils.py | 6 + sahara/plugins/cdh/config_helper.py | 5 +- sahara/plugins/cdh/plugin.py | 4 +- sahara/plugins/cdh/plugin_utils.py | 18 +- .../cdh/v5_11_0/resources/images/centos/wget_repo | 4 +- .../cdh/v5_11_0/resources/images/common/add_jar | 23 +- .../v5_11_0/resources/images/common/install_extjs | 32 +- .../cdh/v5_11_0/resources/images/image.yaml | 12 +- sahara/plugins/cdh/v5_13_0/__init__.py | 0 sahara/plugins/cdh/v5_13_0/cloudera_utils.py | 34 + sahara/plugins/cdh/v5_13_0/config_helper.py | 89 + sahara/plugins/cdh/v5_13_0/deploy.py | 168 + sahara/plugins/cdh/v5_13_0/edp_engine.py | 47 + sahara/plugins/cdh/v5_13_0/images.py | 43 + sahara/plugins/cdh/v5_13_0/plugin_utils.py | 23 + sahara/plugins/cdh/v5_13_0/resources/cdh_config.py | 68 + .../plugins/cdh/v5_13_0/resources/flume-agent.json | 440 ++ .../cdh/v5_13_0/resources/flume-service.json | 164 + .../cdh/v5_13_0/resources/hbase-gateway.json | 122 + .../v5_13_0/resources/hbase-hbaserestserver.json | 452 ++ .../v5_13_0/resources/hbase-hbasethriftserver.json | 458 ++ .../cdh/v5_13_0/resources/hbase-master.json | 530 ++ .../cdh/v5_13_0/resources/hbase-regionserver.json | 884 +++ .../cdh/v5_13_0/resources/hbase-service.json | 692 +++ .../cdh/v5_13_0/resources/hdfs-balancer.json | 128 + .../cdh/v5_13_0/resources/hdfs-datanode.json | 656 +++ .../v5_13_0/resources/hdfs-failovercontroller.json | 344 ++ .../cdh/v5_13_0/resources/hdfs-gateway.json | 116 + .../plugins/cdh/v5_13_0/resources/hdfs-httpfs.json | 464 ++ .../cdh/v5_13_0/resources/hdfs-journalnode.json | 458 ++ .../cdh/v5_13_0/resources/hdfs-namenode.json | 884 +++ .../cdh/v5_13_0/resources/hdfs-nfsgateway.json | 410 ++ .../v5_13_0/resources/hdfs-secondarynamenode.json | 458 ++ .../cdh/v5_13_0/resources/hdfs-service.json | 1352 +++++ .../cdh/v5_13_0/resources/hive-gateway.json | 98 + .../cdh/v5_13_0/resources/hive-hivemetastore.json | 434 ++ .../cdh/v5_13_0/resources/hive-hiveserver2.json | 866 +++ .../resources/hive-metastore-sentry-safety.xml | 15 + .../resources/hive-server2-sentry-safety.xml | 12 + .../cdh/v5_13_0/resources/hive-service.json | 656 +++ sahara/plugins/cdh/v5_13_0/resources/hive-site.xml | 61 + .../cdh/v5_13_0/resources/hive-webhcat.json | 356 ++ .../v5_13_0/resources/hue-hue_load_balancer.json | 260 + .../cdh/v5_13_0/resources/hue-hue_server.json | 404 ++ .../cdh/v5_13_0/resources/hue-kt_renewer.json | 212 + .../plugins/cdh/v5_13_0/resources/hue-service.json | 692 +++ .../resources/images/centos/disable_firewall | 20 + .../resources/images/centos/selinux_permissive | 13 + .../resources/images/centos/turn_off_services | 33 + .../cdh/v5_13_0/resources/images/centos/wget_repo | 27 + .../cdh/v5_13_0/resources/images/common/add_jar | 22 + .../resources/images/common/install_cloudera | 17 + .../v5_13_0/resources/images/common/install_extjs | 30 + .../images/common/unlimited_security_artifacts | 11 + .../cdh/v5_13_0/resources/images/image.yaml | 128 + .../resources/images/ubuntu/turn_off_services | 33 + .../cdh/v5_13_0/resources/images/ubuntu/wget_repo | 31 + .../v5_13_0/resources/impala-catalogserver.json | 542 ++ .../cdh/v5_13_0/resources/impala-impalad.json | 746 +++ .../cdh/v5_13_0/resources/impala-llama.json | 518 ++ .../cdh/v5_13_0/resources/impala-service.json | 572 ++ .../cdh/v5_13_0/resources/impala-statestore.json | 404 ++ .../cdh/v5_13_0/resources/kafka-gateway.json | 62 + .../cdh/v5_13_0/resources/kafka-kafka_broker.json | 584 ++ .../resources/kafka-kafka_mirror_maker.json | 644 +++ .../cdh/v5_13_0/resources/kafka-service.json | 404 ++ sahara/plugins/cdh/v5_13_0/resources/kms-kms.json | 656 +++ .../plugins/cdh/v5_13_0/resources/kms-service.json | 86 + .../resources/ks_indexer-hbase_indexer.json | 374 ++ .../cdh/v5_13_0/resources/ks_indexer-service.json | 188 + .../plugins/cdh/v5_13_0/resources/mapred-site.xml | 157 + .../cdh/v5_13_0/resources/oozie-oozie_server.json | 674 +++ .../cdh/v5_13_0/resources/oozie-service.json | 260 + .../cdh/v5_13_0/resources/priority-one-confs.json | 45 + .../cdh/v5_13_0/resources/sentry-gateway.json | 62 + .../resources/sentry-impala-client-safety.xml | 16 + .../v5_13_0/resources/sentry-sentry_server.json | 320 ++ .../cdh/v5_13_0/resources/sentry-service.json | 230 + .../cdh/v5_13_0/resources/solr-gateway.json | 50 + .../cdh/v5_13_0/resources/solr-service.json | 350 ++ .../cdh/v5_13_0/resources/solr-solr_server.json | 470 ++ .../cdh/v5_13_0/resources/spark-gateway.json | 242 + .../cdh/v5_13_0/resources/spark-service.json | 170 + .../resources/spark-spark_yarn_history_server.json | 422 ++ .../cdh/v5_13_0/resources/sqoop-service.json | 116 + .../cdh/v5_13_0/resources/sqoop-sqoop_server.json | 410 ++ .../v5_13_0/resources/sqoop_client-gateway.json | 92 + .../v5_13_0/resources/sqoop_client-service.json | 80 + .../cdh/v5_13_0/resources/yarn-gateway.json | 530 ++ .../cdh/v5_13_0/resources/yarn-jobhistory.json | 500 ++ .../cdh/v5_13_0/resources/yarn-nodemanager.json | 794 +++ .../v5_13_0/resources/yarn-resourcemanager.json | 686 +++ .../cdh/v5_13_0/resources/yarn-service.json | 554 ++ .../cdh/v5_13_0/resources/zookeeper-server.json | 608 ++ .../cdh/v5_13_0/resources/zookeeper-service.json | 236 + sahara/plugins/cdh/v5_13_0/validation.py | 21 + sahara/plugins/cdh/v5_13_0/versionhandler.py | 37 + sahara/plugins/cdh/v5_5_0/__init__.py | 0 sahara/plugins/cdh/v5_5_0/cloudera_utils.py | 27 - sahara/plugins/cdh/v5_5_0/config_helper.py | 101 - sahara/plugins/cdh/v5_5_0/deploy.py | 168 - sahara/plugins/cdh/v5_5_0/edp_engine.py | 47 - sahara/plugins/cdh/v5_5_0/plugin_utils.py | 23 - sahara/plugins/cdh/v5_5_0/resources/cdh_config.py | 135 - sahara/plugins/cdh/v5_5_0/resources/cdh_config.sh | 3 - .../plugins/cdh/v5_5_0/resources/flume-agent.json | 428 -- .../cdh/v5_5_0/resources/flume-service.json | 164 - .../cdh/v5_5_0/resources/hbase-gateway.json | 110 - .../v5_5_0/resources/hbase-hbaserestserver.json | 440 -- .../v5_5_0/resources/hbase-hbasethriftserver.json | 446 -- .../plugins/cdh/v5_5_0/resources/hbase-master.json | 506 -- .../cdh/v5_5_0/resources/hbase-regionserver.json | 836 --- .../cdh/v5_5_0/resources/hbase-service.json | 740 --- .../cdh/v5_5_0/resources/hdfs-balancer.json | 68 - .../cdh/v5_5_0/resources/hdfs-datanode.json | 644 --- .../v5_5_0/resources/hdfs-failovercontroller.json | 332 -- .../plugins/cdh/v5_5_0/resources/hdfs-gateway.json | 116 - .../plugins/cdh/v5_5_0/resources/hdfs-httpfs.json | 440 -- .../cdh/v5_5_0/resources/hdfs-journalnode.json | 446 -- .../cdh/v5_5_0/resources/hdfs-namenode.json | 812 --- .../cdh/v5_5_0/resources/hdfs-nfsgateway.json | 386 -- .../v5_5_0/resources/hdfs-secondarynamenode.json | 446 -- .../plugins/cdh/v5_5_0/resources/hdfs-service.json | 1388 ----- .../plugins/cdh/v5_5_0/resources/hive-gateway.json | 98 - .../cdh/v5_5_0/resources/hive-hivemetastore.json | 392 -- .../cdh/v5_5_0/resources/hive-hiveserver2.json | 686 --- .../resources/hive-metastore-sentry-safety.xml | 15 - .../resources/hive-server2-sentry-safety.xml | 12 - .../plugins/cdh/v5_5_0/resources/hive-service.json | 704 --- sahara/plugins/cdh/v5_5_0/resources/hive-site.xml | 61 - .../plugins/cdh/v5_5_0/resources/hive-webhcat.json | 332 -- .../v5_5_0/resources/hue-hue_load_balancer.json | 236 - .../cdh/v5_5_0/resources/hue-hue_server.json | 356 -- .../cdh/v5_5_0/resources/hue-kt_renewer.json | 200 - .../plugins/cdh/v5_5_0/resources/hue-service.json | 674 --- .../cdh/v5_5_0/resources/impala-catalogserver.json | 404 -- .../cdh/v5_5_0/resources/impala-impalad.json | 662 --- .../plugins/cdh/v5_5_0/resources/impala-llama.json | 506 -- .../cdh/v5_5_0/resources/impala-service.json | 596 -- .../cdh/v5_5_0/resources/impala-statestore.json | 350 -- .../cdh/v5_5_0/resources/kafka-kafka_broker.json | 476 -- .../v5_5_0/resources/kafka-kafka_mirror_maker.json | 482 -- .../cdh/v5_5_0/resources/kafka-service.json | 374 -- sahara/plugins/cdh/v5_5_0/resources/kms-kms.json | 320 -- .../plugins/cdh/v5_5_0/resources/kms-service.json | 50 - .../v5_5_0/resources/ks_indexer-hbase_indexer.json | 356 -- .../cdh/v5_5_0/resources/ks_indexer-service.json | 188 - .../plugins/cdh/v5_5_0/resources/mapred-site.xml | 157 - .../cdh/v5_5_0/resources/oozie-oozie_server.json | 662 --- .../cdh/v5_5_0/resources/oozie-service.json | 236 - .../cdh/v5_5_0/resources/priority-one-confs.json | 45 - .../resources/sentry-impala-client-safety.xml | 16 - .../cdh/v5_5_0/resources/sentry-sentry_server.json | 308 - .../cdh/v5_5_0/resources/sentry-service.json | 278 - .../plugins/cdh/v5_5_0/resources/solr-gateway.json | 32 - .../plugins/cdh/v5_5_0/resources/solr-service.json | 404 -- .../cdh/v5_5_0/resources/solr-solr_server.json | 440 -- .../cdh/v5_5_0/resources/spark-gateway.json | 158 - .../cdh/v5_5_0/resources/spark-service.json | 152 - .../resources/spark-spark_yarn_history_server.json | 236 - .../cdh/v5_5_0/resources/sqoop-service.json | 110 - .../cdh/v5_5_0/resources/sqoop-sqoop_server.json | 398 -- .../plugins/cdh/v5_5_0/resources/yarn-gateway.json | 482 -- .../cdh/v5_5_0/resources/yarn-jobhistory.json | 464 -- .../cdh/v5_5_0/resources/yarn-nodemanager.json | 728 --- .../cdh/v5_5_0/resources/yarn-resourcemanager.json | 668 --- .../plugins/cdh/v5_5_0/resources/yarn-service.json | 512 -- .../cdh/v5_5_0/resources/zookeeper-server.json | 596 -- .../cdh/v5_5_0/resources/zookeeper-service.json | 224 - sahara/plugins/cdh/v5_5_0/validation.py | 21 - sahara/plugins/cdh/v5_5_0/versionhandler.py | 35 - .../cdh/v5_7_0/resources/images/common/add_jar | 23 +- .../v5_7_0/resources/images/common/install_extjs | 32 +- .../plugins/cdh/v5_7_0/resources/images/image.yaml | 12 +- .../cdh/v5_9_0/resources/images/common/add_jar | 23 +- .../v5_9_0/resources/images/common/install_extjs | 32 +- .../plugins/cdh/v5_9_0/resources/images/image.yaml | 12 +- sahara/plugins/images.py | 1 + sahara/plugins/kerberos.py | 2 +- sahara/plugins/mapr/base/base_health_checker.py | 4 +- sahara/plugins/mapr/plugin.py | 2 - .../mapr/resources/images/centos/epel_dependencies | 6 + .../mapr/resources/images/centos/install_scala | 16 +- .../mapr/resources/images/common/configure_extjs | 32 +- sahara/plugins/mapr/resources/images/image.yaml | 18 +- .../mapr/resources/images/ubuntu/install_scala | 46 +- sahara/plugins/mapr/services/oozie/oozie.py | 21 - .../plugins/mapr/versions/v5_1_0_mrv2/__init__.py | 0 .../plugins/mapr/versions/v5_1_0_mrv2/context.py | 60 - .../mapr/versions/v5_1_0_mrv2/version_handler.py | 76 - sahara/plugins/spark/edp_engine.py | 2 +- sahara/plugins/spark/plugin.py | 4 +- sahara/plugins/storm/config_helper.py | 16 +- sahara/plugins/storm/edp_engine.py | 4 +- sahara/plugins/storm/plugin.py | 6 +- sahara/plugins/vanilla/hadoop2/config_helper.py | 18 - sahara/plugins/vanilla/hadoop2/run_scripts.py | 2 - sahara/plugins/vanilla/v2_7_1/config_helper.py | 27 +- sahara/plugins/vanilla/v2_7_5/__init__.py | 0 sahara/plugins/vanilla/v2_7_5/config_helper.py | 151 + sahara/plugins/vanilla/v2_7_5/edp_engine.py | 81 + sahara/plugins/vanilla/v2_7_5/resources/README.rst | 27 + .../vanilla/v2_7_5/resources/core-default.xml | 1997 +++++++ .../vanilla/v2_7_5/resources/create_hive_db.sql | 9 + .../vanilla/v2_7_5/resources/hdfs-default.xml | 2677 +++++++++ .../vanilla/v2_7_5/resources/hive-default.xml | 5959 ++++++++++++++++++++ .../vanilla/v2_7_5/resources/mapred-default.xml | 2221 ++++++++ .../vanilla/v2_7_5/resources/oozie-default.xml | 2843 ++++++++++ .../vanilla/v2_7_5/resources/yarn-default.xml | 1713 ++++++ sahara/plugins/vanilla/v2_7_5/versionhandler.py | 170 + sahara/plugins/vanilla/v2_8_2/config_helper.py | 27 +- sahara/service/api/v10.py | 2 +- sahara/service/api/v2/clusters.py | 10 +- sahara/service/edp/data_sources/opts.py | 2 +- sahara/service/edp/data_sources/s3/__init__.py | 0 .../service/edp/data_sources/s3/implementation.py | 86 + .../edp/oozie/workflow_creator/workflow_factory.py | 20 + sahara/service/edp/s3_common.py | 10 + sahara/service/edp/spark/engine.py | 18 +- sahara/service/engine.py | 9 +- sahara/service/heat/heat_engine.py | 13 +- sahara/service/heat/templates.py | 37 +- sahara/service/ops.py | 22 + sahara/service/validations/clusters.py | 7 + sahara/service/validations/clusters_schema.py | 16 +- sahara/service/validations/edp/base.py | 2 +- .../validations/node_group_template_schema.py | 5 + sahara/swift/swift_helper.py | 10 + .../unit/plugins/cdh/base_plugin_utils_test.py | 4 +- .../unit/plugins/cdh/v5_5_0/test_config_helper.py | 25 - .../plugins/cdh/v5_5_0/test_config_helper_550.py | 168 - .../unit/plugins/cdh/v5_5_0/test_edp_engine.py | 200 - .../unit/plugins/cdh/v5_5_0/test_plugin_utils.py | 25 - .../plugins/cdh/v5_5_0/test_plugin_utils_550.py | 25 - .../unit/plugins/cdh/v5_5_0/test_spark_engine.py | 69 - .../unit/plugins/cdh/v5_5_0/test_validation.py | 24 - .../unit/plugins/cdh/v5_5_0/test_versionhandler.py | 140 - .../plugins/vanilla/hadoop2/test_config_helper.py | 13 - .../plugins/vanilla/hadoop2/test_run_scripts.py | 2 - .../plugins/vanilla/v2_7_1/test_config_helper.py | 18 +- .../plugins/vanilla/v2_7_5/test_config_helper.py | 73 + .../unit/plugins/vanilla/v2_7_5/test_edp_engine.py | 96 + .../plugins/vanilla/v2_7_5/test_versionhandler.py | 253 + .../plugins/vanilla/v2_8_2/test_config_helper.py | 18 +- .../unit/service/edp/data_sources/s3/__init__.py | 0 .../service/edp/data_sources/s3/test_s3_type.py | 116 + .../service/validation/edp/test_job_executor.py | 4 +- sahara/utils/api.py | 8 +- sahara/utils/cluster.py | 7 + sahara/utils/openstack/heat.py | 9 +- sahara/utils/openstack/keystone.py | 2 + sahara/utils/openstack/nova.py | 5 + sahara/utils/ssh_remote.py | 7 +- setup.cfg | 7 +- test-requirements.txt | 7 +- tools/cover.sh | 26 +- tools/gate/build-images | 2 +- tox.ini | 61 +- 542 files changed, 60227 insertions(+), 34205 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index ce5573f..6cea8b0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,2 +11,2 @@ castellan>=0.16.0 # Apache-2.0 -eventlet!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2 # MIT -Flask!=0.11,<1.0,>=0.10 # BSD +eventlet!=0.18.3,!=0.20.1,>=0.18.2 # MIT +Flask>=1.0.2 # BSD @@ -14 +14 @@ iso8601>=0.1.11 # MIT -Jinja2!=2.9.0,!=2.9.1,!=2.9.2,!=2.9.3,!=2.9.4,>=2.8 # BSD License (3 clause) +Jinja2>=2.10 # BSD License (3 clause) @@ -16 +16 @@ jsonschema<3.0.0,>=2.6.0 # MIT -keystoneauth1>=3.3.0 # Apache-2.0 +keystoneauth1>=3.4.0 # Apache-2.0 @@ -18,2 +18,2 @@ keystonemiddleware>=4.17.0 # Apache-2.0 -oslo.config>=5.1.0 # Apache-2.0 -oslo.concurrency>=3.25.0 # Apache-2.0 +oslo.config>=5.2.0 # Apache-2.0 +oslo.concurrency>=3.26.0 # Apache-2.0 @@ -38 +38 @@ python-swiftclient>=3.2.0 # Apache-2.0 -python-neutronclient>=6.3.0 # Apache-2.0 +python-neutronclient>=6.7.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index a7b7a8f..52c0626 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -14 +13,0 @@ mock>=2.0.0 # BSD -openstackdocstheme>=1.18.1 # Apache-2.0 @@ -16,2 +15 @@ oslotest>=3.2.0 # Apache-2.0 -os-api-ref>=1.4.0 # Apache-2.0 -os-testr>=1.0.0 # Apache-2.0 +stestr>=1.0.0 # Apache-2.0 @@ -20,3 +17,0 @@ pylint==1.4.5 # GPLv2 -reno>=2.5.0 # Apache-2.0 -sphinx!=1.6.6,>=1.6.2 # BSD -sphinxcontrib-httpdomain>=1.3.0 # BSD