We are pumped to announce the release of: sahara 10.0.0: Sahara project This release is part of the stein release series. The source is available from: https://git.openstack.org/cgit/openstack/sahara Download the package from: https://tarballs.openstack.org/sahara/ Please report issues through: https://bugs.launchpad.net/sahara/+bugs For more details, please see below. Changes in sahara 9.0.0.0rc1..10.0.0 ------------------------------------ bb5f75db doc: refer to the split plugin documentation dc17f190 Making Sahara Python 3 compatible 8010e24a grenade: re-enable, really test rocky->master f0a1cc99 Fix the lower-requirements job: libpq-dev, psycopg 2.7 cd68ad36 Add missing ws seperator between words b95503f3 Use authorize instead of enforce for policies f422b31b Fixing policies inconsistencies 423a9bf5 Add API v2 jobs (scenario, tempest); buildimages fixes 47ca63ae add python 3.7 unit test job b21e6969 Adapt to the additional rules from pycodestyle 2.5.0 8f62975a Fixing NTP issues for CDH plugin f6afe5b7 Adding spark build image job 21791d1f Changing hdfs fs to hdfs dfs 55bd5da5 Dynamically loading plugins 7f8c565d Add missing ws separator between words e51dd010 Make sure that default_ntp_server option is exported 1c16c48b Fix version discovery for Python 3 92686f5a Prepare Sahara core for plugin split a83c97f4 Declare APIv2 stable and CURRENT 33489a1f Give the illusion of microversion support 69d74c1a Some polish for APIv2 4bfd800f API v2: fix "local variable 'c' referenced before assignment" 7105a891 APIv2 - Fix 500 on malformed query string on 359b89de Enhance boot from volume 0b8002a9 APIv2 - api-ref documentation for APIv2 81d05942 Deploying Sahara with unversioned endpoints 596c2599 Fix validation of job binary with Python3 904d70f1 Migrate away from oslo_i18n.enable_lazy() 9bf50ce5 APIv2 Changing return payload to project_id 0b8ab036 Fixing cluster scale 5bb7f379 doc: Fix the snippet in "The Script Validator" section 7e24f288 String-related fixes for Python 3 3c165262 fixed word error c4554ccd Add DEBIAN_FRONTEND=noninteractive in front of apt-get install commands 61b0b2e0 Bump the version of hacking to 1.1.0, with few fixes 425c8256 Update devel info: mailing list, meeting time 98ac996d Update http link to https 7a815ac0 Add python 3.6 unit test job cc41e862 Add framework for sahara-status upgrade check 4a44010f doc: restructure the image building documentation 01d465dd Fixing image validation for Ambari 2.3 3f22c4a9 Cleanup tox.ini constraint handling 8bded6c9 Increase the startup time of ambari-server to 180s 3f292dbc Increment versioning with pbr instruction 059457e8 Fix a typo on Storm plugin cluster info (Strom -> Storm) 12b270b0 sahara-image-pack: use curl for tarballs.openstack.org 3a2b3f63 sahara-image-pack: remove bashisms from shell scripts 4e39a45f adds unit test for ssh_remote.replace_remote_line e0fd845d Force the format of ssh key to PEM, at least for now e7a4b58c Add template param for ambari pkg install timeout 98777adc Use templates lower-constraints, update cover job 45714289 grenade: relevant fixes for master (sahara-api/apache) 10e48a37 doc: update distro information and cloud-init users aa514b51 Fixed link for more information about Ambari images c59780a9 Correct repo_id_map for hdp 2.5 b4535e5d Make sahara-grenade job voting on the "gate" queue too 5a3ae85c Import the legacy grenade sahara job b248ef27 Correct Hbase ports in Ambari plugin 4ac73ea6 Fixing anti-affinity for Sahara f8566f6d add python 3.6 unit test job 10b64c23 switch documentation job to new PTI 25369d92 import zuul job settings from project-config 1b659db8 Imported Translations from Zanata 1cf0c5c6 Update reno for stable/rocky 77a106a8 Adapt to Keystone changes: use member instead of Member Diffstat (except docs and test files) ------------------------------------- .gitignore | 1 + .zuul.yaml | 75 +- api-ref/source/index.rst | 1 + api-ref/source/v1.1/index.rst | 6 +- api-ref/source/v2/cluster-templates.inc | 241 + api-ref/source/v2/clusters.inc | 293 + api-ref/source/v2/data-sources.inc | 202 + api-ref/source/v2/event-log.inc | 42 + api-ref/source/v2/image-registry.inc | 244 + api-ref/source/v2/index.rst | 20 + api-ref/source/v2/job-binaries.inc | 256 + api-ref/source/v2/job-templates.inc | 257 + api-ref/source/v2/job-types.inc | 61 + api-ref/source/v2/jobs.inc | 262 + api-ref/source/v2/node-group-templates.inc | 289 + api-ref/source/v2/parameters.yaml | 1183 ++++ api-ref/source/v2/plugins.inc | 179 + .../cluster-template-create-request.json | 17 + .../cluster-template-create-response.json | 82 + .../cluster-template-show-response.json | 82 + .../cluster-template-update-request.json | 11 + .../cluster-template-update-response.json | 67 + .../cluster-templates-list-response.json | 140 + .../samples/clusters/cluster-create-request.json | 9 + .../samples/clusters/cluster-create-response.json | 128 + .../v2/samples/clusters/cluster-scale-request.json | 15 + .../samples/clusters/cluster-scale-response.json | 370 ++ .../v2/samples/clusters/cluster-show-response.json | 128 + .../samples/clusters/cluster-update-request.json | 4 + .../samples/clusters/cluster-update-response.json | 128 + .../samples/clusters/clusters-list-response.json | 327 ++ .../clusters/multiple-clusters-create-request.json | 11 + .../multiple-clusters-create-response.json | 6 + .../data-source-register-hdfs-request.json | 6 + .../data-source-register-hdfs-response.json | 14 + .../data-source-register-swift-request.json | 10 + .../data-source-register-swift-response.json | 14 + .../data-sources/data-source-show-response.json | 14 + .../data-sources/data-source-update-request.json | 4 + .../data-sources/data-source-update-response.json | 14 + .../data-sources/data-sources-list-response.json | 28 + .../event-log/cluster-progress-response.json | 72 + .../image-registry/image-register-request.json | 4 + .../image-registry/image-register-response.json | 25 + .../image-registry/image-show-response.json | 24 + .../image-registry/image-tags-add-request.json | 7 + .../image-registry/image-tags-add-response.json | 27 + .../image-registry/image-tags-delete-request.json | 5 + .../image-registry/image-tags-delete-response.json | 25 + .../image-registry/images-list-response.json | 48 + .../v2/samples/job-binaries/create-request.json | 9 + .../v2/samples/job-binaries/create-response.json | 13 + .../v2/samples/job-binaries/list-response.json | 37 + .../v2/samples/job-binaries/show-data-response | 3 + .../v2/samples/job-binaries/show-response.json | 13 + .../v2/samples/job-binaries/update-request.json | 5 + .../v2/samples/job-binaries/update-response.json | 13 + .../job-templates/job-template-create-request.json | 11 + .../job-template-create-response.json | 35 + .../job-templates/job-template-show-response.json | 26 + .../job-templates/job-template-update-request.json | 4 + .../job-template-update-response.json | 26 + .../job-templates/job-templates-list-response.json | 66 + .../samples/job-types/job-types-list-response.json | 209 + .../source/v2/samples/jobs/cancel-response.json | 120 + api-ref/source/v2/samples/jobs/job-request.json | 20 + api-ref/source/v2/samples/jobs/job-response.json | 30 + .../source/v2/samples/jobs/job-update-request.json | 3 + .../v2/samples/jobs/job-update-response.json | 120 + api-ref/source/v2/samples/jobs/list-response.json | 122 + .../node-group-template-create-request.json | 13 + .../node-group-template-create-response.json | 33 + .../node-group-template-show-response.json | 37 + .../node-group-template-update-request.json | 10 + .../node-group-template-update-response.json | 30 + .../node-group-templates-list-response.json | 76 + .../v2/samples/plugins/plugin-show-response.json | 12 + .../v2/samples/plugins/plugin-update-request.json | 7 + .../v2/samples/plugins/plugin-update-response.json | 32 + .../plugins/plugin-version-show-response.json | 92 + .../v2/samples/plugins/plugins-list-response.json | 41 + bindep.txt | 1 + devstack/plugin.sh | 75 +- devstack/settings | 21 + devstack/upgrade/from-rocky/upgrade-sahara | 18 + devstack/upgrade/resources.sh | 2 +- devstack/upgrade/settings | 13 +- devstack/upgrade/shutdown.sh | 3 +- devstack/upgrade/upgrade.sh | 3 +- .../user/building-guest-images/baremetal.rst | 14 + .../building-guest-images/sahara-image-create.rst | 80 + .../building-guest-images/sahara-image-pack.rst | 90 + lower-constraints.txt | 12 +- playbooks/buildimages/run.yaml | 2 +- playbooks/sahara-grenade/post.yaml | 15 + playbooks/sahara-grenade/run.yaml | 72 + ...d-upgrade-check-framework-9cd18dbc47b0efbd.yaml | 13 + ...pkg-install-timeout-param-d50e5c15e06fa51e.yaml | 4 + .../api-v2-return-payload-a84a609db410228a.yaml | 4 + .../notes/apiv2-microversion-4c1a58ee8090e5a9.yaml | 5 + .../apiv2-stable-release-25ba9920c8e4632a.yaml | 3 + .../notes/enhance-bfv-12bac06c4438675f.yaml | 6 + ...ng-policy-inconsistencies-984020000cc3882a.yaml | 12 + releasenotes/notes/hdfs-dfs-94a9c4f64cf8994f.yaml | 5 + .../notes/ntp-config-51ed9d612132e2fa.yaml | 6 + ...ns-split-from-sahara-core-9ffc5e5d06c9239c.yaml | 5 + .../proxy-user-lowercase-f116f7b7e89274cb.yaml | 6 + .../notes/some-polish-api-v2-2d2e390a74b088f9.yaml | 12 + ...t-validation-query-string-a6cadbf2f9c57d06.yaml | 5 + releasenotes/source/index.rst | 1 + .../locale/en_GB/LC_MESSAGES/releasenotes.po | 8 +- .../source/locale/fr/LC_MESSAGES/releasenotes.po | 63 + releasenotes/source/rocky.rst | 6 + requirements.txt | 2 + roles/build-sahara-images-cli/tasks/main.yaml | 5 + sahara/api/acl.py | 4 +- sahara/api/microversion.py | 30 + sahara/api/middleware/version_discovery.py | 11 +- sahara/api/v2/cluster_templates.py | 31 +- sahara/api/v2/clusters.py | 43 +- sahara/api/v2/data_sources.py | 29 +- sahara/api/v2/images.py | 21 +- sahara/api/v2/job_binaries.py | 33 +- sahara/api/v2/job_templates.py | 58 +- sahara/api/v2/job_types.py | 11 +- sahara/api/v2/jobs.py | 28 +- sahara/api/v2/node_group_templates.py | 31 +- sahara/api/v2/plugins.py | 12 +- sahara/cli/image_pack/api.py | 4 + sahara/cli/image_pack/cli.py | 4 +- sahara/cli/sahara_all.py | 7 - sahara/cli/sahara_api.py | 8 - sahara/cli/sahara_engine.py | 9 - sahara/cli/sahara_status.py | 53 + sahara/cli/sahara_subprocess.py | 17 +- sahara/common/policies/__init__.py | 24 +- sahara/common/policies/base.py | 13 +- sahara/common/policies/cluster.py | 60 + sahara/common/policies/cluster_template.py | 57 + sahara/common/policies/cluster_templates.py | 12 +- sahara/common/policies/clusters.py | 12 - sahara/common/policies/data_source.py | 57 + sahara/common/policies/data_sources.py | 12 +- sahara/common/policies/image.py | 65 + sahara/common/policies/images.py | 27 +- sahara/common/policies/job.py | 49 + sahara/common/policies/job_binaries.py | 14 +- sahara/common/policies/job_binary.py | 62 + .../policies/{job_templates.py => job_template.py} | 12 +- .../edp_engine.py => common/policies/job_type.py} | 22 +- sahara/common/policies/job_types.py | 2 - sahara/common/policies/jobs.py | 11 +- sahara/common/policies/node_group_template.py | 57 + sahara/common/policies/node_group_templates.py | 12 +- sahara/common/policies/plugin.py | 49 + sahara/common/policies/plugins.py | 8 - sahara/conductor/__init__.py | 1 + sahara/conductor/manager.py | 3 + sahara/conductor/objects.py | 9 + sahara/config.py | 2 + sahara/db/migration/alembic_migrations/env.py | 1 + .../versions/035_boot_from_volume_enhancements.py | 70 + sahara/db/sqlalchemy/model_base.py | 1 + sahara/db/sqlalchemy/models.py | 9 + sahara/db/templates/api.py | 1 + sahara/plugins/ambari/__init__.py | 0 sahara/plugins/ambari/client.py | 363 -- sahara/plugins/ambari/common.py | 155 - sahara/plugins/ambari/configs.py | 327 -- sahara/plugins/ambari/deploy.py | 709 --- sahara/plugins/ambari/edp_engine.py | 130 - sahara/plugins/ambari/ha_helper.py | 252 - sahara/plugins/ambari/health.py | 149 - sahara/plugins/ambari/plugin.py | 294 - sahara/plugins/ambari/requests_helper.py | 145 - sahara/plugins/ambari/resources/configs-2.3.json | 1276 ----- sahara/plugins/ambari/resources/configs-2.4.json | 1331 ----- sahara/plugins/ambari/resources/configs-2.5.json | 2008 ------- sahara/plugins/ambari/resources/configs-2.6.json | 2008 ------- sahara/plugins/ambari/resources/generate_config.py | 80 - .../ambari/resources/images/centos/disable_ambari | 8 - .../images/centos/disable_certificate_check | 12 - .../resources/images/centos/disable_firewall | 20 - .../ambari/resources/images/centos/disable_selinux | 12 - .../ambari/resources/images/centos/setup_java_home | 31 - .../images/centos/unlimited_security_artifacts | 11 - .../ambari/resources/images/centos/wget_repo | 9 - .../plugins/ambari/resources/images/common/add_jar | 31 - .../resources/images/common/fix_tls_ambari_agent | 17 - .../images/common/mysql_connector_java_link | 14 - .../ambari/resources/images/common/oracle_java | 41 - sahara/plugins/ambari/resources/images/image.yaml | 111 - .../ambari/resources/images/ubuntu/wget_repo | 10 - sahara/plugins/ambari/validation.py | 226 - sahara/plugins/base.py | 12 +- .../v5_7_0/plugin_utils.py => castellan_utils.py} | 18 +- sahara/plugins/cdh/__init__.py | 0 sahara/plugins/cdh/abstractversionhandler.py | 234 - sahara/plugins/cdh/client/__init__.py | 0 sahara/plugins/cdh/client/api_client.py | 145 - sahara/plugins/cdh/client/clusters.py | 240 - sahara/plugins/cdh/client/cms.py | 84 - sahara/plugins/cdh/client/hosts.py | 90 - sahara/plugins/cdh/client/http_client.py | 143 - sahara/plugins/cdh/client/resource.py | 166 - sahara/plugins/cdh/client/role_config_groups.py | 108 - sahara/plugins/cdh/client/roles.py | 187 - sahara/plugins/cdh/client/services.py | 528 -- sahara/plugins/cdh/client/types.py | 684 --- sahara/plugins/cdh/client/users.py | 62 - sahara/plugins/cdh/cloudera_utils.py | 837 --- sahara/plugins/cdh/commands.py | 116 - sahara/plugins/cdh/config_helper.py | 307 - sahara/plugins/cdh/confighints_helper.py | 46 - sahara/plugins/cdh/db_helper.py | 120 - sahara/plugins/cdh/db_resources/create_hive_db.sql | 4 - .../plugins/cdh/db_resources/create_sentry_db.sql | 4 - sahara/plugins/cdh/deploy.py | 124 - sahara/plugins/cdh/edp_engine.py | 103 - sahara/plugins/cdh/exceptions.py | 78 - sahara/plugins/cdh/health.py | 145 - sahara/plugins/cdh/plugin.py | 125 - sahara/plugins/cdh/plugin_utils.py | 471 -- sahara/plugins/cdh/v5_11_0/__init__.py | 0 sahara/plugins/cdh/v5_11_0/cloudera_utils.py | 28 - sahara/plugins/cdh/v5_11_0/config_helper.py | 101 - sahara/plugins/cdh/v5_11_0/deploy.py | 168 - sahara/plugins/cdh/v5_11_0/edp_engine.py | 47 - sahara/plugins/cdh/v5_11_0/images.py | 43 - sahara/plugins/cdh/v5_11_0/plugin_utils.py | 23 - sahara/plugins/cdh/v5_11_0/resources/cdh_config.py | 63 - sahara/plugins/cdh/v5_11_0/resources/cdh_config.sh | 14 - .../plugins/cdh/v5_11_0/resources/flume-agent.json | 440 -- .../cdh/v5_11_0/resources/flume-service.json | 164 - .../cdh/v5_11_0/resources/hbase-gateway.json | 122 - .../v5_11_0/resources/hbase-hbaserestserver.json | 452 -- .../v5_11_0/resources/hbase-hbasethriftserver.json | 458 -- .../cdh/v5_11_0/resources/hbase-master.json | 518 -- .../cdh/v5_11_0/resources/hbase-regionserver.json | 878 --- .../cdh/v5_11_0/resources/hbase-service.json | 668 --- .../cdh/v5_11_0/resources/hdfs-balancer.json | 128 - .../cdh/v5_11_0/resources/hdfs-datanode.json | 656 --- .../v5_11_0/resources/hdfs-failovercontroller.json | 344 -- .../cdh/v5_11_0/resources/hdfs-gateway.json | 116 - .../plugins/cdh/v5_11_0/resources/hdfs-httpfs.json | 452 -- .../cdh/v5_11_0/resources/hdfs-journalnode.json | 458 -- .../cdh/v5_11_0/resources/hdfs-namenode.json | 872 --- .../cdh/v5_11_0/resources/hdfs-nfsgateway.json | 410 -- .../v5_11_0/resources/hdfs-secondarynamenode.json | 458 -- .../cdh/v5_11_0/resources/hdfs-service.json | 1340 ----- .../cdh/v5_11_0/resources/hive-gateway.json | 98 - .../cdh/v5_11_0/resources/hive-hivemetastore.json | 434 -- .../cdh/v5_11_0/resources/hive-hiveserver2.json | 848 --- .../resources/hive-metastore-sentry-safety.xml | 15 - .../resources/hive-server2-sentry-safety.xml | 12 - .../cdh/v5_11_0/resources/hive-service.json | 656 --- sahara/plugins/cdh/v5_11_0/resources/hive-site.xml | 61 - .../cdh/v5_11_0/resources/hive-webhcat.json | 344 -- .../v5_11_0/resources/hue-hue_load_balancer.json | 248 - .../cdh/v5_11_0/resources/hue-hue_server.json | 392 -- .../cdh/v5_11_0/resources/hue-kt_renewer.json | 212 - .../plugins/cdh/v5_11_0/resources/hue-service.json | 692 --- .../resources/images/centos/disable_firewall | 20 - .../resources/images/centos/selinux_permissive | 13 - .../resources/images/centos/turn_off_services | 33 - .../cdh/v5_11_0/resources/images/centos/wget_repo | 33 - .../cdh/v5_11_0/resources/images/common/add_jar | 22 - .../resources/images/common/install_cloudera | 17 - .../v5_11_0/resources/images/common/install_extjs | 30 - .../images/common/unlimited_security_artifacts | 11 - .../cdh/v5_11_0/resources/images/image.yaml | 131 - .../resources/images/ubuntu/turn_off_services | 33 - .../cdh/v5_11_0/resources/images/ubuntu/wget_repo | 34 - .../v5_11_0/resources/impala-catalogserver.json | 542 -- .../cdh/v5_11_0/resources/impala-impalad.json | 746 --- .../cdh/v5_11_0/resources/impala-llama.json | 518 -- .../cdh/v5_11_0/resources/impala-service.json | 572 -- .../cdh/v5_11_0/resources/impala-statestore.json | 404 -- .../cdh/v5_11_0/resources/kafka-gateway.json | 62 - .../cdh/v5_11_0/resources/kafka-kafka_broker.json | 566 -- .../resources/kafka-kafka_mirror_maker.json | 626 -- .../cdh/v5_11_0/resources/kafka-service.json | 404 -- sahara/plugins/cdh/v5_11_0/resources/kms-kms.json | 638 --- .../plugins/cdh/v5_11_0/resources/kms-service.json | 86 - .../resources/ks_indexer-hbase_indexer.json | 374 -- .../cdh/v5_11_0/resources/ks_indexer-service.json | 188 - .../plugins/cdh/v5_11_0/resources/mapred-site.xml | 157 - .../cdh/v5_11_0/resources/oozie-oozie_server.json | 674 --- .../cdh/v5_11_0/resources/oozie-service.json | 260 - .../cdh/v5_11_0/resources/priority-one-confs.json | 45 - .../cdh/v5_11_0/resources/sentry-gateway.json | 62 - .../resources/sentry-impala-client-safety.xml | 16 - .../v5_11_0/resources/sentry-sentry_server.json | 320 -- .../cdh/v5_11_0/resources/sentry-service.json | 230 - .../cdh/v5_11_0/resources/solr-gateway.json | 50 - .../cdh/v5_11_0/resources/solr-service.json | 350 -- .../cdh/v5_11_0/resources/solr-solr_server.json | 464 -- .../cdh/v5_11_0/resources/spark-gateway.json | 242 - .../cdh/v5_11_0/resources/spark-service.json | 158 - .../resources/spark-spark_yarn_history_server.json | 404 -- .../cdh/v5_11_0/resources/sqoop-service.json | 110 - .../cdh/v5_11_0/resources/sqoop-sqoop_server.json | 410 -- .../cdh/v5_11_0/resources/yarn-gateway.json | 506 -- .../cdh/v5_11_0/resources/yarn-jobhistory.json | 494 -- .../cdh/v5_11_0/resources/yarn-nodemanager.json | 794 --- .../v5_11_0/resources/yarn-resourcemanager.json | 686 --- .../cdh/v5_11_0/resources/yarn-service.json | 554 -- .../cdh/v5_11_0/resources/zookeeper-server.json | 608 -- .../cdh/v5_11_0/resources/zookeeper-service.json | 236 - sahara/plugins/cdh/v5_11_0/versionhandler.py | 37 - sahara/plugins/cdh/v5_13_0/__init__.py | 0 sahara/plugins/cdh/v5_13_0/cloudera_utils.py | 34 - sahara/plugins/cdh/v5_13_0/config_helper.py | 89 - sahara/plugins/cdh/v5_13_0/deploy.py | 168 - sahara/plugins/cdh/v5_13_0/edp_engine.py | 47 - sahara/plugins/cdh/v5_13_0/images.py | 43 - sahara/plugins/cdh/v5_13_0/resources/cdh_config.py | 68 - .../plugins/cdh/v5_13_0/resources/flume-agent.json | 440 -- .../cdh/v5_13_0/resources/flume-service.json | 164 - .../cdh/v5_13_0/resources/hbase-gateway.json | 122 - .../v5_13_0/resources/hbase-hbaserestserver.json | 452 -- .../v5_13_0/resources/hbase-hbasethriftserver.json | 458 -- .../cdh/v5_13_0/resources/hbase-master.json | 530 -- .../cdh/v5_13_0/resources/hbase-regionserver.json | 884 --- .../cdh/v5_13_0/resources/hbase-service.json | 692 --- .../cdh/v5_13_0/resources/hdfs-balancer.json | 128 - .../cdh/v5_13_0/resources/hdfs-datanode.json | 656 --- .../v5_13_0/resources/hdfs-failovercontroller.json | 344 -- .../cdh/v5_13_0/resources/hdfs-gateway.json | 116 - .../plugins/cdh/v5_13_0/resources/hdfs-httpfs.json | 464 -- .../cdh/v5_13_0/resources/hdfs-journalnode.json | 458 -- .../cdh/v5_13_0/resources/hdfs-namenode.json | 884 --- .../cdh/v5_13_0/resources/hdfs-nfsgateway.json | 410 -- .../v5_13_0/resources/hdfs-secondarynamenode.json | 458 -- .../cdh/v5_13_0/resources/hdfs-service.json | 1352 ----- .../cdh/v5_13_0/resources/hive-gateway.json | 98 - .../cdh/v5_13_0/resources/hive-hivemetastore.json | 434 -- .../cdh/v5_13_0/resources/hive-hiveserver2.json | 866 --- .../resources/hive-metastore-sentry-safety.xml | 15 - .../resources/hive-server2-sentry-safety.xml | 12 - .../cdh/v5_13_0/resources/hive-service.json | 656 --- sahara/plugins/cdh/v5_13_0/resources/hive-site.xml | 61 - .../cdh/v5_13_0/resources/hive-webhcat.json | 356 -- .../v5_13_0/resources/hue-hue_load_balancer.json | 260 - .../cdh/v5_13_0/resources/hue-hue_server.json | 404 -- .../cdh/v5_13_0/resources/hue-kt_renewer.json | 212 - .../plugins/cdh/v5_13_0/resources/hue-service.json | 692 --- .../resources/images/centos/disable_firewall | 20 - .../resources/images/centos/selinux_permissive | 13 - .../resources/images/centos/turn_off_services | 33 - .../cdh/v5_13_0/resources/images/centos/wget_repo | 27 - .../cdh/v5_13_0/resources/images/common/add_jar | 22 - .../resources/images/common/install_cloudera | 17 - .../v5_13_0/resources/images/common/install_extjs | 30 - .../images/common/unlimited_security_artifacts | 11 - .../cdh/v5_13_0/resources/images/image.yaml | 128 - .../resources/images/ubuntu/turn_off_services | 33 - .../cdh/v5_13_0/resources/images/ubuntu/wget_repo | 31 - .../v5_13_0/resources/impala-catalogserver.json | 542 -- .../cdh/v5_13_0/resources/impala-impalad.json | 746 --- .../cdh/v5_13_0/resources/impala-llama.json | 518 -- .../cdh/v5_13_0/resources/impala-service.json | 572 -- .../cdh/v5_13_0/resources/impala-statestore.json | 404 -- .../cdh/v5_13_0/resources/kafka-gateway.json | 62 - .../cdh/v5_13_0/resources/kafka-kafka_broker.json | 584 -- .../resources/kafka-kafka_mirror_maker.json | 644 --- .../cdh/v5_13_0/resources/kafka-service.json | 404 -- sahara/plugins/cdh/v5_13_0/resources/kms-kms.json | 656 --- .../plugins/cdh/v5_13_0/resources/kms-service.json | 86 - .../resources/ks_indexer-hbase_indexer.json | 374 -- .../cdh/v5_13_0/resources/ks_indexer-service.json | 188 - .../plugins/cdh/v5_13_0/resources/mapred-site.xml | 157 - .../cdh/v5_13_0/resources/oozie-oozie_server.json | 674 --- .../cdh/v5_13_0/resources/oozie-service.json | 260 - .../cdh/v5_13_0/resources/priority-one-confs.json | 45 - .../cdh/v5_13_0/resources/sentry-gateway.json | 62 - .../resources/sentry-impala-client-safety.xml | 16 - .../v5_13_0/resources/sentry-sentry_server.json | 320 -- .../cdh/v5_13_0/resources/sentry-service.json | 230 - .../cdh/v5_13_0/resources/solr-gateway.json | 50 - .../cdh/v5_13_0/resources/solr-service.json | 350 -- .../cdh/v5_13_0/resources/solr-solr_server.json | 470 -- .../cdh/v5_13_0/resources/spark-gateway.json | 242 - .../cdh/v5_13_0/resources/spark-service.json | 170 - .../resources/spark-spark_yarn_history_server.json | 422 -- .../cdh/v5_13_0/resources/sqoop-service.json | 116 - .../cdh/v5_13_0/resources/sqoop-sqoop_server.json | 410 -- .../v5_13_0/resources/sqoop_client-gateway.json | 92 - .../v5_13_0/resources/sqoop_client-service.json | 80 - .../cdh/v5_13_0/resources/yarn-gateway.json | 530 -- .../cdh/v5_13_0/resources/yarn-jobhistory.json | 500 -- .../cdh/v5_13_0/resources/yarn-nodemanager.json | 794 --- .../v5_13_0/resources/yarn-resourcemanager.json | 686 --- .../cdh/v5_13_0/resources/yarn-service.json | 554 -- .../cdh/v5_13_0/resources/zookeeper-server.json | 608 -- .../cdh/v5_13_0/resources/zookeeper-service.json | 236 - sahara/plugins/cdh/v5_13_0/versionhandler.py | 37 - sahara/plugins/cdh/v5_7_0/__init__.py | 0 sahara/plugins/cdh/v5_7_0/cloudera_utils.py | 28 - sahara/plugins/cdh/v5_7_0/config_helper.py | 101 - sahara/plugins/cdh/v5_7_0/deploy.py | 168 - sahara/plugins/cdh/v5_7_0/edp_engine.py | 47 - sahara/plugins/cdh/v5_7_0/images.py | 43 - sahara/plugins/cdh/v5_7_0/resources/cdh_config.py | 139 - sahara/plugins/cdh/v5_7_0/resources/cdh_config.sh | 3 - .../plugins/cdh/v5_7_0/resources/flume-agent.json | 428 -- .../cdh/v5_7_0/resources/flume-service.json | 164 - .../cdh/v5_7_0/resources/hbase-gateway.json | 110 - .../v5_7_0/resources/hbase-hbaserestserver.json | 440 -- .../v5_7_0/resources/hbase-hbasethriftserver.json | 446 -- .../plugins/cdh/v5_7_0/resources/hbase-master.json | 506 -- .../cdh/v5_7_0/resources/hbase-regionserver.json | 836 --- .../cdh/v5_7_0/resources/hbase-service.json | 740 --- .../cdh/v5_7_0/resources/hdfs-balancer.json | 68 - .../cdh/v5_7_0/resources/hdfs-datanode.json | 644 --- .../v5_7_0/resources/hdfs-failovercontroller.json | 332 -- .../plugins/cdh/v5_7_0/resources/hdfs-gateway.json | 116 - .../plugins/cdh/v5_7_0/resources/hdfs-httpfs.json | 440 -- .../cdh/v5_7_0/resources/hdfs-journalnode.json | 446 -- .../cdh/v5_7_0/resources/hdfs-namenode.json | 812 --- .../cdh/v5_7_0/resources/hdfs-nfsgateway.json | 386 -- .../v5_7_0/resources/hdfs-secondarynamenode.json | 446 -- .../plugins/cdh/v5_7_0/resources/hdfs-service.json | 1400 ----- .../plugins/cdh/v5_7_0/resources/hive-gateway.json | 98 - .../cdh/v5_7_0/resources/hive-hivemetastore.json | 392 -- .../cdh/v5_7_0/resources/hive-hiveserver2.json | 704 --- .../resources/hive-metastore-sentry-safety.xml | 15 - .../resources/hive-server2-sentry-safety.xml | 12 - .../plugins/cdh/v5_7_0/resources/hive-service.json | 722 --- sahara/plugins/cdh/v5_7_0/resources/hive-site.xml | 61 - .../plugins/cdh/v5_7_0/resources/hive-webhcat.json | 332 -- .../v5_7_0/resources/hue-hue_load_balancer.json | 236 - .../cdh/v5_7_0/resources/hue-hue_server.json | 356 -- .../cdh/v5_7_0/resources/hue-kt_renewer.json | 200 - .../plugins/cdh/v5_7_0/resources/hue-service.json | 674 --- .../resources/images/centos/disable_firewall | 20 - .../resources/images/centos/selinux_permissive | 13 - .../resources/images/centos/turn_off_services | 33 - .../cdh/v5_7_0/resources/images/centos/wget_repo | 33 - .../cdh/v5_7_0/resources/images/common/add_jar | 22 - .../resources/images/common/install_cloudera | 17 - .../v5_7_0/resources/images/common/install_extjs | 30 - .../images/common/unlimited_security_artifacts | 11 - .../plugins/cdh/v5_7_0/resources/images/image.yaml | 132 - .../resources/images/ubuntu/turn_off_services | 33 - .../cdh/v5_7_0/resources/images/ubuntu/wget_repo | 36 - .../cdh/v5_7_0/resources/impala-catalogserver.json | 506 -- .../cdh/v5_7_0/resources/impala-impalad.json | 710 --- .../plugins/cdh/v5_7_0/resources/impala-llama.json | 506 -- .../cdh/v5_7_0/resources/impala-service.json | 632 --- .../cdh/v5_7_0/resources/impala-statestore.json | 374 -- .../cdh/v5_7_0/resources/kafka-kafka_broker.json | 476 -- .../v5_7_0/resources/kafka-kafka_mirror_maker.json | 482 -- .../cdh/v5_7_0/resources/kafka-service.json | 374 -- sahara/plugins/cdh/v5_7_0/resources/kms-kms.json | 530 -- .../plugins/cdh/v5_7_0/resources/kms-service.json | 86 - .../v5_7_0/resources/ks_indexer-hbase_indexer.json | 356 -- .../cdh/v5_7_0/resources/ks_indexer-service.json | 188 - .../plugins/cdh/v5_7_0/resources/mapred-site.xml | 157 - .../cdh/v5_7_0/resources/oozie-oozie_server.json | 662 --- .../cdh/v5_7_0/resources/oozie-service.json | 236 - .../cdh/v5_7_0/resources/priority-one-confs.json | 45 - .../resources/sentry-impala-client-safety.xml | 16 - .../cdh/v5_7_0/resources/sentry-sentry_server.json | 308 - .../cdh/v5_7_0/resources/sentry-service.json | 278 - .../plugins/cdh/v5_7_0/resources/solr-gateway.json | 32 - .../plugins/cdh/v5_7_0/resources/solr-service.json | 404 -- .../cdh/v5_7_0/resources/solr-solr_server.json | 440 -- .../cdh/v5_7_0/resources/spark-gateway.json | 170 - .../cdh/v5_7_0/resources/spark-service.json | 152 - .../resources/spark-spark_yarn_history_server.json | 314 -- .../cdh/v5_7_0/resources/sqoop-service.json | 110 - .../cdh/v5_7_0/resources/sqoop-sqoop_server.json | 398 -- .../plugins/cdh/v5_7_0/resources/yarn-gateway.json | 476 -- .../cdh/v5_7_0/resources/yarn-jobhistory.json | 476 -- .../cdh/v5_7_0/resources/yarn-nodemanager.json | 764 --- .../cdh/v5_7_0/resources/yarn-resourcemanager.json | 674 --- .../plugins/cdh/v5_7_0/resources/yarn-service.json | 542 -- .../cdh/v5_7_0/resources/zookeeper-server.json | 596 -- .../cdh/v5_7_0/resources/zookeeper-service.json | 224 - sahara/plugins/cdh/v5_7_0/versionhandler.py | 37 - sahara/plugins/cdh/v5_9_0/__init__.py | 0 sahara/plugins/cdh/v5_9_0/cloudera_utils.py | 28 - sahara/plugins/cdh/v5_9_0/config_helper.py | 101 - sahara/plugins/cdh/v5_9_0/deploy.py | 168 - sahara/plugins/cdh/v5_9_0/edp_engine.py | 47 - sahara/plugins/cdh/v5_9_0/images.py | 43 - sahara/plugins/cdh/v5_9_0/plugin_utils.py | 23 - sahara/plugins/cdh/v5_9_0/resources/cdh_config.py | 139 - sahara/plugins/cdh/v5_9_0/resources/cdh_config.sh | 3 - .../plugins/cdh/v5_9_0/resources/flume-agent.json | 428 -- .../cdh/v5_9_0/resources/flume-service.json | 164 - .../cdh/v5_9_0/resources/hbase-gateway.json | 110 - .../v5_9_0/resources/hbase-hbaserestserver.json | 440 -- .../v5_9_0/resources/hbase-hbasethriftserver.json | 446 -- .../plugins/cdh/v5_9_0/resources/hbase-master.json | 506 -- .../cdh/v5_9_0/resources/hbase-regionserver.json | 842 --- .../cdh/v5_9_0/resources/hbase-service.json | 746 --- .../cdh/v5_9_0/resources/hdfs-balancer.json | 68 - .../cdh/v5_9_0/resources/hdfs-datanode.json | 644 --- .../v5_9_0/resources/hdfs-failovercontroller.json | 332 -- .../plugins/cdh/v5_9_0/resources/hdfs-gateway.json | 116 - .../plugins/cdh/v5_9_0/resources/hdfs-httpfs.json | 440 -- .../cdh/v5_9_0/resources/hdfs-journalnode.json | 446 -- .../cdh/v5_9_0/resources/hdfs-namenode.json | 854 --- .../cdh/v5_9_0/resources/hdfs-nfsgateway.json | 386 -- .../v5_9_0/resources/hdfs-secondarynamenode.json | 446 -- .../plugins/cdh/v5_9_0/resources/hdfs-service.json | 1418 ----- .../plugins/cdh/v5_9_0/resources/hive-gateway.json | 98 - .../cdh/v5_9_0/resources/hive-hivemetastore.json | 416 -- .../cdh/v5_9_0/resources/hive-hiveserver2.json | 806 --- .../resources/hive-metastore-sentry-safety.xml | 15 - .../resources/hive-server2-sentry-safety.xml | 12 - .../plugins/cdh/v5_9_0/resources/hive-service.json | 758 --- sahara/plugins/cdh/v5_9_0/resources/hive-site.xml | 61 - .../plugins/cdh/v5_9_0/resources/hive-webhcat.json | 332 -- .../v5_9_0/resources/hue-hue_load_balancer.json | 236 - .../cdh/v5_9_0/resources/hue-hue_server.json | 368 -- .../cdh/v5_9_0/resources/hue-kt_renewer.json | 200 - .../plugins/cdh/v5_9_0/resources/hue-service.json | 674 --- .../resources/images/centos/disable_firewall | 20 - .../resources/images/centos/selinux_permissive | 13 - .../resources/images/centos/turn_off_services | 33 - .../cdh/v5_9_0/resources/images/centos/wget_repo | 33 - .../cdh/v5_9_0/resources/images/common/add_jar | 22 - .../resources/images/common/install_cloudera | 17 - .../v5_9_0/resources/images/common/install_extjs | 30 - .../images/common/unlimited_security_artifacts | 11 - .../plugins/cdh/v5_9_0/resources/images/image.yaml | 132 - .../resources/images/ubuntu/turn_off_services | 33 - .../cdh/v5_9_0/resources/images/ubuntu/wget_repo | 36 - .../cdh/v5_9_0/resources/impala-catalogserver.json | 530 -- .../cdh/v5_9_0/resources/impala-impalad.json | 734 --- .../plugins/cdh/v5_9_0/resources/impala-llama.json | 506 -- .../cdh/v5_9_0/resources/impala-service.json | 644 --- .../cdh/v5_9_0/resources/impala-statestore.json | 392 -- .../cdh/v5_9_0/resources/kafka-kafka_broker.json | 476 -- .../v5_9_0/resources/kafka-kafka_mirror_maker.json | 482 -- .../cdh/v5_9_0/resources/kafka-service.json | 374 -- sahara/plugins/cdh/v5_9_0/resources/kms-kms.json | 530 -- .../plugins/cdh/v5_9_0/resources/kms-service.json | 86 - .../v5_9_0/resources/ks_indexer-hbase_indexer.json | 362 -- .../cdh/v5_9_0/resources/ks_indexer-service.json | 188 - .../plugins/cdh/v5_9_0/resources/mapred-site.xml | 157 - .../cdh/v5_9_0/resources/oozie-oozie_server.json | 662 --- .../cdh/v5_9_0/resources/oozie-service.json | 260 - .../cdh/v5_9_0/resources/priority-one-confs.json | 45 - .../cdh/v5_9_0/resources/sentry-gateway.json | 62 - .../resources/sentry-impala-client-safety.xml | 16 - .../cdh/v5_9_0/resources/sentry-sentry_server.json | 308 - .../cdh/v5_9_0/resources/sentry-service.json | 290 - .../plugins/cdh/v5_9_0/resources/solr-gateway.json | 50 - .../plugins/cdh/v5_9_0/resources/solr-service.json | 416 -- .../cdh/v5_9_0/resources/solr-solr_server.json | 452 -- .../cdh/v5_9_0/resources/spark-gateway.json | 176 - .../cdh/v5_9_0/resources/spark-service.json | 152 - .../resources/spark-spark_yarn_history_server.json | 314 -- .../cdh/v5_9_0/resources/sqoop-service.json | 110 - .../cdh/v5_9_0/resources/sqoop-sqoop_server.json | 398 -- .../plugins/cdh/v5_9_0/resources/yarn-gateway.json | 506 -- .../cdh/v5_9_0/resources/yarn-jobhistory.json | 476 -- .../cdh/v5_9_0/resources/yarn-nodemanager.json | 770 --- .../cdh/v5_9_0/resources/yarn-resourcemanager.json | 674 --- .../plugins/cdh/v5_9_0/resources/yarn-service.json | 632 --- .../cdh/v5_9_0/resources/zookeeper-server.json | 596 -- .../cdh/v5_9_0/resources/zookeeper-service.json | 224 - sahara/plugins/cdh/v5_9_0/versionhandler.py | 37 - sahara/plugins/cdh/validation.py | 438 -- sahara/plugins/cdh/versionfactory.py | 57 - sahara/plugins/conductor.py | 38 + sahara/plugins/context.py | 74 + sahara/plugins/db.py | 29 + sahara/plugins/edp.py | 96 + sahara/plugins/exceptions.py | 26 + sahara/plugins/fake/edp_engine.py | 15 +- sahara/plugins/fake/plugin.py | 3 +- .../health => plugins}/health_check_base.py | 0 sahara/plugins/images.py | 49 +- .../plugins/{cdh/v5_13_0/validation.py => main.py} | 13 +- sahara/plugins/mapr/__init__.py | 0 sahara/plugins/mapr/abstract/__init__.py | 0 sahara/plugins/mapr/abstract/cluster_context.py | 157 - sahara/plugins/mapr/abstract/cluster_validator.py | 29 - sahara/plugins/mapr/abstract/configurer.py | 29 - sahara/plugins/mapr/abstract/health_checker.py | 24 - sahara/plugins/mapr/abstract/node_manager.py | 37 - sahara/plugins/mapr/abstract/version_handler.py | 81 - sahara/plugins/mapr/base/__init__.py | 0 .../plugins/mapr/base/base_cluster_configurer.py | 405 -- sahara/plugins/mapr/base/base_cluster_context.py | 444 -- sahara/plugins/mapr/base/base_cluster_validator.py | 35 - sahara/plugins/mapr/base/base_edp_engine.py | 94 - sahara/plugins/mapr/base/base_health_checker.py | 121 - sahara/plugins/mapr/base/base_node_manager.py | 200 - sahara/plugins/mapr/base/base_version_handler.py | 198 - sahara/plugins/mapr/domain/__init__.py | 0 sahara/plugins/mapr/domain/configuration_file.py | 194 - sahara/plugins/mapr/domain/distro.py | 96 - sahara/plugins/mapr/domain/node_process.py | 177 - sahara/plugins/mapr/domain/service.py | 246 - sahara/plugins/mapr/images.py | 43 - sahara/plugins/mapr/plugin.py | 110 - sahara/plugins/mapr/resources/add_mapr_repo.sh | 37 - .../plugins/mapr/resources/add_security_repos.sh | 6 - .../mapr/resources/create_disk_list_file.sh | 13 - .../mapr/resources/images/centos/disable_firewall | 20 - .../mapr/resources/images/centos/epel_dependencies | 6 - .../images/centos/install_mapr_core_repository | 27 - .../images/centos/install_mapr_eco_repository | 27 - .../mapr/resources/images/centos/install_openjdk | 14 - .../mapr/resources/images/centos/install_scala | 34 - .../mapr/resources/images/centos/requiretty | 6 - .../resources/images/centos/selinux_permissive | 12 - .../mapr/resources/images/centos/update_repository | 5 - .../mapr/resources/images/common/configure_extjs | 31 - .../mapr/resources/images/common/oracle_java | 42 - .../images/common/resources/package_utils.sh | 226 - .../images/common/resources/packages.json | 140 - .../images/common/resources/spec_5.1.0.json | 46 - .../images/common/resources/spec_5.2.0.json | 50 - .../images/common/resources/spec_5.2.0_ubuntu.json | 47 - .../resources/images/common/resources/versions.py | 83 - sahara/plugins/mapr/resources/images/image.yaml | 205 - .../images/ubuntu/install_mapr_core_repository | 27 - .../images/ubuntu/install_mapr_dependencies | 22 - .../images/ubuntu/install_mapr_eco_repository | 32 - .../mapr/resources/images/ubuntu/install_openjdk | 16 - .../mapr/resources/images/ubuntu/install_scala | 37 - .../mapr/resources/images/ubuntu/update_repository | 5 - sahara/plugins/mapr/resources/install_java.sh | 69 - sahara/plugins/mapr/resources/install_mysql.sh | 31 - .../plugins/mapr/resources/install_mysql_client.sh | 12 - sahara/plugins/mapr/resources/install_scala.sh | 28 - sahara/plugins/mapr/resources/topology.sh | 20 - sahara/plugins/mapr/services/__init__.py | 0 sahara/plugins/mapr/services/drill/__init__.py | 0 sahara/plugins/mapr/services/drill/drill.py | 112 - sahara/plugins/mapr/services/flume/__init__.py | 0 sahara/plugins/mapr/services/flume/flume.py | 47 - sahara/plugins/mapr/services/hbase/__init__.py | 0 sahara/plugins/mapr/services/hbase/hbase.py | 120 - .../services/hbase/resources/hbase-default.json | 31 - sahara/plugins/mapr/services/hive/__init__.py | 0 sahara/plugins/mapr/services/hive/hive.py | 184 - .../mapr/services/hive/resources/hive-default.xml | 59 - .../mapr/services/hive/resources/sentry-db.xml | 27 - .../services/hive/resources/sentry-default.xml | 45 - sahara/plugins/mapr/services/httpfs/__init__.py | 0 sahara/plugins/mapr/services/httpfs/httpfs.py | 40 - .../services/httpfs/resources/httpfs-default.json | 12 - sahara/plugins/mapr/services/hue/__init__.py | 0 sahara/plugins/mapr/services/hue/hue.py | 365 -- .../services/hue/resources/hue_3.10.0.template | 1403 ----- .../mapr/services/hue/resources/hue_3.6.0.template | 947 ---- .../mapr/services/hue/resources/hue_3.7.0.template | 1001 ---- .../mapr/services/hue/resources/hue_3.8.1.template | 1143 ---- .../mapr/services/hue/resources/hue_3.9.0.template | 1184 ---- .../services/hue/resources/hue_sh_3.10.0.template | 130 - .../services/hue/resources/hue_sh_3.7.0.template | 116 - .../services/hue/resources/hue_sh_3.8.1.template | 130 - .../services/hue/resources/hue_sh_3.9.0.template | 130 - .../hue/resources/livy_conf_3.10.0.template | 80 - .../services/hue/resources/livy_sh_3.10.0.template | 35 - sahara/plugins/mapr/services/impala/__init__.py | 0 sahara/plugins/mapr/services/impala/impala.py | 192 - .../services/impala/resources/impala-env.sh.j2 | 64 - sahara/plugins/mapr/services/kafka/__init__.py | 0 sahara/plugins/mapr/services/kafka/kafka.py | 73 - sahara/plugins/mapr/services/mahout/__init__.py | 0 sahara/plugins/mapr/services/mahout/mahout.py | 58 - .../plugins/mapr/services/management/__init__.py | 0 .../plugins/mapr/services/management/management.py | 63 - sahara/plugins/mapr/services/mapreduce/__init__.py | 0 .../plugins/mapr/services/mapreduce/mapreduce.py | 127 - .../mapreduce/resources/mapreduce-cluster.json | 91 - sahara/plugins/mapr/services/maprfs/__init__.py | 0 sahara/plugins/mapr/services/maprfs/maprfs.py | 157 - .../mapr/services/maprfs/resources/cldb.conf | 63 - sahara/plugins/mapr/services/mysql/__init__.py | 0 sahara/plugins/mapr/services/mysql/mysql.py | 207 - .../services/mysql/resources/create_database.sql | 7 - .../mapr/services/mysql/resources/grant_access.sql | 6 - .../mapr/services/mysql/resources/hive_schema.sql | 2 - sahara/plugins/mapr/services/oozie/__init__.py | 0 sahara/plugins/mapr/services/oozie/oozie.py | 168 - .../services/oozie/resources/oozie-default.json | 28 - sahara/plugins/mapr/services/pig/__init__.py | 0 sahara/plugins/mapr/services/pig/pig.py | 57 - sahara/plugins/mapr/services/sentry/__init__.py | 0 .../services/sentry/resources/global-policy.ini | 6 - .../services/sentry/resources/sentry-default.xml | 4 - sahara/plugins/mapr/services/sentry/sentry.py | 171 - sahara/plugins/mapr/services/spark/__init__.py | 0 .../services/spark/resources/spark-env.template | 134 - sahara/plugins/mapr/services/spark/spark.py | 316 -- sahara/plugins/mapr/services/sqoop/__init__.py | 0 sahara/plugins/mapr/services/sqoop/sqoop2.py | 46 - sahara/plugins/mapr/services/swift/__init__.py | 0 .../swift/resources/hadoop-swift-latest.jar | Bin 117363 -> 0 bytes .../services/swift/resources/swift-default.json | 61 - sahara/plugins/mapr/services/swift/swift.py | 58 - sahara/plugins/mapr/services/yarn/__init__.py | 0 .../mapr/services/yarn/resources/yarn-cluster.json | 16 - .../mapr/services/yarn/resources/yarn-node.json | 23 - sahara/plugins/mapr/services/yarn/yarn.py | 144 - sahara/plugins/mapr/util/__init__.py | 0 sahara/plugins/mapr/util/commands.py | 25 - sahara/plugins/mapr/util/event_log.py | 63 - sahara/plugins/mapr/util/general.py | 204 - sahara/plugins/mapr/util/maprfs_helper.py | 108 - sahara/plugins/mapr/util/password_utils.py | 69 - sahara/plugins/mapr/util/service_utils.py | 51 - sahara/plugins/mapr/util/validation_utils.py | 262 - sahara/plugins/mapr/versions/__init__.py | 0 .../plugins/mapr/versions/v5_2_0_mrv2/__init__.py | 0 .../plugins/mapr/versions/v5_2_0_mrv2/context.py | 61 - .../mapr/versions/v5_2_0_mrv2/version_handler.py | 75 - .../mapr/versions/version_handler_factory.py | 57 - .../{cdh/v5_9_0/validation.py => objects.py} | 9 +- sahara/plugins/provisioning.py | 1 + .../test_plugin_utils.py => plugins/resource.py} | 23 +- .../{cdh/v5_7_0/validation.py => service_api.py} | 9 +- sahara/plugins/spark/__init__.py | 0 sahara/plugins/spark/config_helper.py | 517 -- sahara/plugins/spark/edp_engine.py | 62 - sahara/plugins/spark/plugin.py | 575 -- sahara/plugins/spark/resources/README.rst | 21 - sahara/plugins/spark/resources/core-default.xml | 632 --- sahara/plugins/spark/resources/hdfs-default.xml | 709 --- sahara/plugins/spark/resources/spark-cleanup.cron | 2 - .../plugins/spark/resources/spark-env.sh.template | 21 - .../spark/resources/tmp-cleanup.sh.template | 48 - sahara/plugins/spark/resources/topology.sh | 21 - sahara/plugins/spark/run_scripts.py | 92 - sahara/plugins/spark/scaling.py | 111 - sahara/plugins/spark/shell_engine.py | 28 - sahara/plugins/storm/__init__.py | 0 sahara/plugins/storm/config_helper.py | 159 - sahara/plugins/storm/edp_engine.py | 52 - sahara/plugins/storm/plugin.py | 410 -- sahara/plugins/storm/run_scripts.py | 54 - .../v5_13_0/plugin_utils.py => swift_helper.py} | 17 +- .../{cdh/v5_11_0/validation.py => swift_utils.py} | 9 +- sahara/plugins/testutils.py | 34 + .../topology_helper.py} | 22 +- sahara/plugins/utils.py | 149 +- sahara/plugins/vanilla/__init__.py | 0 sahara/plugins/vanilla/abstractversionhandler.py | 71 - sahara/plugins/vanilla/confighints_helper.py | 45 - sahara/plugins/vanilla/edp_engine.py | 40 - sahara/plugins/vanilla/hadoop2/__init__.py | 0 sahara/plugins/vanilla/hadoop2/config.py | 479 -- sahara/plugins/vanilla/hadoop2/config_helper.py | 329 -- sahara/plugins/vanilla/hadoop2/keypairs.py | 94 - sahara/plugins/vanilla/hadoop2/oozie_helper.py | 51 - .../vanilla/hadoop2/recommendations_utils.py | 48 - .../vanilla/hadoop2/resources/create_oozie_db.sql | 5 - .../vanilla/hadoop2/resources/post_conf.template | 33 - .../vanilla/hadoop2/resources/spark-cleanup.cron | 2 - .../hadoop2/resources/tmp-cleanup.sh.template | 48 - .../plugins/vanilla/hadoop2/resources/topology.sh | 21 - .../vanilla/hadoop2/resources/zoo_sample.cfg | 29 - sahara/plugins/vanilla/hadoop2/run_scripts.py | 293 - sahara/plugins/vanilla/hadoop2/scaling.py | 147 - sahara/plugins/vanilla/hadoop2/starting_scripts.py | 85 - sahara/plugins/vanilla/hadoop2/utils.py | 89 - sahara/plugins/vanilla/hadoop2/validation.py | 158 - sahara/plugins/vanilla/plugin.py | 108 - sahara/plugins/vanilla/utils.py | 69 - sahara/plugins/vanilla/v2_7_1/__init__.py | 0 sahara/plugins/vanilla/v2_7_1/config_helper.py | 151 - sahara/plugins/vanilla/v2_7_1/edp_engine.py | 81 - sahara/plugins/vanilla/v2_7_1/resources/README.rst | 28 - .../vanilla/v2_7_1/resources/core-default.xml | 1902 ------- .../vanilla/v2_7_1/resources/create_hive_db.sql | 9 - .../vanilla/v2_7_1/resources/hdfs-default.xml | 2323 -------- .../vanilla/v2_7_1/resources/hive-default.xml | 1873 ------ .../vanilla/v2_7_1/resources/mapred-default.xml | 2128 ------- .../vanilla/v2_7_1/resources/oozie-default.xml | 2546 --------- .../vanilla/v2_7_1/resources/yarn-default.xml | 1623 ------ sahara/plugins/vanilla/v2_7_1/versionhandler.py | 170 - sahara/plugins/vanilla/v2_7_5/__init__.py | 0 sahara/plugins/vanilla/v2_7_5/config_helper.py | 151 - sahara/plugins/vanilla/v2_7_5/edp_engine.py | 81 - sahara/plugins/vanilla/v2_7_5/resources/README.rst | 27 - .../vanilla/v2_7_5/resources/core-default.xml | 1997 ------- .../vanilla/v2_7_5/resources/create_hive_db.sql | 9 - .../vanilla/v2_7_5/resources/hdfs-default.xml | 2677 --------- .../vanilla/v2_7_5/resources/hive-default.xml | 5959 -------------------- .../vanilla/v2_7_5/resources/mapred-default.xml | 2221 -------- .../vanilla/v2_7_5/resources/oozie-default.xml | 2843 ---------- .../vanilla/v2_7_5/resources/yarn-default.xml | 1713 ------ sahara/plugins/vanilla/v2_7_5/versionhandler.py | 170 - sahara/plugins/vanilla/v2_8_2/__init__.py | 0 sahara/plugins/vanilla/v2_8_2/config_helper.py | 151 - sahara/plugins/vanilla/v2_8_2/edp_engine.py | 81 - sahara/plugins/vanilla/v2_8_2/resources/README.rst | 27 - .../vanilla/v2_8_2/resources/core-default.xml | 2414 -------- .../vanilla/v2_8_2/resources/create_hive_db.sql | 9 - .../vanilla/v2_8_2/resources/hdfs-default.xml | 3138 ----------- .../vanilla/v2_8_2/resources/hive-default.xml | 5959 -------------------- .../vanilla/v2_8_2/resources/mapred-default.xml | 1892 ------- .../vanilla/v2_8_2/resources/oozie-default.xml | 2843 ---------- .../vanilla/v2_8_2/resources/yarn-default.xml | 2635 --------- sahara/plugins/vanilla/v2_8_2/versionhandler.py | 170 - sahara/plugins/vanilla/versionfactory.py | 57 - sahara/service/coordinator.py | 3 +- sahara/service/edp/hdfs_helper.py | 2 +- sahara/service/edp/job_manager.py | 2 +- sahara/service/edp/resources/launch_command.py | 1 + sahara/service/health/common.py | 2 +- sahara/service/health/verification_base.py | 2 +- sahara/service/heat/templates.py | 34 +- sahara/service/ntp_service.py | 4 +- sahara/service/ops.py | 45 +- sahara/service/quotas.py | 2 +- sahara/service/validation.py | 19 + sahara/service/validations/cluster_templates.py | 2 +- sahara/service/validations/clusters_schema.py | 8 +- .../service/validations/edp/job_binary_internal.py | 3 +- sahara/service/validations/edp/job_execution.py | 8 +- .../validations/node_group_template_schema.py | 9 + sahara/service/validations/shares.py | 2 +- sahara/swift/swift_helper.py | 2 +- .../unit/plugins/ambari/test_requests_helper.py | 96 - .../unit/plugins/cdh/base_config_helper_test.py | 187 - .../unit/plugins/cdh/base_plugin_utils_test.py | 323 -- .../unit/plugins/cdh/test_confighints_helper.py | 59 - .../unit/plugins/cdh/v5_11_0/test_config_helper.py | 25 - .../plugins/cdh/v5_11_0/test_config_helper_5110.py | 168 - .../unit/plugins/cdh/v5_11_0/test_edp_engine.py | 168 - .../unit/plugins/cdh/v5_11_0/test_spark_engine.py | 69 - .../unit/plugins/cdh/v5_11_0/test_validation.py | 24 - .../plugins/cdh/v5_11_0/test_versionhandler.py | 140 - .../unit/plugins/cdh/v5_7_0/test_config_helper.py | 25 - .../plugins/cdh/v5_7_0/test_config_helper_570.py | 168 - .../unit/plugins/cdh/v5_7_0/test_edp_engine.py | 200 - .../unit/plugins/cdh/v5_7_0/test_plugin_utils.py | 25 - .../plugins/cdh/v5_7_0/test_plugin_utils_570.py | 25 - .../unit/plugins/cdh/v5_7_0/test_spark_engine.py | 69 - .../unit/plugins/cdh/v5_7_0/test_validation.py | 24 - .../unit/plugins/cdh/v5_7_0/test_versionhandler.py | 140 - .../unit/plugins/cdh/v5_9_0/test_config_helper.py | 25 - .../plugins/cdh/v5_9_0/test_config_helper_590.py | 168 - .../unit/plugins/cdh/v5_9_0/test_edp_engine.py | 200 - .../unit/plugins/cdh/v5_9_0/test_plugin_utils.py | 25 - .../plugins/cdh/v5_9_0/test_plugin_utils_590.py | 25 - .../unit/plugins/cdh/v5_9_0/test_spark_engine.py | 69 - .../unit/plugins/cdh/v5_9_0/test_validation.py | 24 - .../unit/plugins/cdh/v5_9_0/test_versionhandler.py | 140 - .../unit/plugins/mapr/test_cluster_context.py | 284 - .../unit/plugins/test_base_plugins_support.py | 7 +- .../vanilla/hadoop2/resources/dfs-report.txt | 62 - .../vanilla/hadoop2/resources/yarn-report.txt | 6 - .../plugins/vanilla/hadoop2/test_config_helper.py | 196 - .../unit/plugins/vanilla/hadoop2/test_configs.py | 41 - .../plugins/vanilla/hadoop2/test_edp_engine.py | 89 - .../plugins/vanilla/hadoop2/test_oozie_helper.py | 66 - .../unit/plugins/vanilla/hadoop2/test_plugin.py | 47 - .../vanilla/hadoop2/test_recommendation_utils.py | 60 - .../plugins/vanilla/hadoop2/test_run_scripts.py | 313 - .../unit/plugins/vanilla/hadoop2/test_scaling.py | 259 - .../vanilla/hadoop2/test_starting_scripts.py | 152 - .../unit/plugins/vanilla/hadoop2/test_utils.py | 141 - .../plugins/vanilla/hadoop2/test_validation.py | 111 - .../plugins/vanilla/test_confighints_helper.py | 59 - .../plugins/vanilla/v2_7_1/test_config_helper.py | 73 - .../unit/plugins/vanilla/v2_7_1/test_edp_engine.py | 96 - .../plugins/vanilla/v2_7_1/test_versionhandler.py | 253 - .../plugins/vanilla/v2_7_5/test_config_helper.py | 73 - .../unit/plugins/vanilla/v2_7_5/test_edp_engine.py | 96 - .../plugins/vanilla/v2_7_5/test_versionhandler.py | 253 - .../plugins/vanilla/v2_8_2/test_config_helper.py | 73 - .../unit/plugins/vanilla/v2_8_2/test_edp_engine.py | 96 - .../plugins/vanilla/v2_8_2/test_versionhandler.py | 253 - .../internal_db/test_internal_db_type.py | 144 +- .../edp/job_binaries/job_binary_manager_support.py | 62 +- .../edp/job_binaries/manila/test_manila_type.py | 286 +- .../unit/service/edp/job_binaries/test_base.py | 18 +- .../unit/service/health/test_verification_base.py | 4 +- .../validation/edp/test_job_binary_internal.py | 2 +- .../service/validation/edp/test_job_executor.py | 36 +- .../service/validation/edp/test_job_interface.py | 1 + .../test_cluster_template_create_validation.py | 2 +- .../service/validation/test_share_validations.py | 4 +- sahara/utils/api.py | 89 +- sahara/utils/crypto.py | 4 + sahara/utils/files.py | 8 +- sahara/utils/hacking/checks.py | 7 +- sahara/utils/hacking/import_checks.py | 9 +- sahara/utils/openstack/keystone.py | 1 + sahara/utils/procutils.py | 8 +- sahara/utils/proxy.py | 2 +- sahara/utils/ssh_remote.py | 74 +- sahara/utils/xmlutils.py | 11 +- setup.cfg | 9 +- test-requirements.txt | 4 +- tools/gate/build-images | 5 +- tox.ini | 14 +- 985 files changed, 9723 insertions(+), 205579 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 6cea8b0d..28d95414 100644 --- a/requirements.txt +++ b/requirements.txt @@ -17,0 +18 @@ keystonemiddleware>=4.17.0 # Apache-2.0 +microversion-parse>=0.2.1 # Apache-2.0 @@ -29,0 +31 @@ oslo.service!=1.28.1,>=1.24.0 # Apache-2.0 +oslo.upgradecheck>=0.1.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 52c06260..12dde8e4 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -5 +5 @@ -hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 +hacking>=1.1.0 # Apache-2.0 @@ -16 +16 @@ stestr>=1.0.0 # Apache-2.0 -psycopg2>=2.6.2 # LGPL/ZPL +psycopg2>=2.7.3 # LGPL/ZPL