We are chuffed to announce the release of: openstack-cyborg 5.0.0: Distributed Acceleration Management as a Service This release is part of the victoria release series. The source is available from: https://opendev.org/openstack/openstack-cyborg Download the package from: https://tarballs.openstack.org/cyborg/ For more details, please see below. 5.0.0 ^^^^^ New Features ************ * In the Victoria release, cyborg introduced the new scoped RBAC policy authorization for API access, and partially implemented the blueprints. What implemented are new default rules in base policy and device_profile policy. During the development period(victoria and wallaby releases), the new and old policy will both work because a deployment sets "cyborg.conf [oslo_policy] enforce_scope = False" as the default set. Although users can set "cyborg.conf [oslo_policy] enforce_scope = True" by default in their deployment, if they want to ignore old rules and support new rules only. After we implement all the features, we'll give two cycles transition period for operators. For specification of new policy, please refer to policy default refresh. * Scope Cyborg introduced "scope_type" to protect each policy. Cyborg support two types of "sope_type" with their combination. "['system']", "['project']" and "['system', 'project']". To know each policy "scope_type", please refer the Policy Reference This feature is disabled by default can be enabled via config option "[oslo_policy]enforce_scope" in "cyborg.conf" * New Defaults Configuration Policies are default to Admin, Member and Reader roles. Old roles are also supproted. You can switch to new defaults via config option "[oslo_policy]enforce_new_defaults" in "cyborg.conf" file. * New Base policy roles Cyborg introduced seven basic roles based on the new defaults combined with different scope_types. * project_reader * project_member * project_admin * system_admin * system_reader * system_admin_or_owner * system_or_project_reader * New Defaults for device_profile APIs Rewrite check string(authorization rules) using new personas for device profile APIs. Add "checkstr=base.PROJECT_READER_OR_SYSTEM_READER" and deprecated "checkstr=base.deprecated_default" for * "cyborg:device_profile:get_one" * "cyborg:device_profile:get_all" Add "check_str=base.SYSTEM_ADMIN" and deprecated "check_str=base.deprecated_is_admin" for * "cyborg:device_profile:create" Add "check_str=base.SYSTEM_ADMIN" and deprecated "base.deprecated_default" for * "cyborg:device_profile:delete" * Added policy configuration guide on cyborg doc page Please refer to policy configuration guide (https://specs.openstack.org/openstack/cyborg- specs/specs/ussuri/approved/policy-defaults-refresh.html) (https://docs.openstack.org/cyborg/latest/configuration/policy.html) (https://docs.openstack.org/cyborg/latest/configuration/policy- guide.html) Upgrade Notes ************* * The default value of "[oslo_policy] policy_file" config option has been changed from "policy.json" to "policy.yaml". Cyborg policy new defaults since 5.0.0 and current default value of "[oslo_policy] policy_file" config option ("policy.json") does not work when "policy.json" is generated by oslopolicy-sample-generator (https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy- sample-generator.html) tool. Refer to bug 1875418 (https://bugs.launchpad.net/nova/+bug/1875418) for more details. Also check oslopolicy-convert-json-to-yaml (https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy- convert-json-to-yaml.html) tool to convert the JSON to YAML formatted policy file in backward compatible way. Deprecation Notes ***************** * The old basic personas below are marked as deprecated rules in base policy. * public_api * allow * deny * admin_api * is_admin * admin_or_owner * admin_or_user Changes in openstack-cyborg 4.0.0..5.0.0 ---------------------------------------- 62d7842 Add releasenote for policy refresh base and device profile policies e800b51 add document for cyborg new policy ffe409d Change default policy file from JSON to YAML c9a656d Update TOX_CONSTRAINTS_FILE for stable/victoria 98cebb1 Update .gitreview for stable/victoria 01e9a74 [goal] Fixing lower constraints for Ubuntu Focal edbc577 Programming support (v2 Deployable API) 5435d29 Add releasenote for programming method c053e5f Change arq delete to cyborg-conductor service write db 3b2faa0 Change arq create to cyborg-conductor service write db 9c1714b Introduce scope_types in Device Profiles Actions b1c223a Add new default roles in Device Profile API policies c39d180 Add releasenote for Intel QAT driver. 660b252 Suppress policy deprecated warnings in tests 270d12e Add new default rules and mapping in policy base class 27d5a7d Add policies tests for existing device_profile APIs 05f762b Add Inspur FPGA driver releasenote 61450fa Keep Inspur FPGA deployable_name unique 3108d28 Add Inspur FPGA driver fd18064 Add releasenote for add project_id for arq patch api 0154175 keep FPGA driver's deployable_name unique 970e673 Add Inspur FPGA driver matrix 3ade5f8 Add a new column in driver matrix eeae741 keep GPU driver's deployable_name unique de0823d Add instance project_id for arq patch 3fdb4f6 Correct spell error from `seperate` to `separate` d560ec1 Add a new driver for Intel QAT card 9179ed3 Optimize microversion support 40259a2 Revert "Add new default policy" c4f63f4 Add new default policy bf94ef8 update API documentation ce1cef3 Removed unused image api methods 9b99e56 Remove deprecated cyborg.image.download.modules 8826575 Remove api_servers references 59fc114 hacking: force explicit import of python's mock e6cdf3e Use uwsgi binary from path and docs job fix 6b0e1f8 Use unittest.mock instead of third party mock 8ed9135 Remove future imports 7ee7de1 Remove OPAE dependency in devstack 981d37c Enable openstackdocs config to storyboard b92dadb Always reset CONF when starting the wsgi app eef6ddd Switch to newer openstackdocstheme and reno versions 4561a16 Fix flake8 error from hacking <3.0.1 4153c1f add ut for get arqs list 4c941db Fix hacking min version to 3.0.1 1204f42 Monkey patch original current_thread _active 039b923 Fix fake attribute old interface 23eb486 optimize gpu driver generate attribute list 872b36f Fix copy-paste typo e045686 Remove translation sections from setup.cfg 771dab0 Update docs building cd97b0e Add py38 package metadata 68b7044 Add Python3 victoria unit tests dae645b Update master for stable/ussuri Diffstat (except docs and test files) ------------------------------------- .gitignore | 1 + .gitreview | 1 + .zuul.yaml | 2 +- api-ref/source/conf.py | 8 +- babel.cfg | 2 - bindep.txt | 7 + cyborg/accelerator/drivers/fpga/base.py | 3 +- cyborg/accelerator/drivers/fpga/inspur/__init__.py | 0 cyborg/accelerator/drivers/fpga/inspur/driver.py | 40 +++ cyborg/accelerator/drivers/fpga/inspur/sysinfo.py | 158 ++++++++++++ cyborg/accelerator/drivers/fpga/intel/driver.py | 30 +-- cyborg/accelerator/drivers/fpga/intel/sysinfo.py | 12 +- cyborg/accelerator/drivers/fpga/utils.py | 2 +- cyborg/accelerator/drivers/gpu/utils.py | 17 +- cyborg/accelerator/drivers/qat/__init__.py | 0 cyborg/accelerator/drivers/qat/base.py | 46 ++++ cyborg/accelerator/drivers/qat/intel/__init__.py | 0 cyborg/accelerator/drivers/qat/intel/driver.py | 36 +++ cyborg/accelerator/drivers/qat/intel/sysinfo.py | 204 +++++++++++++++ cyborg/agent/manager.py | 17 +- cyborg/agent/rpcapi.py | 10 +- cyborg/api/__init__.py | 18 ++ cyborg/api/controllers/v2/arqs.py | 39 ++- cyborg/api/controllers/v2/deployables.py | 66 ++++- cyborg/api/controllers/v2/device_profiles.py | 10 +- cyborg/api/controllers/v2/devices.py | 6 +- cyborg/api/controllers/v2/utils.py | 22 ++ cyborg/api/controllers/v2/versions.py | 23 +- cyborg/api/rest_api_version_history.rst | 19 +- cyborg/api/wsgi_app.py | 3 + cyborg/cmd/__init__.py | 6 + cyborg/cmd/status.py | 30 ++- cyborg/common/authorize_wsgi.py | 207 +++++++++++++++ cyborg/common/constants.py | 10 +- cyborg/common/exception.py | 9 + cyborg/common/policy.py | 227 +--------------- cyborg/conductor/manager.py | 32 +++ cyborg/conductor/rpcapi.py | 33 +++ cyborg/conf/agent.py | 3 +- cyborg/conf/glance.py | 23 -- .../alembic/versions/7e6f1f107f2b_add_qat_type.py | 22 ++ cyborg/db/sqlalchemy/models.py | 2 +- cyborg/hacking/checks.py | 28 ++ cyborg/image/api.py | 94 +------ cyborg/image/download/__init__.py | 54 ---- cyborg/image/glance.py | 68 +---- cyborg/objects/arq.py | 3 +- cyborg/objects/driver_objects/driver_deployable.py | 2 +- cyborg/objects/ext_arq.py | 2 +- cyborg/objects/extarq/ext_arq_job.py | 9 +- cyborg/objects/extarq/fpga_ext_arq.py | 6 +- cyborg/policies/__init__.py | 34 +++ cyborg/policies/base.py | 167 ++++++++++++ cyborg/policies/device_profiles.py | 111 ++++++++ .../drivers/aichip/huawei/test_ascend.py | 2 +- .../accelerator/drivers/fpga/inspur/__init__.py | 0 .../accelerator/drivers/fpga/inspur/test_driver.py | 111 ++++++++ .../drivers/fpga/intel/prepare_test_data.py | 2 +- .../accelerator/drivers/fpga/intel/test_driver.py | 8 +- .../unit/accelerator/drivers/fpga/test_base.py | 2 + .../unit/accelerator/drivers/gpu/test_utils.py | 8 +- .../unit/accelerator/drivers/qat/intel/__init__.py | 0 .../drivers/qat/intel/prepare_test_data.py | 265 +++++++++++++++++++ .../accelerator/drivers/qat/intel/test_driver.py | 109 ++++++++ .../unit/accelerator/drivers/qat/test_base.py | 29 +++ .../accelerator/drivers/spdk/nvmf/test_nvmf.py | 2 +- .../accelerator/drivers/spdk/vhost/test_vhost.py | 2 +- .../unit/api/controllers/v2/test_deployables.py | 2 +- .../api/controllers/v2/test_device_profiles.py | 32 +-- .../unit/api/controllers/v2/test_fpga_program.py | 143 ++++++++++ devstack/lib/cyborg | 23 +- devstack/lib/opae | 57 ---- devstack/plugin.sh | 3 - devstack/settings | 13 +- .../{sample_config.rst => sample-config.rst} | 0 .../{sample_policy.rst => sample-policy.rst} | 9 + ...ADME.policy.json.txt => README.policy.yaml.txt} | 2 +- etc/cyborg/{policy.json => policy.yaml} | 0 .../add-programming-method-051b9c4244c35c71.yaml | 6 + ...oject_id-to-arq-patch-api-8270cbb26c68af46.yaml | 8 + .../notes/fpga-driver-8b1635e92b1297c1.yaml | 2 +- .../notes/inspur-fpga-driver-0257d2aeda9537fb.yaml | 6 + .../notes/intel-qat-driver-a7b4c8f110d41e0b.yaml | 8 + ...file-default-value-change-de14a3688357b081.yaml | 14 + ...h_base_and_device_profile-cef00fca580d2323.yaml | 87 +++++++ ...-dependency-from-devstack-f6db83bb37761340.yaml | 13 + .../removed-download-modules-540fa0607d7df967.yaml | 7 + releasenotes/source/conf.py | 7 +- releasenotes/source/index.rst | 1 + releasenotes/source/ussuri.rst | 6 + requirements.txt | 9 +- setup.cfg | 21 +- test-requirements.txt | 10 +- tools/config/cyborg-policy-generator.conf | 2 +- tox.ini | 3 +- 132 files changed, 3417 insertions(+), 871 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 80242f5..d9d60cc 100644 --- a/requirements.txt +++ b/requirements.txt @@ -9 +9 @@ six>=1.8.0 # MIT -eventlet>=0.12.0,!=0.18.3,!=0.20.1,!=0.21.0,!=0.23.0,!=0.25.0 # MIT +eventlet>=0.26.0 # MIT @@ -17 +17 @@ oslo.service>=1.0.0,!=1.28.1 # Apache-2.0 -oslo.db>=4.1.0 # Apache-2.0 +oslo.db>=4.44.0 # Apache-2.0 @@ -20 +20 @@ oslo.upgradecheck>=0.1.0 # Apache-2.0 -oslo.utils>=3.33.0 # Apache-2.0 +oslo.utils>=4.5.0 # Apache-2.0 @@ -22 +22 @@ oslo.versionedobjects>=1.31.2 # Apache-2.0 -oslo.policy>=0.5.0 # Apache-2.0 +oslo.policy>=3.4.0 # Apache-2.0 @@ -29 +28,0 @@ psutil>=3.2.2 # BSD -mock>=2.0.0 # BSD diff --git a/test-requirements.txt b/test-requirements.txt index bcf6fa9..9f8c255 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -5 +5 @@ -hacking>=3.0,<3.1.0 # Apache-2.0 +hacking>=3.0.1,<3.1.0 # Apache-2.0 @@ -10,2 +10 @@ fixtures>=3.0.0 # Apache-2.0/BSD -mock>=2.0.0 # BSD -ddt>=1.0.1 # MIT +ddt>=1.2.1 # MIT @@ -13 +12 @@ oslotest>=3.2.0 # Apache-2.0 -stestr>=2.0.0 # Apache-2.0/BSD +stestr>=2.2.0 # Apache-2.0 @@ -16 +15 @@ testscenarios>=0.4 # Apache-2.0/BSD -testtools>=0.9.32 # MIT +testtools>=2.4.0 # MIT @@ -19 +17,0 @@ doc8>=0.6.0 # Apache-2.0 -pycodestyle>=2.0.0 # MIT License