We exuberantly announce the release of: openstack-cyborg 4.0.0: Distributed Acceleration Management as a Service This release is part of the ussuri release series. The source is available from: https://opendev.org/openstack/openstack-cyborg Download the package from: https://tarballs.openstack.org/cyborg/ For more details, please see below. Changes in openstack-cyborg 3.0.0..4.0.0 ---------------------------------------- a553468 Fix bandit error: Ascend driver:[B602:subprocess_popen_with_shell_equals_true] dd443b5 Fix bandit error: [B104:hardcoded_bind_all_interfaces] d8445ac Fix bandit error: [B108:hardcoded_tmp_directory] 1365c69 revert device and deployable when resource provider create fail 561cba9 Change bandit job from non-voting to voting 3af3a3e Fix bandit error: SPDK driver:[B602:subprocess_popen_with_shell_equals_true] ef0b219 Refactor v2 arq api ca283d0 Bump openstacksdk>=0.46.0 07a8e30 Introduce microversion 3431a15 Finish v2 API doc 782cf4c Cleanup py27 support 353d932 Optimize device profiles APIs d4441da Upgrade minversion to match python3 2b52672 Update contributing.rst per Ussuri goal and guidelines. f01c80b Update Nova API microversion in nova_client.py. 1f9a808 Add comments for deletion of ARQs by instance or ARQ UUIDs. a6f8433 Update hacking for Python3 afc1245 Check during ARQ bind that the target instance has no other ARQs. ec0eab0 default host to socket.gethostname() 31d23b7 Generate PDF documentation 31660c3 Add Cyborg API Reference to Cyborg index page 002e31a Add 'description' field to the device profile object fc7a85d Add description column to device_profiles db a9ca2c5 Deployable V2 API implementation 96aa80a Always stop cyborg-api process uwsgi container f5a47ab Fix setting cyborg agent hostname 76bdd3f add support for installing cyborg client dd2024e simplify devstack plugin b3aa875 move functions from devstack/plugin.sh to devstack/lib/cyborg 6709982 support standard devstack plugin cloning 0303fde Fix GPU's improper cpid_type allowed value and attach_handle_type value b0492aa Bugfix: Set the appropriate status code for ServiceUnavailable 0bf59df Improve the accuracy of device profile API responses in docs 6633fa3 Bugfix: expose placement service unavailable before diff cyborg db 6aff4de Bump oslo.messaging version >= 10.3.0 c7d7f31 Remove V1 API and related codes 4c4b732 Fix minor error in extarq job e9b16df Optimize deployble object module c616da8 Remove useless interfaces in agent 985b152 move setting to devstack/settings 8393680 Remove useless interfaces in cond 9831730 Add obj_make_compatible() 3c7e086 Delete sandbox directory eb12f68 add support for multi node deployments to fake driver d79ef4d Revert "Solve py37 timeout" 1345beb add testcases for async job bind 1e1b269 Fix exceptions defined with improper _msg_fmt d279c22 Avoid creating a root provider when parent is not found 16ac719 change default SimpleQueue to _PySimpleQueue for queue b123b10 add testcase for check_bindings_result failed 39b33b4 UT for job manager 08af601 Solve py37 timeout 12c448c Use ResourceNotFound replace ControlpathIDNotFound cb1b3ee Improve UT for cyborg/db ExtArq e014259 Remove useless attributes list in Deployable e7c6783 Improve UT for cyborg/db deployable 078014c Improve UT for cyborg/db device e2e1e3f Improve UT for cyborg/db attach handle aa2aa69 Remove the invalid specs from doc/source c87c232 Fix warning in logs that '' is not a valid UUID. d3648dc Send a separate bind event to Nova for each ARQ in an instance. 107cc7e Some bug fixes in async bind path. e4dfc6f bugs fix for compatibility issues between Py2 and Py3 5b6f26a Guess for the root cause of timeout acbc64f Enhance the db layer to verify filters a83f431 Document the alembic CLI better in README d443b27 Update gpu driver 6b28ca2 Improve UT for cyborg/db device profile d8cbe09 Define fake db objects for UT 33e74c0 testcase for FPGAExtARQ 4bf582a Use ResourceNotFound replace RP and Image NotFound 298ab6c Remove useless get_test_accelerator method and fix uuid error cc6ae38 Use ResourceNotFound replace DeployableNotFound 801df74 Set default value in get fpga trait ba05f63 Use ResourceNotFound replace DeviceNotFound d3b0f22 Use ResourceNotFound replace AttachHandleNotFound 2b8ff38 Implementation of Device v2 API 1bbf188 Use ResourceNotFound replace ExtArq and Attribute NotFound 23e32d1 Use ResourceNotFound replace DeviceProfileNotFound d765a34 Introduce bandit security linter 6db04c0 Add created_time and update_time for device profile API 7d98ff2 Improve objects/device.py UT coverage from 82% to 100% b35f6ea BugFix: Modify objects/device.py get_by_device_id method afa4a2a Modify api-paste.ini v1 to v2 59b8047 Refactor _gen_resource_inventory 115b7cc Enable UT for objects/deployable.py and fix import error b77df15 BugFix: modify the max allocation limit of acceleration resources 6363cb9 Do not fail ARQ binding for specific Nova notification errors. 914a944 Replace pycodestyle with pep8 for python3 compatible b7ff61e Remove 'base object' that no longer exist 7fceee8 Add 5 placeholder db migration scripts for Train ad4ec82 Update api-ref: Add accelerator requests(ARQs) section 408521a Update api-ref: deprecate v1 add v2 device_profile section d470442 Set ARQ get_all return 423 when any ARQ for vm is not resolved 4837062 Add procname for uwsgi based service cyborg-api d496229 Set ignore_basepython_conflict (fixes confusing pep8 message) 7c7dba5 Improve objects/attach_handle.py UT coverage from 68% to 100% cb38eb9 Remove api in doc f13a0a6 BugFix: FakeDirver put RC FPGA error 6b9ab02 Stop testing python2.7 f0d3326 Set compute API microversion to 'latest' when calling Nova. 92ff3f7 Simplify devstack script for generating cyborg.conf c867db9 Increase #accelerators in fake driver. a09a847 Fix method names and coding conventions in Placement client. 5751ceb Exclude cyborg tests dir in UT cover report f48c422 Bugfix: modify the log location for successful async bind ARQs 7074c4b Bugfix: _needs_programming lack one situation process 9dcbbb3 Enable openstack-cover-jobs in zuul check for cyborg 3b5201d Update docstring for apply_patch 9fb411e Remove useless glance api_servers define 7d78a3e Switch to Ussuri jobs a835765 Fix attach_handle status error when creating vm successfully 82985ed bug fix: ensure resource class instead of getting it. 123f9c9 Add GPU to supportted device. e8b3051 Bump the openstackdocstheme extension to 1.20 52494d8 API documentation update. 95e01e6 Update driver section. 86f10e2 Cleanup index / add in driver support matrix d0578cf async job for bind eea6751 Update master for stable/train 71df0a2 Fix the hardcoding of user role using sdk_adapter approach Diffstat (except docs and test files) ------------------------------------- .coveragerc | 12 + .gitignore | 3 +- .stestr.conf | 2 +- .zuul.yaml | 27 +- CONTRIBUTING.rst | 24 +- api-ref/source/conf.py | 17 - api-ref/source/index.rst | 2 +- api-ref/source/v1/deployables.inc | 119 ----- api-ref/source/v1/index.rst | 22 - api-ref/source/v2/arqs.inc | 115 +++++ api-ref/source/v2/deployables.inc | 48 ++ api-ref/source/v2/device_profile.inc | 118 +++++ api-ref/source/v2/devices.inc | 47 ++ api-ref/source/v2/index.rst | 18 + api-ref/source/v2/parameters.yaml | 73 ++++ cyborg/accelerator/drivers/aichip/huawei/ascend.py | 43 +- cyborg/accelerator/drivers/fake.py | 49 ++- cyborg/accelerator/drivers/fpga/intel/sysinfo.py | 12 +- cyborg/accelerator/drivers/gpu/utils.py | 64 ++- cyborg/accelerator/drivers/modules/generic.py | 15 +- .../drivers/spdk/util/pyspdk/py_spdk.py | 19 +- cyborg/agent/manager.py | 43 +- cyborg/agent/resource_tracker.py | 5 +- cyborg/agent/rpcapi.py | 22 - cyborg/api/config.py | 2 +- cyborg/api/controllers/base.py | 81 +++- cyborg/api/controllers/root.py | 66 ++- cyborg/api/controllers/types.py | 8 +- cyborg/api/controllers/v1/__init__.py | 62 --- cyborg/api/controllers/v1/accelerators.py | 194 -------- cyborg/api/controllers/v1/deployables.py | 237 ---------- cyborg/api/controllers/v2/__init__.py | 74 +++- cyborg/api/controllers/v2/api_version_request.py | 181 -------- cyborg/api/controllers/v2/arqs.py | 78 +++- cyborg/api/controllers/v2/deployables.py | 136 ++++++ cyborg/api/controllers/v2/device_profiles.py | 24 +- cyborg/api/controllers/v2/devices.py | 130 ++++++ cyborg/api/controllers/v2/versions.py | 41 ++ cyborg/api/hooks.py | 43 +- cyborg/api/middleware/auth_token.py | 2 +- cyborg/api/rest_api_version_history.rst | 22 + cyborg/common/constants.py | 66 ++- cyborg/common/exception.py | 159 +++---- cyborg/common/nova_client.py | 92 ++-- cyborg/common/placement_client.py | 66 +-- cyborg/common/policy.py | 23 +- cyborg/common/utils.py | 284 +++++++++++- cyborg/conductor/manager.py | 281 +++++------- cyborg/conductor/rpcapi.py | 98 ----- cyborg/conf/__init__.py | 5 +- cyborg/conf/api.py | 2 +- cyborg/conf/default.py | 57 +-- cyborg/conf/glance.py | 17 - cyborg/conf/nova.py | 42 ++ cyborg/conf/placement.py | 42 ++ cyborg/context.py | 45 ++ cyborg/db/api.py | 11 +- cyborg/db/sqlalchemy/alembic/README | 4 +- .../alembic/versions/22fb1af2d51e_placeholder.py | 20 + .../alembic/versions/57539722e5cf_placeholder.py | 20 + .../60d8ac91fd20_add_description_field_to_dps.py | 19 + .../alembic/versions/62bcf2610c5d_placeholder.py | 20 + .../alembic/versions/7a4fd0fc3f8c_placeholder.py | 20 + .../alembic/versions/7b696fd94949_placeholder.py | 20 + cyborg/db/sqlalchemy/api.py | 229 +++++----- cyborg/db/sqlalchemy/models.py | 1 + cyborg/hacking/checks.py | 27 +- cyborg/image/glance.py | 40 +- cyborg/objects/__init__.py | 1 + cyborg/objects/base.py | 30 ++ cyborg/objects/control_path.py | 8 +- cyborg/objects/deployable.py | 87 +--- cyborg/objects/device.py | 18 +- cyborg/objects/device_profile.py | 19 +- cyborg/objects/ext_arq.py | 306 ++++--------- .../controllers/v1 => objects/extarq}/__init__.py | 0 cyborg/objects/extarq/ext_arq_job.py | 247 +++++++++++ cyborg/objects/extarq/fpga_ext_arq.py | 249 +++++++++++ cyborg/quota.py | 1 + .../drivers/aichip/huawei/test_ascend.py | 13 +- .../unit/accelerator/drivers/gpu/test_utils.py | 25 +- .../unit/accelerator/drivers/test_fake_driver.py | 142 ++++++ .../unit/api/controllers/v1/_test_fpga_program.py | 48 -- .../unit/api/controllers/v2/test_deployables.py | 97 ++++ .../api/controllers/v2/test_device_profiles.py | 2 + .../unit/api/controllers/v2/test_microversion.py | 96 ++++ .../v1/base.py => image/test_glance.py} | 15 +- .../{_test_deployable.py => test_deployable.py} | 115 +---- devstack/lib/cyborg | 180 +++----- devstack/lib/opae | 4 - devstack/plugin.sh | 111 ++--- devstack/settings | 45 ++ .../accelerator_requests-create-resp.json | 25 ++ .../accelerator_requests-getone-resp.json | 20 + .../accelerator_requests-list-resp.json | 49 +++ .../accelerator_requests-patch-curl.json | 12 + .../accelerator_requests-post-curl.json | 6 + .../accelerator_requests-update-resp.json | 51 +++ .../deployables/deployable-create-resp.json | 36 -- .../deployables/deployable-patch-curl.json | 10 - .../deployables/deployable-post-curl.json | 26 -- .../deployables/deployable-update-resp.json | 72 --- .../deployables/deployables-getone-resp.json | 54 +-- .../deployables/deployables-list-resp.json | 74 +--- .../device_profiles-create-resp.json | 20 + .../device_profiles-getone-resp.json | 22 + .../device_profiles/device_profiles-list-resp.json | 43 ++ .../device_profiles-post-curl-with-bitstream.json | 11 + .../device_profiles/device_profiles-post-curl.json | 10 + .../{installation.rst => install-from-pip.rst} | 8 +- .../specs/pike/approved/cyborg-api-proposal.rst | 414 ------------------ .../specs/pike/approved/cyborg-conductor.rst | 143 ------ .../specs/pike/approved/cyborg-driver-proposal.rst | 163 ------- .../approved/cyborg-fpga-driver-proposal.rst | 193 -------- .../queens/approved/cyborg-fpga-model-proposal.rst | 346 --------------- .../specs/queens/approved/cyborg-internal-api.rst | 265 ----------- .../queens/approved/cyborg-nova-interaction.rst | 187 -------- .../approved/cyborg-spdk-driver-proposal.rst | 221 ---------- .../rocky/approved/cyborg-agent-driver-api.rst | 222 ---------- .../rocky/approved/cyborg-fpga-bitstream-spec.rst | 253 ----------- .../approved/cyborg-fpga-programming-proposal.rst | 200 --------- .../specs/rocky/approved/cyborg-nova-sched.rst | 486 --------------------- .../specs/rocky/approved/resource-quotas.rst | 204 --------- etc/cyborg/api-paste.ini | 6 +- ...ription_to_device_profile-3c2efcbd54dac7b0.yaml | 6 + ...python2-support-in-ussuri-e64f79db4e88ca19.yaml | 5 + ...ce-bandit-security-linter-339d3f12b6200d64.yaml | 7 + .../introduce-microversion-39c7f5cc6af4a139.yaml | 8 + releasenotes/source/conf.py | 23 - releasenotes/source/index.rst | 1 + releasenotes/source/train.rst | 6 + requirements.txt | 5 +- sandbox/rock.tar.gz | Bin 199857 -> 0 bytes setup.cfg | 7 +- setup.py | 9 - test-requirements.txt | 5 +- tox.ini | 55 ++- 186 files changed, 7129 insertions(+), 7371 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 0175791..80242f5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14 +14 @@ oslo.context>=2.9.0 # Apache-2.0 -oslo.messaging>=5.29.0 # Apache-2.0 +oslo.messaging>=10.3.0 # Apache-2.0 @@ -31,0 +32,3 @@ oslo.privsep>=1.32.0 # Apache-2.0 +cursive>=0.2.1 # Apache-2.0 +microversion_parse>=0.2.1 # Apache-2.0 +openstacksdk>=0.46.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index e200e58..bcf6fa9 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -5 +5 @@ -hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 +hacking>=3.0,<3.1.0 # Apache-2.0 @@ -6,0 +7 @@ hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0 +bandit>=1.6.0 # Apache-2.0 @@ -17,0 +19 @@ doc8>=0.6.0 # Apache-2.0 +pycodestyle>=2.0.0 # MIT License @@ -19,0 +22 @@ os-resource-classes>=0.5.0 # Apache-2.0 +cursive>=0.2.1 # Apache-2.0