We are excited to announce the release of: openstack-placement 2.0.0: Resource provider inventory usage and allocation service This release is part of the train release series. The source is available from: https://opendev.org/openstack/openstack-placement Download the package from: https://pypi.org/project/openstack-placement For more details, please see below. Changes in openstack-placement 1.0.0.0rc1..2.0.0 ------------------------------------------------ ba1e9495 Add --skip-locks flag to mysql-migrate-db.sh 4dde9535 Fix domain name in install doc (2) 3f1a89c7 Fix domain name in install doc 6399f01c Update TOX/UPPER_CONSTRAINTS_FILE for stable/train 3d64e1df Update .gitreview for stable/train 7a6403c6 Add Train upgrade notes abb14fe2 Add train-prelude release note 04b6c2fb Clean up contributor document ac1e0f8a Clean up document's index 10fc2d35 Update the constraints url 13da0bff Fix section structure for pdf docs b93200d7 Build pdf docs 804a1158 Un-cap jsonschema for python3.6/3.7 support 9ece3d35 Deprecate [placement]/policy_file config option 76a5c83c Update nested-magic spec for root_member_of 15ab8280 Move nested magic spec to implemented 3eb86273 Update setup.cfg to include project_urls 10146978 Fix misspell word 156b0833 Fix typo in microversion sequence test 8272dfee Fix links to migration scripts 75fb2fe8 Clarify the NOTE associated with ordering of middleware 7327f2ee Merge request log and request id middlewares 3a732f87 Refactor exclude_nested_providers() 25732260 Get usages in _build_provider_summaries() 768cc889 Add place-held _static dir for Sphinx 2.2.0 31794c85 api-ref: fix typo in aggregates note aecd0f34 Avoid duplicate ProviderSummary in _merge_candidates 5dda479f Add a rw_ctx.psum_res_by_rp_rc, for clarity 803d673c Use rp.id instead of uuid in _rp_rc_key 3f82e3a9 Add rw_ctx.parent_uuid_by_rp_uuid, for clarity 978408b9 Add gabbits using a DeepNUMANetworkFixture 9d256fa6 gabbi test for same_subtree with an ancestry hole a6e1a19a Add tests demonstrating overlapping same_subtreeZ 0e199901 Fix allocation bug in NUMANetworkFixture 3c102181 Use expanding bindparam in get_traits_by_provider_tree e9623705 Copy AllocationRequestResource only when necessary 1a7d39be Add __copy__ method to AllocationRequest{,Resource} b16b48ad Correct SQL docstring on _get_usages_by_provider_trees f69f5845 Use another expanding bindparam in _get_usages_by_provider_trees 6de94f98 Move provider_ids_from_rp_ids to allocation_candidate and fix 5fd2d18c Optimize trait creation to check existence first d65fb810 Improve docs and comments for provider_ids_from_rp_ids 8da825ef Remove double join in provider_ids_from_rp_ids 9820b1bd Clean up the extend_usages_by_provider_tree method d6c151f2 Trivial: Remove duplicate usage of db context 5883c794 Use expanding bindparam in extend_usages_by_provider_tree 51e4f419 Make _get_trees_with_traits return a set 5dd9a290 Track usage info on RequestWideSearchContext 0386f534 Further optimize _build_provider_summaries 6ef820c9 Add RequestWideSearchContext.summaries_by_id ed030851 Add apache benchmark (ab) to end of perfload jobs 07d7749c Implement a more complex nested-perfload topology 7464ff6e Run nested-perfload parallel correctly 5466ec18 Make placement base API return version without auth d4d9457a Use expanding bindparam in provider_ids_from_rp_ids in_ 5d3243f4 Use __slots__ in commonly used objects 8626b651 Remove ProviderIds namedtuple 069c531b _get_all_by_filters_from_db do not cast to list of dict d0416808 Bump os-traits minimum to 0.16.0 835081ac Blacklist sphinx 2.1.0 (autodoc bug) a0e2c027 Use TraitCache for Trait.get_by_name 3b484c80 Extra gabbi tests for same_subtree b1a7338d Use integrated-gate-placement zuul template b09f2f91 Make a TraitCache similar to ResourceClassCache 7efcb2e7 Further simplify microversion utils 99b107fb Update api-ref to point to API-SIG microversion spec 7b3c179f Update api-ref location 8eedcfce Run 'tempest-ipv6-only' job in gate a24868fd Bump os-resource-classes requirements d76d6ab7 Extract a _get_schema from list_allocation_candidates 2fc321ae Move rc_cache onto RequestContext baaaff17 Make placement testing easier on osx d2b452d7 Simplify placement.microversion:_fully_qualified_name a09defd6 api-ref: Document generations 4ae8ffea Add placement.query.missing_value in api-ref 78583754 Add Python 3 Train unit tests 3f786431 Doc `same_subtree` queryparam 328f7a3a Add query.duplicate_key and .bad_value in api-ref efaa3644 Follow up fix for same_subtree documentation 23300f28 Trivial: Fix docs for functions 8395e3f0 Support `same_subtree` queryparam 8c4ad849 tox: Stop building api-ref docs with the main docs 4b4dd5ec Add whereto for testing redirect rules 3659df6e Update implemented spec and spec document handling d1005782 Correct variable use and naming in mappings tests 06256d86 Manage mappings on AllocationRequest object 7cadf3e8 Recreate: incorrect mappings with group_policy=none ed0af2e4 Fix up some inaccuracies in perfload comments and logs ed4a9a1d Spec: Support Consumer Types 2772dafc Bump os-traits minimum to 0.15.0 41287a74 Remove gate/post_test_hook.sh 180c83b5 Centralize and clarify pip in the docs 243757dd Mention OsProfiler in the testing doc 66378006 Add OsProfiler config options to generated reference 0d9c510d Trivial: Update document for Request IDs 38d3d5d1 Add a test for granular member_of not flowing down 698645ca Miscellaneous doc/comment/log cleanups b733786a Microversion 1.35: root_required 5a6884ae RequestWideParams and RequestWideSearchContext c774416d Refactor anchors_for_sharing_providers 85fcc8e7 research_context._get_roots_with_traits() c00d0433 Spec for nested magic 1 40c97d73 Add support for osprofiler in wsgi 3b040f58 Move non-nested perfload shell commands to script 8723bd77 Nested provider performance testing 99242b83 Update SUSE install documentation e1aec515 Remove overly-verbose allocation request log 9de03e1b Uniquify allocation mappings 6483cd26 Remove a redundant test 71114d9e Add missing suffix-related docstrings d38844e3 Implement allocation candidate mappings eb079134 Prepare objects for allocation request mappings 01e69915 Remove incomplete consumer inline migrations 221c65a7 Add a blocker migration for missing consumer records e1783b00 Correctly limit provider summaries when nested 88b6c816 Add NUMANetworkFixture for gabbits 81937773 Stabilize AllocationRequest hash 910b466c perfload with written allocations 4cca0ee1 Bump os-traits to latest release (0.14.0) 88c6ad9c Optionally run a wsgi profiler when asked 7db53444 Bump os-traits requirements 7049e407 Resource provider - request group mapping in allocation candidate 289130fc Bump openstackdocstheme to 1.30.0 7b8e2a8a Reuse cache result for sharing providers capacity f8bbda15 Move seek providers with resource to context d75bdbff Remove normalize trait map func 7f4b79b7 Cache provider ids in requested aggregates fb71a6ab Move search functions to the research context file daf7285a Add RequestGroupSearchContext class fc35e311 Modernize CORS config and setup 7db2e293 Add olso.middleware.cors to conf generator e0b35bc3 Don't run functional.db tests in nova functional run aeb65d4c Trivial: Fix comment for LEFT join 5f4da5e0 Use trait strings in ProviderSummary objects 346509f8 Avoid traversing summaries in _check_traits_for_alloc_request 7d39ee72 Canary test for os-traits version cb28c8dd Fix typo in usage.yaml and usage-policy.yaml 4bfffd7f Bump os-resource-classes requirements 4d05bb8f Fixups from removing null provider protections e0efa65e Remove null root provider protections 4606e55d Add blocker alembic migration for null root_provider_ids 4af1df94 Change "Missing Root Provider IDs" upgrade check to a failure fb0f6f26 Allow [a-zA-Z0-9_-]{1,64} for request group suffix e98b5df3 Add 'docs' worklist to worklist table 43c859b4 Cap sphinx for py2 to match global requirements 73b29cd6 Enhance debug logging in allocation candidate handling 727fb88d Skip _exclude_nested_providers() if not nested fea9bad7 Raise os-traits os-resource-classes constraints c62ed6cc Package db migration scripts in placement pypi dist df12d4d2 Skip notification sample tests when running nova functional e621d300 Run nova-tox-functional-py36 in the placement gate e9160abc Update worklist information for contributors ae9bdbaa Remind people to use postgresql-migrate-db.sh when migrating data a2513d8e Replace git.openstack.org URLs with opendev.org URLs da3e8795 Dropping the py35 testing 931a9e12 OpenDev Migration Patch 1b5710db Add links to storyboard worklists to contributing.rst d921ad4b api-ref: fix formatting in member_of param for 1.21 c3a1aefe Fix arg typos in contributing.rst 5d0daffa Remove dead code 3d29a4b7 Correct task status when a task is under review 852294bd Rename api-ref parameters e6f234fe FUP on negative-aggregate-membership series a4276417 Fix a broken link in a release note 4040f898 Refactor aggregate _get_trees_matching_all() 1060123a Refactor ResourceProviderListTestCase 0a3dcadb Negative member_of query with microversion 1.32 69a70004 Prepare for negative member queryparam 2 cc959d53 Fill in the Writing Code section of contributing.rst 73a585d7 Fill in the New Features section of contributing.rst aa1ca270 Fill in reviewing section of contributing.rst fcd99b33 Fill in the bugs section of contributing.rst bf36ed61 Fix debug log getting allocation_candidates 0cbf2635 Remove fake resource class from fake_ensure_cache 3752f045 s/rc_cache.ensure_rc_cache/rc_cache.ensure/ a83c5a4c Add initial framing for a contributing doc da1a588b Remove use of oslo.i18n and translation c99492cf Replace openstack.org git:// URLs with https:// 48948bce Initial structure for in-tree specs f7f52316 Add register_opts param to PlacementFixture 4b35ee2c Revert "Centralize registration of logging options" 6eed58d3 Update master for stable/stein bbc2af0b Spec: Support filtering by forbidden aggregates 2 4baa6cb6 Spec: Support filtering by forbidden aggregates 1 820bf629 Prepare for negative member_of queryparam f4f98293 Add link to case studies in alloc_cands api-ref 0045e59f Document alloc-candidates-in-tree Diffstat (except docs and test files) ------------------------------------- .gitignore | 1 - .gitreview | 3 +- .zuul.yaml | 47 +- README.rst | 6 +- api-ref/source/aggregates.inc | 2 +- api-ref/source/allocation_candidates.inc | 20 +- api-ref/source/allocations.inc | 22 +- api-ref/source/errors.inc | 9 + api-ref/source/generations.inc | 48 + api-ref/source/index.rst | 1 + api-ref/source/parameters.yaml | 227 +++- api-ref/source/request-ids.inc | 4 +- api-ref/source/reshaper.inc | 1 + api-ref/source/resource_providers.inc | 2 +- api-ref/source/root.inc | 4 +- .../get-allocation_candidates-1.34.json | 96 ++ .../to-stein.rst => admin/upgrade-to-stein.rst} | 23 +- .../approved/2005473-support-consumer-types.rst | 357 ++++++ .../2005297-negative-aggregate-membership.rst | 386 ++++++ .../train/implemented/2005575-nested-magic-1.rst | 636 ++++++++++ ...uest-group-mapping-in-allocation-candidates.rst | 577 +++++++++ etc/placement/config-generator.conf | 5 +- gate/README | 18 +- gate/gabbits/nested-perfload.yaml | 183 +++ gate/perfload-nested-loader.sh | 24 + gate/perfload-nested-runner.sh | 108 ++ gate/perfload-runner.sh | 123 ++ gate/perfload-server.sh | 30 + gate/post_test_hook.sh | 75 -- lower-constraints.txt | 6 +- placement/attribute_cache.py | 163 +++ placement/auth.py | 4 +- placement/cmd/manage.py | 33 +- placement/cmd/status.py | 34 +- placement/conf/__init__.py | 6 + placement/conf/placement.py | 11 + placement/context.py | 3 + .../611cd6dffd7b_block_on_null_root_provider_id.py | 42 + .../b5c396305c25_block_on_null_consumer.py | 50 + placement/deploy.py | 68 +- placement/errors.py | 5 + placement/exception.py | 103 +- placement/handler.py | 7 +- placement/handlers/aggregate.py | 9 +- placement/handlers/allocation.py | 46 +- placement/handlers/allocation_candidate.py | 117 +- placement/handlers/inventory.py | 75 +- placement/handlers/reshaper.py | 23 +- placement/handlers/resource_class.py | 17 +- placement/handlers/resource_provider.py | 26 +- placement/handlers/trait.py | 54 +- placement/handlers/usage.py | 3 +- placement/handlers/util.py | 9 +- placement/i18n.py | 36 - placement/lib.py | 233 +++- placement/microversion.py | 51 +- placement/objects/allocation.py | 95 +- placement/objects/allocation_candidate.py | 806 ++++++------ placement/objects/inventory.py | 3 +- placement/objects/research_context.py | 1297 ++++++++++++++++++++ placement/objects/resource_class.py | 14 +- placement/objects/resource_provider.py | 1065 +--------------- placement/objects/rp_candidates.py | 8 + placement/objects/trait.py | 71 +- placement/objects/usage.py | 12 +- placement/policy.py | 46 +- placement/requestlog.py | 55 +- placement/resource_class_cache.py | 142 --- placement/rest_api_version_history.rst | 103 ++ placement/schemas/allocation.py | 23 + placement/schemas/allocation_candidate.py | 23 +- placement/schemas/common.py | 5 + placement/schemas/reshaper.py | 5 + .../functional/db/test_allocation_candidates.py | 777 +++++++++--- ...urce_class_cache.py => test_attribute_cache.py} | 30 +- .../gabbits/allocation-candidates-bug-1792503.yaml | 86 +- .../allocation-candidates-mappings-numa.yaml | 134 ++ .../allocation-candidates-mappings-sharing.yaml | 77 ++ .../gabbits/allocation-candidates-member-of.yaml | 114 +- .../allocation-candidates-root-required.yaml | 291 +++++ .../functional/gabbits/allocation-candidates.yaml | 13 + .../functional/gabbits/allocations-mappings.yaml | 100 ++ .../functional/gabbits/granular-same-subtree.yaml | 538 ++++++++ .../gabbits/resource-classes-policy.yaml | 2 +- .../gabbits/resource-provider-aggregates.yaml | 118 ++ .../functional/gabbits/same-subtree-deep.yaml | 78 ++ .../unit/objects/test_allocation_candidate.py | 88 +- placement/util.py | 117 +- placement/wsgi.py | 25 +- placement_db_tools/__init__.py | 21 + placement_db_tools/mysql-migrate-db.sh | 307 +++++ placement_db_tools/postgresql-migrate-db.sh | 287 +++++ playbooks/nested-perfload.yaml | 20 + playbooks/perfload.yaml | 120 +- ...cation-candidate-mappings-e00cf6deadcee9ab.yaml | 14 + ...on-candidate-same_subtree-aeed7b2570293dfb.yaml | 11 + ...-candidates-root_required-bfe4f96f96a2a5db.yaml | 12 + ...ate-placement-policy-file-1777dc2e92d8363c.yaml | 11 + .../fix-osprofiler-support-78b34a92c32fd30f.yaml | 7 + .../granular-request-suffix-a7fd857eadc16b56.yaml | 12 + ...ted-allocation-candidates-0886e569d15ad951.yaml | 11 + ...tive-aggregate-membership-1dde3cbe27c69279.yaml | 34 + .../notes/stein-prelude-779b0dbfe65cf9ac.yaml | 2 +- .../notes/train-prelude-06739452ba2f66d9.yaml | 23 + ...require-root-provider-ids-60bc374ac354f41e.yaml | 9 + releasenotes/source/index.rst | 1 + releasenotes/source/stein.rst | 6 + requirements.txt | 10 +- setup.cfg | 26 +- test-requirements.txt | 7 +- tools/mysql-migrate-db.sh | 302 +---- tools/postgresql-migrate-db.sh | 288 +---- tox.ini | 34 +- 170 files changed, 11898 insertions(+), 4428 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index b109c7c9..e0563112 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6 +6 @@ pbr!=2.1.0,>=2.0.0 # Apache-2.0 -SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT +SQLAlchemy>=1.2.19 # MIT @@ -10 +10 @@ WebOb>=1.8.2 # MIT -jsonschema<3.0.0,>=2.6.0 # MIT +jsonschema>=2.6.0 # MIT @@ -22 +21,0 @@ oslo.policy>=1.35.0 # Apache-2.0 -oslo.i18n>=3.15.3 # Apache-2.0 @@ -25,2 +24,3 @@ oslo.upgradecheck>=0.2.0 # Apache-2.0 -os-resource-classes>=0.2.0 # Apache-2.0 -os-traits>=0.4.0 # Apache-2.0 +# NOTE(efried): Sync lower-constraints.txt for os-traits & os-resource-classes. +os-resource-classes>=0.5.0 # Apache-2.0 +os-traits>=0.16.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 1b4d1249..3f8a93fc 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -9,2 +9,2 @@ mock>=2.0.0 # BSD -psycopg2>=2.6.2 # LGPL/ZPL -PyMySQL>=0.7.6 # MIT License +psycopg2>=2.6.2; sys_platform!='darwin' # LGPL/ZPL +PyMySQL>=0.7.6; sys_platform!='darwin' # MIT License @@ -18,0 +19,3 @@ wsgi-intercept>=1.7.0 # MIT License + +# needed to generate osprofiler config options +osprofiler>=1.4.0 # Apache-2.0