We jubilantly announce the release of: tempest 21.0.0: OpenStack Integration Testing This release is part of the train release series. The source is available from: https://opendev.org/openstack/tempest Download the package from: https://pypi.org/project/tempest Please report issues through: https://bugs.launchpad.net/tempest/+bugs For more details, please see below. 21.0.0 ^^^^^^ Prelude ******* This is an intermediate release during the Train development cycle to mark the end of support for Pike in Tempest. After this release, Tempest will support below OpenStack Releases: * Stein * Rocky * Queens Current development of Tempest is for OpenStack Train development cycle. New Features ************ * Add "qos-policies" and "qos-minimum-bandwidth-rule" clients to Tempest to make possible the testing of the placement based bandwidth allocation feature. The following API calls are available for tempest from now: "QoS policies" client: * GET /qos/policies * POST /qos/policies * GET /qos/policies/{policy_id} * PUT /qos/policies/{policy_id} * DELETE /qos/policies/{policy_id} "QoS minimum bandwidth rules" client: * GET qos/policies/{policy_id}/minimum_bandwidth_rules * POST /qos/policies/{policy_id}/minimum_bandwidth_rules * GET qos/policies/{policy_id}/minimum_bandwidth_rules/{rule_id} * PUT qos/policies/{policy_id}/minimum_bandwidth_rules/{rule_id} * DELETE /qos/policies/{policy_id}/minimum_bandwidth_rules/{rule_id} * Add list host API support to the volume v3 client library. This feature enables callers to list all hosts for a given project. * Add migrate volume API support to the volume v3 client library. This features allows callers to migrate volumes between backends. * Add show default group type details API to v3 group_types_client library. * show_default_group_type * New decorator "unstable_test" is added to "tempest.lib.decorators". It can be used to mark some test as unstable thus it will be still run by tempest but job will not fail if this test will fail. Such test will be skipped in case of failure. It can be used for example when there is known bug related which cause irregular tests failures. Marking such test as unstable will help other developers to get their job done and still run this test to get additional debug data or to confirm if some potential fix really solved the issue. * A new parameter, compute/compute_volume_common_az is introduced to specify availability zone where tempest creates instances and volumes for scenario tests, to allow us to run scenario tests in the deployment which has multiple availability zones and cinder/cross_az_attach in nova.conf is set to False. * New library interface to set the API microversion on Service Clients. "APIMicroversionFixture," can be used to set the API microversion on multiple services. This Fixture will take care of reseting the service microversion to None once test is finished. * Add microversion support for scenario tests. Scenario test calls multiple service API within same test and many services like compute, volume and placement etc support API microversion. With microversion support in scenario test, we can call different service API with different microvesion. Which means we can implement scenario tests for microversion also. Currently Scenario manager support below services microversion: * Compute * Volume * Placement Upgrade Notes ************* * Remove deprecated config option "endpoint_type" from "identity" group. Use "v2_public_endpoint_type" from "identity" group instead. Remove deprecated config option "tenant_isolation_domain_name" from "auth" group. Use "default_credentials_domain_name" from "auth" group instead. * Remove deprecated config option "admin_username" from "identity" groups. Use "admin_username" from "auth" instead. Remove deprecated config option "admin_tenant_name" from "auth" and "identity" groups. Use "admin_project_name" from "auth" instead. Remove deprecated config option "admin_password" from "identity" groups. Use "admin_password" from "auth" instead. Remove deprecated config option "admin_domain_name" from "identity" groups. Use "admin_domain_name" from "auth" instead. Deprecation Notes ***************** * The config option "CONF.network.dns_servers" is no longer used anywhere, so it is deprecated and will be removed in the future. Other Notes *********** * New configuration options "[compute]/certified_image_ref" and "[compute]/certified_image_trusted_certs" have been introduced. These are required in order to run the "ServerShowV263Test" test and allow a signed image with the required img_signature_* properties set along with a list of trusted certificates to be used during the test. Changes in tempest 20.0.0..21.0.0 --------------------------------- abd07b42c Add releasenote to tag the end of support for Pike 95358523e Define the Integrated-gate-object-storage gate template ee7fd1f7e Define the Integrated-gate-compute gate template 31609dc1d Define the Integrated-gate-placement gate template 03c9c94c8 add show default group type details and unit test a52572347 add unittest for volume encryption types a699445d2 Define the Integrated-gate-storage gate template 5072a2408 Define the Integrated-gate-networking gate template 9c25b1d0b Fix tempest run doc indentation b3daeb4e1 Filter out floating IPs from server IPs list 0187a6647 Remove nova-lxd plugin from blacklist 012dc98fb Remove networking-ansible from tempest plugins list 3f5748906 Remove cyborg-tempest-plugin from BLACKLIST e1afe4e75 Fix invalid assertIsNone state cff4ed761 compute: Correctly pass certified_image_ref as image_id 83f524af2 Use assert_flavor_equal to check flavor before and after microversion 2.46 aa91d42bb Revert "Skip test_server_connectivity_cold_migration_revert until fixed" 2957b4020 Raise non 404 Error in generate-tempest-plugin-list.py 6617b834f Add project prefix to blacklist b02555364 Stop checking the error message in test_boot_with_low_ram 7f7c232b2 compute: Make image certificate validation a configurable feature 00e6d6c2e Specify AZ for non-scenario tests 97857948d Fix cleanup NotImplemented error 1c976a93d Remove RetryFilter from defaults e453fc2b9 fup: fix docsting in manager.py 0ded98ff6 Pass more accurate error message to DeleteErrorException ac8ae64d5 Publish BLACKLIST on Tempest plugin registry page e36fe67e8 Publish non-openstack namespace tempest plugins 979e494da Specify availability zone to create instances and volumes c87a06b3c Minimum bandwidth allocation with placement f4d359245 Remove some deprecated auth and identity options c98a3254c Immutable user source: Non-admin V2 and V3 tests 80105b7c5 Move tox execution to current docs section e8a5e5c82 Add QoS policies and minimum bandwidth rule client 09c4eb99c Add microversion support in scenario tests 0ea2c01d7 Break wait_for_volume_resource_status when error_extending 8802c83be Fix invalid parameter passed to create_volume 1a457ac53 Fix region leaks after testing endpoints 5402cc3f8 Remove some deprecated identity options c6b0f140b Print passed and failed plugins at the end 32a4b11f1 Remove vitrage-tempest-plugin from blacklist 85dfbf6ef Remove nova-cells-v1 job from experimental queue 13731b020 Change IPv6 ICMP security group rule check 349ba7541 Replace git.openstack.org URLs with opendev.org URLs 0b14b1e0f Add compute API response schema validation for 2.71 fe0de68da Set default value for tox tasks 5511ea020 Fix mismatch error due to map in python3 7f5dd855c Add cyborg-tempest-plugin to required-projects and remove octavia 8460cb184 Remove airship-tempest-plugin from blacklist e027ea931 Make dns_servers option deprecated 39fe69517 Remove legacy-periodic-tempest-dsvm-neutron-full-ocata 925188d2d Make test_hotplug_nic test first check the IP 514495b1a Revert "Refactor random naming" 65c891107 Improve ipv6 configuration 23325c0ee Fix resource leaks in EndPointGroupsTest f5cff8b44 [Trivial]Remove unused variable assignments c248f5962 Fix the bug of deleting security group after testing changing password. 748dd8df7 Handle properly 404 responses from Neutron b9b6e37a7 Add support for compute API v2.70 - os-volume_attachments 27ba933ab Handle 'path' query parameter for test_novnc e5597401f Introduce an attached volume migration test 2b1751e46 Use show_quota_class_set after update_quota_class_set 8df5fdcbe Add assertions to attached volume retype test 682965e39 OpenDev Migration Patch 4beff29c3 Add the api type check when check the param of api_microversion 9a740dd26 Fix invalid filter passed to list_volumes ad8737c7f Add APIMicroversionFixture to library interface 21f53012f Add unstable_test decorator 762e98c23 Skip test_volume_backup_create_get_detailed_list_restore_delete 7ea7e0a14 update identity role tests to work w/ pre-prov be64e1ae7 Ensure all image cache volumes are removed before removing the volume type 77a1933bd Run integration jobs for requirements.txt changes a7f16b291 Uncap jsonschema 1bbfa121b Fix test_volume_swap_with_multiattach 598b1ae85 Improve router deletion logging in tempest cleanup fd01fe944 Improve logging in tempest cleanup a8cb701cc Init placement client in tempest Manager object c07edb5f8 Mention autopep8 in Hacking.rst 5b98592b9 Add Tempest gate job for stable/stein 154621187 Add release notes page for version 20.0.0 5d06623da Add missing validation for the compute/positive ff3851b59 Replace openstack.org git:// URLs with https:// 787e69426 Remove the Ocata stable branch jobs from master gate c68546f06 Fix potential issue with adding a fixed ip ac6224ee6 Use nova zuulv3 job for nova v2 API 196ab8a6c Use base.delete_server in test_servers.py 33c3e60a3 Adds protocol options for test_cross_tenant_traffic 897fc7c8a Remove sys unused Diffstat (except docs and test files) ------------------------------------- .gitreview | 2 +- .zuul.yaml | 409 +++++++++++++++------ HACKING.rst | 4 +- README.rst | 24 +- REVIEWING.rst | 2 +- .../tempest-blacklisted-plugins-registry.header | 7 + ...sed-minimum-bw-allocation-8e5854d5754cec68.yaml | 25 ++ ...-v3-volume-client-library-ad3529260db58f00.yaml | 8 + ...-to-v3-group-types-client-65f717878cf52da0.yaml | 6 + ...d-unstable_test-decorator-a73cf97d4ffcc796.yaml | 11 + .../notes/bug-1647999-7aeda50a8d082d4c.yaml | 8 + ...rtificate_compute_feature-c56efb520d54aff5.yaml | 8 + ...ort-profile-config-option-d67f5cb31f1bc34c.yaml | 2 +- ...recate-dns_servers-option-0xf2f297ee47a5ff.yaml | 6 + ...ntermediate-train-release-0146c624cff9a57f.yaml | 12 + ..._api_microversion_fixture-f52308fc6b6b89f2.yaml | 7 + ...auth-and-identity-options-xa1xd9b8fb948g4f.yaml | 8 + ...precated-identity-options-0ffxd1b8db928e43.yaml | 11 + ...oversion-in-scenario-test-b4fbfdd3a977fc58.yaml | 14 + releasenotes/source/index.rst | 1 + releasenotes/source/v20.0.0.rst | 6 + requirements.txt | 2 +- tempest/api/compute/admin/test_quotas.py | 11 +- tempest/api/compute/admin/test_volume_swap.py | 13 + tempest/api/compute/base.py | 7 +- .../api/compute/flavors/test_flavors_negative.py | 10 +- .../api/compute/servers/test_attach_interfaces.py | 53 ++- tempest/api/compute/servers/test_novnc.py | 19 +- tempest/api/compute/servers/test_server_actions.py | 17 +- tempest/api/compute/servers/test_servers.py | 60 +-- tempest/api/compute/test_extensions.py | 2 +- tempest/api/compute/volumes/test_attach_volume.py | 12 +- .../api/identity/admin/v3/test_endpoint_groups.py | 2 + tempest/api/identity/admin/v3/test_endpoints.py | 34 +- .../identity/admin/v3/test_endpoints_negative.py | 6 +- tempest/api/identity/admin/v3/test_roles.py | 4 + tempest/api/identity/v2/test_users.py | 6 + tempest/api/identity/v3/test_catalog.py | 4 +- tempest/api/identity/v3/test_users.py | 8 + .../admin/test_external_network_extension.py | 1 + .../admin/test_floating_ips_admin_actions.py | 19 +- tempest/api/network/admin/test_negative_quotas.py | 10 +- tempest/api/network/admin/test_ports.py | 16 +- tempest/api/network/admin/test_routers.py | 10 +- tempest/api/network/admin/test_routers_dvr.py | 11 +- tempest/api/network/admin/test_routers_negative.py | 4 +- tempest/api/network/base_security_groups.py | 4 +- tempest/api/network/test_allowed_address_pair.py | 10 +- tempest/api/network/test_extra_dhcp_options.py | 4 +- tempest/api/network/test_floating_ips.py | 60 ++- tempest/api/network/test_floating_ips_negative.py | 2 + tempest/api/network/test_networks.py | 8 +- tempest/api/network/test_ports.py | 47 ++- tempest/api/network/test_routers.py | 47 ++- tempest/api/network/test_security_groups.py | 18 +- tempest/api/volume/admin/test_volume_retype.py | 2 +- tempest/api/volume/base.py | 4 + tempest/api/volume/test_volumes_backup.py | 1 + tempest/api/volume/test_volumes_extend.py | 2 +- tempest/api/volume/test_volumes_snapshots.py | 2 +- tempest/clients.py | 3 + tempest/cmd/cleanup.py | 13 +- tempest/cmd/cleanup_service.py | 92 +++-- tempest/cmd/run.py | 10 +- tempest/common/compute.py | 15 +- tempest/common/utils/linux/remote_client.py | 50 +++ tempest/common/waiters.py | 35 +- tempest/config.py | 73 ++-- tempest/exceptions.py | 5 + .../api_schema/response/compute/v2_1/volumes.py | 3 +- .../api_schema/response/compute/v2_16/servers.py | 3 + .../api_schema/response/compute/v2_19/servers.py | 3 + .../api_schema/response/compute/v2_26/servers.py | 3 + .../api_schema/response/compute/v2_3/servers.py | 3 + .../api_schema/response/compute/v2_47/servers.py | 3 + .../api_schema/response/compute/v2_48/servers.py | 3 + .../api_schema/response/compute/v2_54/servers.py | 3 + .../api_schema/response/compute/v2_57/servers.py | 3 + .../api_schema/response/compute/v2_6/servers.py | 3 + .../api_schema/response/compute/v2_63/servers.py | 3 + .../api_schema/response/compute/v2_70/__init__.py | 0 .../api_schema/response/compute/v2_70/servers.py | 80 ++++ .../api_schema/response/compute/v2_71/__init__.py | 0 .../api_schema/response/compute/v2_71/servers.py | 81 ++++ .../api_schema/response/compute/v2_8/servers.py | 3 + .../api_schema/response/compute/v2_9/servers.py | 3 + tempest/lib/common/api_microversion_fixture.py | 82 +++++ tempest/lib/common/api_version_utils.py | 5 +- tempest/lib/decorators.py | 42 +++ tempest/lib/services/compute/servers_client.py | 9 +- tempest/lib/services/network/__init__.py | 10 +- tempest/lib/services/network/qos_client.py | 70 ++++ .../network/qos_minimum_bandwidth_rules_client.py | 73 ++++ .../lib/services/volume/v3/group_types_client.py | 12 + tempest/lib/services/volume/v3/volumes_client.py | 27 +- tempest/scenario/manager.py | 130 ++++++- .../scenario/test_minbw_allocation_placement.py | 195 ++++++++++ .../scenario/test_network_advanced_server_ops.py | 1 - tempest/scenario/test_network_basic_ops.py | 13 +- tempest/scenario/test_network_v6.py | 5 +- tempest/scenario/test_security_groups_basic_ops.py | 44 ++- tempest/scenario/test_volume_migrate_attached.py | 103 +++++- .../test_qos_minimum_bandwidth_rules_client.py | 137 +++++++ .../volume/v3/test_encryption_types_client.py | 28 +- .../services/volume/v3/test_group_types_client.py | 23 ++ tools/format.sh | 26 +- tools/generate-tempest-plugins-list.py | 71 +++- tools/generate-tempest-plugins-list.sh | 47 ++- .../tempest-integrated-gate-compute-blacklist.txt | 13 + ...empest-integrated-gate-networking-blacklist.txt | 17 + ...st-integrated-gate-object-storage-blacklist.txt | 17 + ...tempest-integrated-gate-placement-blacklist.txt | 19 + .../tempest-integrated-gate-storage-blacklist.txt | 13 + tools/tempest-plugin-sanity.sh | 57 +-- tox.ini | 83 ++++- 125 files changed, 2917 insertions(+), 534 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 7520d424a..bf38faed2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6 +6 @@ cliff!=2.9.0,>=2.8.0 # Apache-2.0 -jsonschema<3.0.0,>=2.6.0 # MIT +jsonschema>=2.6.0 # MIT