We are jazzed to announce the release of: searchlight 2.0.0: OpenStack Search Service This release is part of the ocata release series. Download the package from: https://tarballs.openstack.org/searchlight/ For more details, please see below. 2.0.0 ^^^^^ Add support for the new Neutron tenant RBAC policy functionality. This allows networks to be shared across tenants. Introduce a pipeline architecture to configure publishers for resource plugins. Added 'size' and 'from' as synonyms of 'limit' and 'offset', to closer match the Elasticsearch API. When user requests a reindex by specifying --type parameter, searchlight may copy existing Elasticsearch data to reindex, if these old indices or type mappings are missing, newly created indices will have incomplete resource data. Some Neutron ports that do not send notificatons to Searchlight. Do not initially index them. In Elasticsearch 2.x field names cannot contain '.', so when indexing document with property contains '.' in field name, it will fail with error "cannot contain '.'". New Features ************ * Adds notification handler for nova server group. * Notifications for flavor are now supported. * Adds ironic plugin with nodes, ports and chassis support. * Add support for the new Neutron tenant RBAC policy functionality. This allows networks to be shared across tenants. * Publishers can be configured at plugin level or top level in configuration file. When Searchlight does incremental updates via notifications, configured publishers will receive information about modified resources and publish them to subscribers outside Searchlight. * Adds on option "--notification-less" to "searchlight-manage index sync" to re-sync only plugins that don't have notifications. * In order to more closely match the Elasticsearch API, 'size' and 'from' can be used in place of 'limit' and 'offset'. * Add "error" notification level processing to listener. Known Issues ************ * The projects list will not present in the flavor related versioned notifications if you update the flavor access only because of a lazy-loaded bug in nova. Upgrade Notes ************* * To enable the notifications for flavor you need to config *notifications_topics_exchanges = versioned_notifications,nova* under section *resource_plugin:os_nova_flavor* * If you want to search field contains '.', you need to replace '.' with '++', e.g. 'sahara_tag_2.7.1' -> 'sahara_tag_2++7++1'. * Need to specify auth_type or auth_plugin(deprecated) to v2password in service_credentials group to make index work properly if you were using keystone v2 auth. Bug Fixes ********* * Fix a bug that incomplete types may be copyed during reindex. Add checks about data integrity before copying existing Elasticsearch data, warn user if there is any missing index or type mapping and exit. Users can suppress exit behavior by using force parameter. * Searchlight is indexing unused Neutron ports. Some Neutron ports do not result in notifications to Searchlight. We do not want to initially index them. * Replace '.' with '++' when saving documents to Elasticsearch, and convert back when searching results. Changes in searchlight 1.0.0.0rc1..2.0.0 ---------------------------------------- 30e64ae Updated from global requirements 9eacee8 Update UPPER_CONSTRAINTS_FILE for stable/ocata 5b4536d Update .gitreview for stable/ocata 6ba886b Use keystoneauth1 session instead of keystoneclient 1157d5c Handle ironic service not being present c823ebe Disable ironic plugin by default in devstack c7e9e69 Fix filtered queries for Elasticsearch 5 28b2701 Searchlight ironic plugin 741f419 Handle nova rename events 174284c Fix search policy unit test failure 1544a6c Implement flavor notification handler f2a3ce0 Fix server group rbac test 3593e3c Fix typo in utils.py 4b45f22 Updated from global requirements 5252235 Add a hacking to check double word typo automatically 3bd009f Fix HACKING.rst with hacking checks 4e1e6fd Update nova config reference file. 56a9add Removing Unused imports dd4f5ce Implement pipeline afc5ed8 Add metadata field and updated_at field to server group index 4e2ab07 Add server group notification handler c6664ec Support tests coverage report efab8bc Fix multi-typo in searchlight 19ba6ff Fix typo bfdb08e Updated from global requirements aef31cc Fix dot issue in field name 9ee9d89 Remove unused using_elasticsearch_v2 function 97b11aa Trival fix api-ref 10485a1 Updated from global requirements a8be92a Delete deprecated Hacking in tox.ini f974cbc Updated from global requirements 527706f Warn user if types using es to reindex don't exist 231780e Updated from global requirements 706a3fd Add alt text for badges a497b76 Use default jdk in devstack setup f44a90f Add tools/test-setup.sh script to setup test environment ba05d5f Fix typo in Helper.py 6126674 Use ConfigParser instead of SafeConfigParser in Python 3 6fca79e Show team and repo badges on README 1b6ca1a Replace six.iteritems() with .items() 019d33b Duplicated ES reindex when plugins in different groups b471409 Add option to reindex notification-less plugins 3cd5834 Remove uneccessary "in" from CONTRIBUTING.rst 0adbf58 Use messagingv2 by default ddab5e9 Replaces uuid.uuid4 with uuidutils.generate_uuid() 837a61c Modify variable's using method in Log Messages 6571bc6 Fix a typo in manage.py & utils.py c8b328e Fix typo in helper.py 61c2fd1 Allow notification handler cls to be None f9b2d67 Updated from global requirements 556e62e Updated from global requirements 682a7e6 Comment 'man_pages' config in conf.py 362bb12 Fix the issue that plugins list not shown on Searchlight Indexing page 3d1823e Change "Openstack" to "OpenStack" db19cd5 Remove mox3 in test-requirement.txt 6335ec7 Allow 'size' and 'from' as search parameters cfb7fdf Allow listener to process "error" notification priority dbed7e5 Drop MANIFEST.in - it's not needed by pbr 8ba8008 Remove html_static_path from doc ecf68bf Tool to translate a *.po file. 0128b2f Add http_proxy_to_wsgi to api-paste 0eaadd4 Don't include openstack/common in flake8 exclude list 84cb0e2 Enable release notes translation 171733a Fix typo: remove redundant 'the' f903665 Fix typo in notification_handlers.py d987b7f Fix typo in docstring 4b8ff3b Use upper constraints for all jobs in tox.ini f1804f2 Updated from global requirements 0a35be8 Fix a typo in documentation 5f34e51 Fix a typo in authoring-plugins.rst,etc 97b8bcd api-ref: Add Searchlight api-ref 3c60230 Use upper-constraints for tox envs 80a728c Docstrings should not start with a space eabdaf2 TrivialFix: Remove default=None when set value in Config aef7fed RBAC for network sharing f0a4630 Return __searchlight-user-role with highlighting 10d47a0 Restrict image member list 5d50149 Support X_IS_ADMIN_PROJECT header on auth response f9ad0ea Don't listen for dns.zone.exists 13e9cfa Security group rule race condition be5ef61 Log indexing exceptions at 'exception' level 4387547 Updated from global requirements 33a14d4 Do not index some Neutron ports. c15df25 'deleted_at' Date field needed for get_version 88fa016 Cinder mapping incorrect. 60d2b52 Update reno for stable/newton 09dc0f1 Remove duplicate project facets from cinder plugins Closes-Bug: #1558606 Diffstat (except docs and test files) ------------------------------------- .gitreview | 1 + CONTRIBUTING.rst | 2 +- HACKING.rst | 33 +- MANIFEST.in | 16 - README.rst | 14 + api-ref/source/http-codes.yaml | 55 ++ api-ref/source/index.rst | 7 +- .../samples/create-search-admin-request.json | 6 + .../samples/create-search-aggregation-request.json | 11 + .../create-search-aggregation-response.json | 64 ++ .../source/samples/create-search-all-request.json | 5 + .../source/samples/create-search-all-response.json | 61 ++ .../create-search-complex-boolean-request.json | 25 + .../create-search-complex-boolean-response.json | 93 +++ .../samples/create-search-free-text-request.json | 9 + .../samples/create-search-free-text-response.json | 45 ++ .../create-search-multiple-type-request.json | 6 + .../create-search-phrase-field-request.json | 8 + .../create-search-phrase-field-response.json | 45 ++ .../samples/create-search-single-type-request.json | 6 + .../create-search-single-type-response.json | 44 ++ .../list-facets-all-response-exclude-options.json | 78 +++ .../source/samples/list-facets-all-response.json | 90 +++ .../list-facets-include-fields-response.json | 11 + .../source/samples/list-facets-type-response.json | 27 + api-ref/source/samples/list-plugins-response.json | 14 + api-ref/source/search-parameters.yaml | 170 +++++ api-ref/source/searchlight-v1.inc | 696 +++++++++++++++++++++ devstack/local.conf | 11 +- devstack/plugin.sh | 20 +- etc/api-paste.ini | 6 +- etc/policy.json | 9 +- ...roup-notification-handler-d10aed78bede2c12.yaml | 3 + ...nfigurable-index-settings-59e00230f669c1a8.yaml | 2 +- .../bp-flavor-notifications-c9454eb39ab8f567.yaml | 11 + .../notes/bp-ironic-plugin-922204f24ecb2f16.yaml | 3 + .../bp-neutron-tenant-rbac-0922f23111965f7c.yaml | 9 + .../bp-pipeline-architecture-dc2d10d5609af1f9.yaml | 11 + .../bp-reindex-interval-a43c37189e948d28.yaml | 4 + ...bug-1568923-add-size-from-4dd681d72bc0ef6c.yaml | 8 + .../notes/bug-1596710-6a5287f0cd77cba0.yaml | 15 + ...bug-1621914-neutron-ports-58cf966264c09ce4.yaml | 8 + .../notes/bug-1622343-648b6b2b85e67209.yaml | 11 + .../notes/bug-1653752-a34a9a85deb92ed2.yaml | 5 + .../listener-error-level-582765d9d48d4e01.yaml | 3 + releasenotes/source/conf.py | 3 + releasenotes/source/index.rst | 1 + releasenotes/source/newton.rst | 6 + requirements.txt | 27 +- searchlight/api/middleware/context.py | 9 +- searchlight/api/v1/search.py | 94 +-- searchlight/cmd/manage.py | 125 ++-- searchlight/common/config.py | 2 - searchlight/common/property_utils.py | 9 +- searchlight/common/resource_types.py | 4 + searchlight/common/utils.py | 29 +- searchlight/common/wsgi.py | 12 +- searchlight/elasticsearch/__init__.py | 30 +- searchlight/elasticsearch/plugins/base.py | 118 +++- .../elasticsearch/plugins/cinder/__init__.py | 4 +- .../elasticsearch/plugins/cinder/snapshots.py | 1 - .../cinder/snapshots_notification_handler.py | 19 +- .../elasticsearch/plugins/cinder/volumes.py | 3 +- .../plugins/cinder/volumes_notification_handler.py | 23 +- .../plugins/designate/notification_handlers.py | 75 ++- .../elasticsearch/plugins/glance/__init__.py | 7 +- searchlight/elasticsearch/plugins/glance/images.py | 39 +- .../plugins/glance/images_notification_handler.py | 37 +- .../elasticsearch/plugins/glance/metadefs.py | 23 +- .../glance/metadefs_notification_handler.py | 306 +++++---- searchlight/elasticsearch/plugins/helper.py | 52 +- .../elasticsearch/plugins/ironic/__init__.py | 43 ++ .../elasticsearch/plugins/ironic/chassis.py | 61 ++ searchlight/elasticsearch/plugins/ironic/nodes.py | 71 +++ .../plugins/ironic/notification_handlers.py | 187 ++++++ searchlight/elasticsearch/plugins/ironic/ports.py | 71 +++ .../elasticsearch/plugins/ironic/resources.py | 113 ++++ .../elasticsearch/plugins/neutron/__init__.py | 19 + .../elasticsearch/plugins/neutron/networks.py | 57 +- .../plugins/neutron/notification_handlers.py | 342 +++++++--- searchlight/elasticsearch/plugins/neutron/ports.py | 44 +- .../elasticsearch/plugins/neutron/subnets.py | 23 +- searchlight/elasticsearch/plugins/nova/__init__.py | 53 +- searchlight/elasticsearch/plugins/nova/flavors.py | 16 +- .../elasticsearch/plugins/nova/hypervisors.py | 3 - .../plugins/nova/notification_handler.py | 251 ++++++-- .../elasticsearch/plugins/nova/servergroups.py | 4 +- searchlight/elasticsearch/plugins/nova/servers.py | 2 + .../elasticsearch/plugins/openstack_clients.py | 96 +-- .../elasticsearch/plugins/swift/__init__.py | 7 +- .../elasticsearch/plugins/swift/containers.py | 16 +- searchlight/elasticsearch/plugins/swift/objects.py | 28 +- .../plugins/swift/swift_notification_handler.py | 84 ++- searchlight/elasticsearch/plugins/utils.py | 46 +- searchlight/hacking/checks.py | 16 + searchlight/listener.py | 58 +- searchlight/opts.py | 8 +- searchlight/pipeline.py | 120 ++++ searchlight/publisher/__init__.py | 0 searchlight/publisher/log.py | 29 + searchlight/service_policies.py | 3 +- .../unit/test_neutron_securitygroup_plugin.py | 126 +++- setup.cfg | 5 + test-requirements.txt | 13 +- tools/check_translate.sh | 127 ++++ tools/test-setup.sh | 14 + tox.ini | 15 +- 162 files changed, 6741 insertions(+), 1340 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index b90f6f9..9d189ea 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5 +5 @@ -pbr>=1.6 # Apache-2.0 +pbr>=1.8 # Apache-2.0 @@ -16,2 +16,2 @@ Routes!=2.0,!=2.3.0,>=1.12.3;python_version!='2.7' # MIT -WebOb>=1.2.3 # MIT -oslo.config>=3.14.0 # Apache-2.0 +WebOb>=1.6.0 # MIT +oslo.config!=3.18.0,>=3.14.0 # Apache-2.0 @@ -22,3 +22,3 @@ oslo.service>=1.10.0 # Apache-2.0 -oslo.utils>=3.16.0 # Apache-2.0 -stevedore>=1.16.0 # Apache-2.0 -keystonemiddleware!=4.1.0,!=4.5.0,>=4.0.0 # Apache-2.0 +oslo.utils>=3.18.0 # Apache-2.0 +stevedore>=1.17.1 # Apache-2.0 +keystonemiddleware>=4.12.0 # Apache-2.0 @@ -30 +30 @@ Paste # MIT -python-keystoneclient!=2.1.0,>=2.0.0 # Apache-2.0 +python-keystoneclient>=3.8.0 # Apache-2.0 @@ -36,3 +36,3 @@ oslo.i18n>=2.1.0 # Apache-2.0 -oslo.log>=1.14.0 # Apache-2.0 -oslo.messaging>=5.2.0 # Apache-2.0 -oslo.policy>=1.9.0 # Apache-2.0 +oslo.log>=3.11.0 # Apache-2.0 +oslo.messaging>=5.14.0 # Apache-2.0 +oslo.policy>=1.17.0 # Apache-2.0 @@ -47,2 +47,2 @@ python-designateclient>=1.5.0 # Apache-2.0 -python-glanceclient!=2.4.0,>=2.3.0 # Apache-2.0 -python-novaclient!=2.33.0,>=2.29.0 # Apache-2.0 +python-glanceclient>=2.5.0 # Apache-2.0 +python-novaclient!=7.0.0,>=6.0.0 # Apache-2.0 @@ -51 +51,2 @@ python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0 -python-swiftclient>=2.2.0 # Apache-2.0 +python-swiftclient>=3.2.0 # Apache-2.0 +python-ironicclient>=1.11.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 2459aff..63f85cc 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -12 +12 @@ Babel>=2.3.4 # BSD -coverage>=3.6 # Apache-2.0 +coverage>=4.0 # Apache-2.0 @@ -14 +13,0 @@ fixtures>=3.0.0 # Apache-2.0/BSD -mox3>=0.7.0 # Apache-2.0 @@ -16,2 +15,2 @@ mock>=2.0 # BSD -sphinx!=1.3b1,<1.3,>=1.2.1 # BSD -requests>=2.10.0 # Apache-2.0 +sphinx!=1.3b1,<1.4,>=1.2.1 # BSD +requests!=2.12.2,>=2.10.0 # Apache-2.0 @@ -20 +19 @@ testtools>=1.4.0 # MIT -psutil<2.0.0,>=1.1.1 # BSD +psutil>=3.0.1 # BSD @@ -22 +21 @@ oslotest>=1.10.0 # Apache-2.0 -reno>=1.8.0 # Apache2 +reno>=1.8.0 # Apache-2.0 @@ -28 +27 @@ os-api-ref>=1.0.0 # Apache-2.0 -oslosphinx!=3.4.0,>=2.5.0 # Apache-2.0 +oslosphinx>=4.7.0 # Apache-2.0