[openstack-announce] [new][searchlight] searchlight 1.0.0 release (newton)

We are delighted to announce the release of: searchlight 1.0.0: OpenStack Search Service This release is part of the newton release series. For more details, please see below. 1.0.0 ^^^^^ The initial mitaka release (0.2) did not support ElasticSearch 2.x. ElasticSearch made several changes to internal index data mappings that were not backwards compatible with ElasticSearch 1.x. This release has all the changes required for Searchlight to support both ElasticSearch 1.x and 2.x. Searchlight now supports oslo pools instead of requiring a different topic for each listener. This simplifies deployment configuration. Previously, services would have to be specifically configured to send notifications to the Searchlight topic. With this change, Searchlight can share topics with other services such as Ceilometer. Adds ability to include region_name in mapping for multi-region deployments. Allow aggregations as part of a search request. Add new functionality to searchlight-manage to force a re-indexing from Elasticsearch only. This option allows new mappings to be applied to the Elasticsearch indices without the overhead of retrieving data from the plugin APIs. Allow dynamic index settings to allow Searchlight to be tuned differently from other indices on a shared Elasticsearch cluster. Searchlight will provide resource type info for any reference IDs in the facet results. Support version as a search param to include _version in search results. Adds the ability to attain a document count per resource type from the /facets endpoint. Doc count is always returned for a type; field listing is optional. Added multi-thread support for indexing. Allow wildcard character as an argument to searchlight-manage index sync --type Added support for neutron floating IPs. Added suppport for neutron security groups and security group rules. Add support for the new Neutron tenant RBAC policy functionality. This allows networks to be shared across tenants. Add support to searchlight-manage for displaying with Elasticsearch indices/aliases are currently being used by Searchlight. Reduce the number of calls to the nova API on instance boots. The ability to distribute resource type indexing across resource groups has now been fully enabled. This allows for improved indexing speed and clustering customization. Adds support for service policy files. Simplifies existing API policy support. Added project_id to mappings. Disable returning options for the facet fields. Add checks for resource group name configuration. Correct facet options on analyzed string fields. Fix the information returned by the API plugins endpoint. No longer index the data of child plugins as part of indexing a parent plugin, which fixes ConflictErrors occuring specifying parent plugins as the argument to searchlight-manage index sync --type. Fix routing on delete operations for neutron resources. Remove support for Designate v1 api because it has already been deprecated. Simplify list zones and recordsets code by using v2 api's built-in pagination. Add "updated_at" field to all mappings. Some Neutron ports that do not send notificatons to Searchlight. Do not initially index them. Fix mappings for fields erroneously marked as 'nested', and allow facet options for 'object' fields. Improved index sync performance. Change the way of doing facets search, replace use of search_type=count with size=0 New Features ************ * Added microversion support for Nova plugins, the default API version for Nova plugins is 2.1. User can change the API microversion for Nova using compute_api_version config option added in section service_credentials:nova. * Adds nova plugin for server groups. * Adds configuration option to include region_name in documents to assist with multi-region tribe node Elasticsearch setups. * Allows aggregations to be requested (as 'aggs' or 'aggregations' in a search request body). See https://www.elastic.co/guide/en/elasticsearch/reference/current /search-aggregations.html * Add a new option to the searchlight-manage index command to force a re-indexing from Elasticsearch only. The plugin APIs will not be used with option. * Allows a deployer to override some index-level Elasticsearch settings for all Searchlight-created indices. This is useful when running Searchlight on a shared Elasticsearch cluster. Settings are overriden using the elasticsearch.index_settings config option. * Searchlight will provide resource type info for any reference IDs in the facet results. Parent reference types will also be part of the info This will allow UI's and CLI's to explore the data seamlessly * version is supported as a search parameter to return an always- incrementing _version value with each search result. * Includes 'doc_count' in /facets response for each resource type. * Adds nova plugin for flavors. * Adds nova plugin for hypervisors. * When searchlight-manage is indexing, all resource types will be indexed in a separate thread. Multiple threads will be run concurrently. * The --type argument to searchlight-manage index sync accepts a wildcard as the final character. * Adds neutron floating IP support (OS::Neutron::FloatingIP). * Added suppport for neutron security groups and security group rules (OS::Neutron::SecurityGroup). * Add support for the new Neutron tenant RBAC policy functionality. This allows networks to be shared across tenants. * Add support to searchlight-manage for displaying with Elasticsearch indices/aliases are currently being used by Searchlight. The indices/aliases are based on the specified configuration file. * Replace some of the nova get server API calls with partial document updates that just update the OS-EXT-STS state fields. * Reenabled multiple indices by allowing plugins to be managed under different resource groups. * API configuration can include service policy files such that access to various resources can match whether their service API allows access for a user. * Openstack is moving away from 'tenant' as the nomenclature for keystone projects, preferring 'project' instead. Some services already return tenant_id and project_id (or just project_id); this patch adds project_id to those that do not. * Currently the "facet" endpoint will return all information relating to a facet field, including the options. The new parameter "exclude_options" to the facet endpoint will prevent the options for all facet fields from being returned. This option will avoid an aggregation query being performed on Elasticsearch, improving performance. The default behavior (not specifying this option) is return the options for all facet fields. Known Issues ************ * There are no notifications for server groups from nova yet, so we recommend putting it in its own resource group and scheduling a cron job to periodically re-sync. This will create a very low overhead way to keep the index up to date. The index latency will be dependent on how often you re-sync the data. * This is a change that is not compatible with the API response in Mitaka; changes have been made in Newton to the python client and UI. * There are no notifications for flavors from nova yet, so we recommend putting it to its own resource group and scheduling a cron job to re-sync with little overhead. * There are no notifications for hypervisor from nova yet, so we recommend putting it to its own resource group and scheduling a cron job to re-sync with little overhead. * Currently no notifications are sent on status changes for floating IPs (see https://bugs.launchpad.net/neutron/+bug/1593793) * Using multiple resource groups will impact sort behavior when sorting on fields across resource types when all the resource types don't have the requested 'sort-by field'. Follow the guidelines below to avoid errors. https://www.elastic.co/guide/en/elasticsearch/reference/current /search-request-sort.html#_ignoring_unmapped_fields https://www.elastic.co/guide/en/elasticsearch/reference/current /search-request-sort.html#_missing_values * Deployment should use Designate v2 api. if v1 api is in use, designate plugin won't function correctly. Upgrade Notes ************* * To support ElasticSearch 1.x and 2.x, several internal data mappings have been updated. If you previously deployed Searchlight, you will need to reindex your resource data using "searchlight- manage index sync". You may re-index everything or limit it to "OS::Nova::Server", "OS::Glance::Image", "OS::Glance::Metadef", "OS::Cinder:Volume", and "OS::Cinder::Snapshot". * To start using a pool with a shared topic instead of separate topics: * Update the respective service configuration files for searchlight enabled plugins to only publish to a single topic (e.g. set "notification_topics = notifications"). If you have Ceilometer enabled, this must be the same topic which Ceilometer uses. You also typically must restart each service for the change to take effect. For example, in "nova.conf", "glance-api.conf", "cinder.conf", "neutron.conf", and "designate.conf": notification_topics = notifications * Update "searchlight.conf". In the "[resource_plugin]" section, set "notifications_topic" to match the shared topic that you set in the service configurations. In the "[listener]" sections, set "notifications_pool" to your desired pool name. The default name starting in Newton is "searchlight". The "notifications_pool" does not have to match anything from other services configuration files. For example: [resource_plugin] notifications_topic = notifications [listener] notifications_pool = searchlight * A version of python-searchlightclient greater than 2.1 must be used with this change, as well as the Newton UI version. * The existing searchlight policy.json has been simplified. The only supported options for resource policy are to enable or disable it for all request types (i.e. the 'allow', 'query' 'facet' suffixes are no longer in effect). * Mappings have changed for several types; it is strongly recommended all data be reindexed and the API service restarted. Deprecation Notes ***************** * Two fields returned in the API plugins endpoint 'name' and 'index' have been removed. Bug Fixes ********* * Bug 1570213 Apply query to highlight query * Bug 1532010 Ensure consistency in mapping field types * Bug 1570674 Fix unicode error when booting instance from volume * Bug 1570199 Fix inconsistent mapping in image plugin * Bug 1568709 Remove port.create.end handler from nova * Bug 1565015 Add volume.retype event * Bug 1583215 Correct Cinder exchange value * Bug 1583215 Enable notification messaging pools * Overwrite child plugin's resource_group_name during parent registration. Child plugins must live on the same resource group as their parents. This fixes a issue that a misconfiguration of child plugin's resource_group_name results in inconsistentency with the parent plugin. * Facet options don't work well with analyzed string fields, because the aggregation runs on tokens. These fields have non-analyzed raw subfields for sorting. Include raw subfields in facet results so that users can use them to do exact matches. * The API plugins API endpoint is returning outdated, duplicated and incomplete information. The hat-trick of uselessness. This fix will correct that. The duplicated field ('name') has been removed. The outdated field has been renamed (from 'index' to 'alias-search'). The missing field has been added ('alias-listener'). * Bug 1584148 Version conflict during re-indexing * Fixes a bug where routing wasn't specified for delete operations on neutron child resources. * Fixes some compatibility issues around metafield access in es 1.x versus es 2.x. * All Elasticsearch mappings need to include the field "udated_at". This field is used by Searchlight UI to sort the results. * Searchlight is indexing unused Neutron ports. Some Neutron ports do not result in notifications to Searchlight. We do not want to initially index them. * Bug 1619047 Unnest nova server flavor and image mapping properties Other Notes *********** * The maximum number of threads that will run concurrently can be controlled by setting "max_workers" in the Searchlight configuration file. [manage] max_workers = 6 * Searchlight uses configured resource group name to create ElasticSearch indices. Some characters are invalid to index names and will cause faiure of index creation. Add verification to check if resource group name is valid. * In mitaka, indexing a parent plugin also reindexed the data for any children it had. That is no longer the case. * Improved searchlight-manage index sync performance by disabling automatic refresh of newly created indices until they are fully populated and activated for search queries. * Facets search is aggregation-only, no actual documents need to be returned. We do it by setting search_type in search request before, but it has been deprecated in Elasticsearch 2.x, use recommended size=0 instead. Changes in searchlight 0.2.0.0rc1..1.0.0 ---------------------------------------- 58b7ed1 api-ref: Add Searchlight api-ref def91d6 Pin tox upper-constraints for stable/newton 79d21a9 Return __searchlight-user-role with highlighting 7fb2e41 RBAC for network sharing 42b7d7b Support X_IS_ADMIN_PROJECT header on auth response 31f19de 'deleted_at' Date field needed for get_version 12950c7 Restrict image member list 54e05ec Do not index some Neutron ports. 6550b19 Cinder mapping incorrect. 99c2e77 Log indexing exceptions at 'exception' level 6eda437 Security group rule race condition 00f7edc Update .gitreview for stable/newton 4ca188e Indicate 'nested' on facet types 87fc192 Disable returning options for the facet fields. 0f9e6f1 Add "updated_at" field to all mappings. 177818b Handle nova error states correctly 241ec28 standardize release note page ordering 3625a7c Allow aggregations in search requests 3f406fa Change Elasticsearch installation doc 3d47b25 Change Designate recordset mapping ed1bfac Replace some 'nested' mapping types with 'object' dbdbb67 [api-ref] Remove temporary block code in conf.py 8ff2c39 to_dict instead of __dict__ for cinder resource 0b99d8e Verify resource group name 18b120f Fix exceptions error for image plugin 304362e Fix translation of exceptions f8f7d24 Use more specific asserts in tests d8ea63b Revert "Allow aggregations in search requests" 004bfc1 Raise NotImplementedError instead of NotImplemented 58a8870 Add region name to mappings and documents f8e4e3e Allow aggregations in search requests fb0676d Config logABug feature for searchlight api-ref 71510e9 Updated from global requirements afbac8e API plugin endpoint needs aliases d783c87 TrivialFix: Remove logging import unused 77ac53b Updated from global requirements 0d64da5 Get ready for os-api-ref sphinx theme change 0233200 Updated from global requirements 328f825 Service policy file support 3d7b4a1 Document the Searchlight architecture 535a3da Add nova server groups plugin 26e73de :Move elasticsearch install to plugin, use v2.3.4 63f0f56 Ignore nova deleting state changes 001e471 Force Elasticsearch indexing 1a9e06d Updated from global requirements f25b28b Fix nova exchange exception 3fd15c8 Add warning for plugins without notifications f408598 api-ref: add base structure 87fc290 Updated from global requirements 2652ed5 Add totals to facets 194e778 Support 'shelve' and 'reboot' instance notifications 5cda277 Allow wildcard in index --type argument f7d1a48 Flavor Plugin ae6a7b7 Log notification events 18c2061 Fixing typo in manage.py a7e1a8a Remove discover from test-requirements 558a6c5 Display ES indices 72c2b86 Updated from global requirements c575f3e Speed up Searchlight indexing 4fc6786 Small improvement for dynamic index settings doc fa70962 Fix --type handling for indexing parent plugins dcec06c Cleanup i18n marker functions to match Oslo docs 5d1030a Updated from global requirements 056be20 Remove designate v1 artifacts 79b0efd A typo commit in the documentation 7d0146a Add Neutron security group plugin 993811f Add Python 3.5 classifier and venv 771c5e6 Replace assertEqual(None, *) with assertIsNone in tests 89acc42 Fixes: Typo Error in helper.py 01eee1d Correct reraising of exception 6ad19a5 Spelling fixes in README 4fc01db Updated from global requirements 6a87126 Config: no need to set default=None bee851b Allow dynamic index settings 0e4c767 Reduce number of API calls on nova boot 85680b4 Neutron floating IP support. fbfef7b Expand designate functional tests 8736baa Fix typo errors in the description 0b6a5c0 make string.letters python3 compatible 32fd04b Handle JSON object in hypervisor serialization d36e9ed Updated from global requirements f6fe6fd Updated from global requirements 217480f Enabled Neutron notifications in sample local.conf a15e3e6 Determine routing before delete on neutron events 21042fd Add microversion support for nova plugins 0074de2 Image type should be a facet 028943d Expose 'version' as a search parameter 33066df Correct doc_values info in authoring plugins docs 2e9bac3 Updated from global requirements 3beda06 Change Nova API version to V2.1 04ec2a1 Add project_id to all serialized documents 3e64068 Fix reindexing conflict handling in es2 e58d3f3 Correct es 2.x functional test config bf693c5 Imported Translations from Zanata fdd0c01 Updated from global requirements 94a2660 Return raw subfields on facets of analyzed string fields 1d440ff Updated from global requirements 09b81c4 Don't accept domain-scoped tokens for auth 87ad59b Fix python3 compatible issue in listener 4074746 python 3.4 support for urlparse a924228 Remove mention of oslo.log 'verbose' option 16446d9 Correct wrong link in doc 787752a Use Notification Pools by Default ea75c13 Added os_endpoint_type to various OpenStack clients 6798c5d Add Release notes for fixes backported to 0.2.1 942de0a Enable notification messaging pools 83f389a Updated from global requirements e4878e0 Adds Nova Functional Tests f2ed51b Correct cinder 'exchange' value 7f56603 Resource types for reference and parent id 75f46cb Replace use of search_type=count with size=0 54a0687 Updated from global requirements 204d7b4 Apply query to highlight query 21b2960 Include zone_id in Designate Recordset mapping a2dbacc remove verbose option in devstack plugin 31f58bf Simple Script for Generating Resources ebcb52b Overwrite child plugin resource_group_name durning parent registration d2b536f Updated from global requirements 4592cdf Improve plugin authoring information 4f56a1e Reenable multiple elasticsearch indices e55c7ff Updated from global requirements 67354da Ensure consistency in mapping field types 0d927ef Add freeform search syntax to docs f7f5573 Imported Translations from Zanata 1142b72 Error deleting index in initial syncing 2d1e87a Updated from global requirements 5d9bcba use thread safe fnmatch 83d6e6b Document that initial sync is mandatory 730f9f1 Hypervisor plugin 3bb82e0 Allow limit=0 In Searches 789f39a [Trivial] Remove unnecessary executable privilege aeb4fc2 Make func test indexing more consistent 101c734 Add cleanup in functional tests e6f649c Making searchlight-manage python-3 compatible c7e4fcd Fix unicode error when booting instance from volume 4d06c6a Imported Translations from Zanata 2a16363 Updated from global requirements ccbb9b2 Add release note for index refresh disabling 81f7956 Zero Downtime Re-indexing Functional Tests. 1e4f0f6 Updated from global requirements 195a1f7 Add volume.retype event 93c6cfa Remove port.create.end handler from nova 725862a Breaking up utils.py 477d508 Disable index refresh during data re-sync 5db5612 Fix inconsistent mapping in image plugin 8eb705d Use delete_index instead of alias_listener_update 8bcca43 Updated from global requirements 52146ec Add region_name to service_credentials for devstack 7ad2648 Updated from global requirements d9b1230 Listen to instance volume attach/detach events acd51ea Add searchlight-ui to example local.conf 99b0e51 Add Link to client and ui ca1aeed Zero Downtime Re-indexing Unit Tests. 800b46c Proposed clarification on zero-downtime reindexing doc 2a44b63 Backward compatibility with designate v1 api 4abf961 Disable oslo_config file discovery in tests 3b7dc51 Backward compatibility with designate v1 api bcc8243 Increase heap for elasticsearch in func tests 1e80553 Fix _is_multiple_alias_exception signature 574a095 Add missing zero-downtime indexing documentation 8a287fc Increase heap for elasticsearch in func tests 3b132f8 Eliminate stray asterisk 600b691 Fix _is_multiple_alias_exception signature a73cd1d Add missing zero-downtime indexing documentation 9fecc29 Disable oslo_config file discovery in tests bb5ff07 Re-indexing optimization for doc_type cb2137b Re-indexing optimization for doc_type 25ca131 Add missing mitaka release notes 531447d Zero Downtime Re-indexing Error Handling. bc9eeaa Zero Downtime Re-indexing Error Handling. b1d339b Don't index DHCP ports ba67c2c Add missing mitaka release notes 3826498 Update reno for stable/mitaka a5ccab8 Update .gitreview for stable/mitaka 22e05a8 Don't index DHCP ports d5c3b46 Replace deprecated LOG.warn with LOG.warning Diffstat (except docs and test files) ------------------------------------- .gitreview | 1 + README.rst | 2 +- api-ref/source/conf.py | 240 ++++++ api-ref/source/http-codes.yaml | 55 ++ api-ref/source/index.rst | 25 + .../samples/create-search-admin-request.json | 6 + .../samples/create-search-aggregation-request.json | 11 + .../create-search-aggregation-response.json | 64 ++ .../source/samples/create-search-all-request.json | 5 + .../source/samples/create-search-all-response.json | 61 ++ .../create-search-complex-boolean-request.json | 25 + .../create-search-complex-boolean-response.json | 93 +++ .../samples/create-search-free-text-request.json | 9 + .../samples/create-search-free-text-response.json | 45 ++ .../create-search-multiple-type-request.json | 6 + .../create-search-phrase-field-request.json | 8 + .../create-search-phrase-field-response.json | 45 ++ .../samples/create-search-single-type-request.json | 6 + .../create-search-single-type-response.json | 44 ++ .../list-facets-all-response-exclude-options.json | 78 ++ .../source/samples/list-facets-all-response.json | 90 +++ .../list-facets-include-fields-response.json | 11 + .../source/samples/list-facets-type-response.json | 27 + api-ref/source/samples/list-plugins-response.json | 14 + api-ref/source/search-parameters.yaml | 170 ++++ api-ref/source/searchlight-v1.inc | 695 ++++++++++++++++ devstack/README.rst | 6 +- devstack/local.conf | 65 +- devstack/plugin.sh | 127 ++- etc/policy.json | 36 +- images/Figure1.png | Bin 0 -> 38618 bytes images/Figure2.png | Bin 0 -> 183985 bytes images/Figure3.png | Bin 0 -> 158874 bytes images/Figure4.png | Bin 0 -> 239153 bytes .../notes/add-0_2_1-notes-e1f8910ad27d88bb.yaml | 62 ++ ...nova-microversion-support-8b40da4458678f20.yaml | 6 + .../add-server-groups-plugin-3dff1b85a3f716f5.yaml | 10 + .../notes/bp-add-region-name-69fbe5b84b2883c6.yaml | 8 + ...allow-aggregation-queries-fd7d39cb33068be1.yaml | 8 + ...indexing-into-new-mapping-bd78e94842fd2705.yaml | 11 + ...nfigurable-index-settings-59e00230f669c1a8.yaml | 11 + ...source-type-relationships-d697d0aa6648a560.yaml | 9 + .../notes/bp-expose-version-be6403da43341c98.yaml | 8 + .../notes/bp-facet-totals-369e7cd81b15f658.yaml | 16 + releasenotes/notes/bp-flavor-plugin.yaml | 7 + .../bp-hypervisor-plugin-5d17ed1005004b43.yaml | 7 + ...x-performance-enhancement-c8dc21da98f45628.yaml | 12 + .../bp-index-role-separation-535e01ba3bb2252e.yaml | 11 + ...bp-indexing-type-wildcard-94c38a719a9210af.yaml | 7 + .../bp-neutron-floatingip-1291f3a76f0e4aed.yaml | 8 + .../bp-neutron-securitygroup-6c1531e9f80030bd.yaml | 7 + .../bp-neutron-tenant-rbac-0922f23111965f7c.yaml | 9 + .../bp-orphaned-es-aliases-24c5b5dcd2eb3a41.yaml | 10 + .../bp-reduce-nova-api-calls-64eef45bd19fa89c.yaml | 8 + ...reenable-multiple-indices-3c05b9dde9d45d62.yaml | 13 + ...port-service-policy-files-c7e5eeb83c6f4126.yaml | 13 + ...-zero-downtime-reindexing-a607d52b8711d845.yaml | 24 + .../bug-1545069-project-id-f8fe895ecce095ff.yaml | 10 + .../notes/bug-1547118-3cfde976f32c3b5c.yaml | 9 + ...bug-1557778-facet-options-798bba4cb4e3067a.yaml | 13 + .../notes/bug-1558630-fb26ae51f17fa9e2.yaml | 8 + .../notes/bug-1558884-231498c4a6f56a31.yaml | 9 + .../bug-1583681-plugin-info-d7307783f36cad25.yaml | 13 + ...bug-1584148-indexing-type-07ea84db2e6c1da4.yaml | 11 + ...g-1588540-neutron-routing-e7187a02535d8b01.yaml | 8 + .../notes/bug-1592984-34d9417422c37e37.yaml | 9 + .../bug-16175150-fix-mapping-a51e535a13deed96.yaml | 7 + ...bug-1621914-neutron-ports-58cf966264c09ce4.yaml | 8 + ...s-1619047-nested-mappings-4ddb2d0b067b3093.yaml | 10 + ...refresh-during-reindexing-e0152b6524385745.yaml | 7 + ...use-size-in-facets-search-438d4c4317879dfc.yaml | 9 + releasenotes/source/index.rst | 3 +- releasenotes/source/mitaka.rst | 6 + requirements.txt | 34 +- searchlight/api/__init__.py | 0 searchlight/api/common.py | 5 - searchlight/api/middleware/context.py | 22 +- searchlight/api/middleware/gzip.py | 3 +- searchlight/api/middleware/version_negotiation.py | 6 +- searchlight/api/policy.py | 80 +- searchlight/api/v1/__init__.py | 0 searchlight/api/v1/router.py | 0 searchlight/api/v1/search.py | 115 ++- searchlight/api/versions.py | 6 +- searchlight/cmd/__init__.py | 13 +- searchlight/cmd/api.py | 4 + searchlight/cmd/control.py | 4 +- searchlight/cmd/manage.py | 410 ++++++++-- searchlight/common/config.py | 4 +- searchlight/common/exception.py | 10 +- searchlight/common/property_utils.py | 12 +- searchlight/common/resource_types.py | 45 ++ searchlight/common/utils.py | 46 +- searchlight/common/wsgi.py | 12 +- searchlight/context.py | 15 + searchlight/elasticsearch/__init__.py | 36 +- searchlight/elasticsearch/plugins/base.py | 425 ++++++---- .../elasticsearch/plugins/cinder/__init__.py | 7 +- .../elasticsearch/plugins/cinder/snapshots.py | 27 +- .../cinder/snapshots_notification_handler.py | 18 +- .../elasticsearch/plugins/cinder/volumes.py | 36 +- .../plugins/cinder/volumes_notification_handler.py | 12 +- .../elasticsearch/plugins/designate/__init__.py | 29 +- .../plugins/designate/notification_handlers.py | 54 +- .../elasticsearch/plugins/designate/recordsets.py | 39 +- .../elasticsearch/plugins/designate/zones.py | 34 +- .../elasticsearch/plugins/glance/__init__.py | 21 +- searchlight/elasticsearch/plugins/glance/images.py | 70 +- .../plugins/glance/images_notification_handler.py | 13 +- .../elasticsearch/plugins/glance/metadefs.py | 48 +- .../glance/metadefs_notification_handler.py | 6 +- searchlight/elasticsearch/plugins/helper.py | 619 +++++++++++++++ .../elasticsearch/plugins/neutron/__init__.py | 35 + .../elasticsearch/plugins/neutron/floatingips.py | 108 +++ .../elasticsearch/plugins/neutron/networks.py | 65 +- .../plugins/neutron/notification_handlers.py | 357 ++++++++- searchlight/elasticsearch/plugins/neutron/ports.py | 41 +- .../elasticsearch/plugins/neutron/routers.py | 29 +- .../plugins/neutron/security_groups.py | 100 +++ .../elasticsearch/plugins/neutron/subnets.py | 25 +- searchlight/elasticsearch/plugins/nova/__init__.py | 70 +- searchlight/elasticsearch/plugins/nova/flavors.py | 110 +++ .../elasticsearch/plugins/nova/hypervisors.py | 106 +++ .../plugins/nova/notification_handler.py | 351 +++++++++ .../elasticsearch/plugins/nova/servergroups.py | 84 ++ searchlight/elasticsearch/plugins/nova/servers.py | 98 ++- .../plugins/nova/servers_notification_handler.py | 101 --- .../elasticsearch/plugins/openstack_clients.py | 72 +- .../elasticsearch/plugins/swift/__init__.py | 4 +- .../elasticsearch/plugins/swift/accounts.py | 11 +- .../elasticsearch/plugins/swift/containers.py | 13 +- searchlight/elasticsearch/plugins/swift/objects.py | 18 +- .../plugins/swift/swift_notification_handler.py | 29 +- searchlight/elasticsearch/plugins/utils.py | 755 ++++++------------ searchlight/listener.py | 46 +- .../locale/de/LC_MESSAGES/searchlight-log-info.po | 47 ++ searchlight/locale/de/LC_MESSAGES/searchlight.po | 445 +++++++++++ .../locale/es/LC_MESSAGES/searchlight-log-info.po | 47 ++ searchlight/locale/es/LC_MESSAGES/searchlight.po | 413 ++++++++++ searchlight/locale/fr/LC_MESSAGES/searchlight.po | 424 ++++++++++ searchlight/locale/it/LC_MESSAGES/searchlight.po | 415 ++++++++++ searchlight/locale/ja/LC_MESSAGES/searchlight.po | 390 +++++++++ .../ko_KR/LC_MESSAGES/searchlight-log-info.po | 47 ++ .../locale/ko_KR/LC_MESSAGES/searchlight.po | 385 +++++++++ .../pt_BR/LC_MESSAGES/searchlight-log-info.po | 47 ++ .../locale/pt_BR/LC_MESSAGES/searchlight.po | 413 ++++++++++ searchlight/locale/ru/LC_MESSAGES/searchlight.po | 405 ++++++++++ .../locale/tr_TR/LC_MESSAGES/searchlight.po | 390 +++++++++ .../locale/zh_CN/LC_MESSAGES/searchlight.po | 375 +++++++++ .../locale/zh_TW/LC_MESSAGES/searchlight.po | 374 +++++++++ searchlight/notifier.py | 7 - searchlight/plugin.py | 2 +- searchlight/service_policies.py | 103 +++ .../unit/test_neutron_securitygroup_plugin.py | 213 +++++ setup.cfg | 6 + test-requirements.txt | 14 +- test-scripts/README.rst | 22 + test-scripts/generate-swift-data.py | 9 +- test-scripts/listener.py | 72 ++ test-scripts/resources.sh | 214 +++++ tools/colorizer.py | 3 +- tox.ini | 19 +- 231 files changed, 20319 insertions(+), 2387 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index f8bcd25..b90f6f9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,2 +14,2 @@ PasteDeploy>=1.5.0 # MIT -Routes!=2.0,!=2.1,>=1.12.3;python_version=='2.7' # MIT -Routes!=2.0,>=1.12.3;python_version!='2.7' # MIT +Routes!=2.0,!=2.1,!=2.3.0,>=1.12.3;python_version=='2.7' # MIT +Routes!=2.0,!=2.3.0,>=1.12.3;python_version!='2.7' # MIT @@ -17,3 +17,3 @@ WebOb>=1.2.3 # MIT -oslo.config>=3.7.0 # Apache-2.0 -oslo.concurrency>=3.5.0 # Apache-2.0 -oslo.context>=0.2.0 # Apache-2.0 +oslo.config>=3.14.0 # Apache-2.0 +oslo.concurrency>=3.8.0 # Apache-2.0 +oslo.context>=2.9.0 # Apache-2.0 @@ -21,4 +21,4 @@ oslo.middleware>=3.0.0 # Apache-2.0 -oslo.service>=1.0.0 # Apache-2.0 -oslo.utils>=3.5.0 # Apache-2.0 -stevedore>=1.5.0 # Apache-2.0 -keystonemiddleware!=4.1.0,>=4.0.0 # Apache-2.0 +oslo.service>=1.10.0 # Apache-2.0 +oslo.utils>=3.16.0 # Apache-2.0 +stevedore>=1.16.0 # Apache-2.0 +keystonemiddleware!=4.1.0,!=4.5.0,>=4.0.0 # Apache-2.0 @@ -30 +30 @@ Paste # MIT -python-keystoneclient!=1.8.0,!=2.1.0,>=1.6.0 # Apache-2.0 +python-keystoneclient!=2.1.0,>=2.0.0 # Apache-2.0 @@ -37,2 +37,2 @@ oslo.log>=1.14.0 # Apache-2.0 -oslo.messaging>=4.0.0 # Apache-2.0 -oslo.policy>=0.5.0 # Apache-2.0 +oslo.messaging>=5.2.0 # Apache-2.0 +oslo.policy>=1.9.0 # Apache-2.0 @@ -41 +41 @@ oslo.serialization>=1.10.0 # Apache-2.0 -osprofiler>=1.1.0 # Apache-2.0 +osprofiler>=1.4.0 # Apache-2.0 @@ -43 +43 @@ osprofiler>=1.1.0 # Apache-2.0 -elasticsearch<2.0,>=1.3.0 # Apache-2.0 +elasticsearch<=3.0.0,>=2.0.0 # Apache-2.0 @@ -47 +47 @@ python-designateclient>=1.5.0 # Apache-2.0 -python-glanceclient>=2.0.0 # Apache-2.0 +python-glanceclient!=2.4.0,>=2.3.0 # Apache-2.0 @@ -49,2 +49,2 @@ python-novaclient!=2.33.0,>=2.29.0 # Apache-2.0 -python-neutronclient!=4.1.0,>=2.6.0 # Apache-2.0 -python-cinderclient>=1.3.1 # Apache-2.0 +python-neutronclient>=5.1.0 # Apache-2.0 +python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index 3cc4eee..2459aff 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -9 +9 @@ hacking<0.11,>=0.10.0 -Babel>=1.3 # BSD +Babel>=2.3.4 # BSD @@ -13,2 +13 @@ coverage>=3.6 # Apache-2.0 -discover # BSD -fixtures>=1.3.1 # Apache-2.0/BSD +fixtures>=3.0.0 # Apache-2.0/BSD @@ -16,3 +15,3 @@ mox3>=0.7.0 # Apache-2.0 -mock>=1.2 # BSD -sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2 # BSD -requests!=2.9.0,>=2.8.1 # Apache-2.0 +mock>=2.0 # BSD +sphinx!=1.3b1,<1.3,>=1.2.1 # BSD +requests>=2.10.0 # Apache-2.0 @@ -23 +22 @@ oslotest>=1.10.0 # Apache-2.0 -reno>=0.1.1 # Apache2 +reno>=1.8.0 # Apache2 @@ -27,0 +27 @@ httplib2>=0.7.5 # MIT +os-api-ref>=1.0.0 # Apache-2.0