We are pumped to announce the release of: zaqar 3.0.0: OpenStack Queuing and Notification Service This release is part of the newton release series. For more details, please see below. 3.0.0 ^^^^^ New Features ************ * Add a new webhook notifier using trust authentication. When using the 'trust+' URL prefix, Zaqar will create a Keystone trust for the user, and then use it when a notification happens to authenticate against Keystone and send the token to the endpoint. * Support 'post_data' and 'post_headers' options on subscribers, allowing customization of the payload when having a webhook subscriber. The 'post_data' option supports the '$zaqar_message$' string template, which will be replaced by the serialized JSON message if specified. * Queues now behave lazy in subscriptions also. So there is no need for the user to pre-create a queue before creating a subscription for this queue. Zaqar will create the queue automatically on the subscription creation request. As before, all subscriptions will continue to stay active even if the corresponding queue was deleted. * Currently Zaqar can support more built-in/reserved attributes in queue. For now there are two important attributes 'max_messages_post_size' and 'max_message_ttl'. With this feature, when user query queues Zaqar will show those two attributes (read from config file if there is no customized value from user) in queue metadata so that user can know what value it is. * Currently, the v1 API is still accessible though it has been deprecated for a while. And we're going to deprecate v1.1 soon. To keep the backward compatibility, a new config option - "enable_deprecated_api_versions" is added so that operator can totally turn off an API version or still support it by adding the API version to the list of the new config option. * Now before users send messages to subscribers through a queue, the subscribers should be confirmed first. Zaqar only sends messages to the confirmed subscribers. This feature supports "webhook" and "mailto" subscribers with mongoDB or redis backend. The "mailto" part will be done in O cycle. Set "require_confirmation = True" to enable this feature. The default value is "False" now and we will enable it by default after one or two cycles. Deprecation Notes ***************** * Zaqar API v2 has been released for several cycles and it is integrated as the default API version by most of the OpenStack services. So it is time to deprecated v1.1 in favor of v2. Now in Newton cycle, Zaqar API v1.1 is officially deprecated. Bug Fixes ********* * When access the root path of Zaqar service, for example: curl GET http://127.0.0.1:8888/, user will see 401 error. Which will cause some front end proxy (like HAProxy) to complain. Now this issue has been fixed. * Query for all subscriptions on a given queue by taking into account the returned marker, if any. Without this fix, only 10 subscriptions can be extracted from database to send notification. * In IPv6 management network environment, starting Zaqar server will run into 'Address family for hostname not support' error when use WSGI simple server. The root cause is that Python's TCPServer implementation is hard-coded to use IPv4, even in IPv6 environments. Now this issue has been fixed. Changes in zaqar 2.0.0.0rc1..3.0.0 ---------------------------------- 1c77c6e Don't check key format in mongodb 77d7b1a Update UPPER_CONSTRAINTS_FILE for stable/newton 70a72bb Update .gitreview for stable/newton 49d8dd1 Remove small duplication from message storage 3701257 Consolidate constants to a module b745146 Officially deprecate API v1.1 6767092 Fix links in devstack/README.rst f38cecf Disable DEPRECATED API versions by default da130bf Updated from global requirements 4bae6c7 Config logABug feature for zaqar api-ref 9f5b518 Clean imports in code 96f9e8b Remove openstack-doc-tools 3b6932f Use upper constraints for all jobs in tox.ini 6ff9be9 Get ready for os-api-ref sphinx theme change 69c7997 Subscription Confirmation Support-1 96aae7c Fix a minor error in API reference fb775a9 Remove the useless function cf6210d Don't check for project-id if accessing the / route 77d0567 Adding Negative tests cases to Zaqar v2 8ef5fc9 Start accepting **kwargs in service clients c7fa056 Fix errors in API reference for message resource ef50510 modify the home-page info with the developer documentation 8bdc215 Add flavors api ref 930a075 Support `messages` API ref 43765a0 Cleanup i18n marker functions to match Oslo usage 4083856 Updated from global requirements 28cc171 This patch removes extra spaces from files. 9cfa5bc Fix falcon.HTTPConflict initialization getting error 8703ead Add Python 3.5 classifier and venv 8480440 Remove docs,comments,references to Oslo Incubator 37cbe90 Fixes: Typo in comments 51604b4 Add a trust notifier task 0966c7a Ensure queue exists before get/update the claim fabc0b1 Add pools api ref 62e92fc Disable warnerrors=true in setup.cfg 18fc577 Add `Pre-signed queue' api ref 329b2d9 Config: no need to set default=None cc16004 Updated from global requirements 31d6bb6 Fix issue with subscription list in Redis 349839d Remove tempest smoke tests run d0a129f (redis)Fix the 503 error when get/delete messages ef0f8ba Add `Health` api ref 1772e66 Fix the wrong check of X-PROJECT-ID 65bf61d Add subscription api ref 87af703 Cleanup integration tests setup 228f4a6 Clean the auth app after authentication failure 721925c Add Claims for api ref 53b5954 Let v2 tempest base on the base tests 95c9c5d Fix debug log missing issue 35cc77f Remove leftovers on websocket disconnections 7c6eecd Fix config parsing in zaqar-server 18c169b Updated from global requirements d8cde2a Fix post install jobs e439b2e Relocates cli options registration from bootstrap to server 6cb3855 Delete in tree config ref b8a70e4 Add release note for bug 1590173 9a3b019 Add Queues for api ref 5134d30 Add Zaqar installation guide 069bc54 Fix incorrect message when getting non-existing claim 4c2b7e0 Make queues lazy in subscriptions e9dbb19 Add release notes 989f477 Use is_valid_ipv6 from oslo.utils d30ceae Use debug instead of warning for pipeline implement error 61100fe Fix subscription limit per queue e72dc08 Fix fail of getting claim after claim expire(Redis) c1f0ac1 Make the notifier max_workers configurable 897424a Show 'age' field in subscriptions (redis) 3b32dfb Show 'age' field in subscriptions (mongodb) 682eae0 Add reserved metadata check c5640b8 Support Tempest test as gate job 8e5be21 Avoid multi-threading problem with sql backend 48f3e9f Fix fail of claim after some message expire(Redis) 437c26a Always use pip constraints da4009d Init commit to add API ref in tree 4f1bff4 Updated from global requirements c3af668 Define context.roles with base class e007e47 Use run() wrapper instead of connection.execute() 6250cde Improve the log to avoid unexpected ERROR ebc6721 Rename notification/task to notification/tasks c3ebf0d Replace tempest-lib with tempest.lib 437c2fe Refactor zaqar/storage/utils.py to respect DRY 7e87ade remove verbose option in devstack plugin e6d975f Fix falcon confilct in requirements.txt 58e8429 Fix py34 gate job 3f15c0b Move to falcon middleware d744f81 Changed assertTrue(isinstance()) to assertIsInstance. 592304e Trivial: remove openstack/common in flake8 exclude list a066bca Fix cache decorator tests 4a9f313 Imported Translations from Zanata 4f6ac58 Updated from global requirements cae19a2 Fix wrong claim error message fc612ce Default message ttl is needed to change correctly b59c304 Fix auth issue when accessing root path "/" 8237744 Make sure use IPv6 sockets for Zaqar in IPv6 environment e4a5fd4 Show default attributes for queue e78c257 Updated from global requirements 0810e12 Added samples in multiple languages 791cc6f Updated from global requirements 82e02a1 Fix tempest tests list d5286f9 Fix service_available opt registration and extra comma b645feb expires should be checked when using pre-signed url da85355 Validate PUT of reserved queue attributes metadata 400fc6e Validate PUT of reserved queue attributes metadata 89c20fb Warn on upcoming deprecations for v1.0 and v1.1 9671738 Generate oslo.cache options via tox -e genconfig eb3dcee Warn on upcoming deprecations for v1.0 and v1.1 0a191c2 Update reno for stable/mitaka 05efac8 Update .gitreview for stable/mitaka 7f91ccc Use assertEqual/GreaterEqual/LessEqual Diffstat (except docs and test files) ------------------------------------- .gitreview | 1 + api-ref/source/claims.inc | 216 +++++++ api-ref/source/conf.py | 250 ++++++++ api-ref/source/flavors.inc | 205 +++++++ api-ref/source/health.inc | 66 +++ api-ref/source/index.rst | 27 + api-ref/source/messages.inc | 328 +++++++++++ api-ref/source/parameters.yaml | 434 ++++++++++++++ api-ref/source/pools.inc | 205 +++++++ api-ref/source/queues.inc | 311 ++++++++++ api-ref/source/samples/claim_messages_request.json | 4 + .../source/samples/claim_messages_response.json | 12 + api-ref/source/samples/claim_query_response.json | 15 + api-ref/source/samples/claim_update_request.json | 4 + api-ref/source/samples/flavor-create-request.json | 3 + api-ref/source/samples/flavor-list-response.json | 22 + api-ref/source/samples/flavor-show-response.json | 12 + api-ref/source/samples/flavor-update-request.json | 3 + api-ref/source/samples/flavor-update-response.json | 12 + api-ref/source/samples/health-response.json | 48 ++ .../samples/messages-get-byids-response.json | 15 + api-ref/source/samples/messages-get-response.json | 11 + api-ref/source/samples/messages-list-response.json | 32 + api-ref/source/samples/messages-post-request.json | 18 + api-ref/source/samples/messages-post-response.json | 6 + api-ref/source/samples/pool-create-request.json | 8 + api-ref/source/samples/pool-list-response.json | 24 + api-ref/source/samples/pool-show-response.json | 7 + api-ref/source/samples/pool-update-request.json | 8 + api-ref/source/samples/pool-update-response.json | 7 + api-ref/source/samples/queue-create-request.json | 5 + .../source/samples/queue-pre-signed-request.json | 5 + .../source/samples/queue-pre-signed-response.json | 16 + api-ref/source/samples/queue-show-response.json | 5 + api-ref/source/samples/queue-stats-response.json | 7 + api-ref/source/samples/queue-update-request.json | 7 + api-ref/source/samples/queue-update-response.json | 3 + api-ref/source/samples/queues-list-response.json | 22 + .../samples/subscription-create-request-http.json | 5 + .../samples/subscription-create-request-mail.json | 8 + .../samples/subscription-create-response.json | 3 + .../source/samples/subscription-show-response.json | 10 + .../samples/subscription-update-request.json | 7 + .../samples/subscriptions-list-response.json | 26 + api-ref/source/samples/versions-list-response.json | 55 ++ api-ref/source/subscription.inc | 225 +++++++ api-ref/source/versions.inc | 38 ++ devstack/README.rst | 12 +- devstack/gate/gate_hook.sh | 19 +- devstack/plugin.sh | 16 +- devstack/settings | 5 + .../api/zaqar.notification.tasks.webhook.rst | 7 + ...qar.openstack.common.cache._backends.memory.rst | 7 - .../api/zaqar.openstack.common.cache.backends.rst | 7 - .../api/zaqar.openstack.common.cache.cache.rst | 7 - .../api/zaqar.openstack.common.fileutils.rst | 7 - .../api/zaqar.openstack.common.gettextutils.rst | 7 - .../api/zaqar.openstack.common.lockutils.rst | 7 - .../api/zaqar.openstack.common.timeutils.rst | 7 - .../zaqar-config-ref/autogen/zaqar.flagmappings | 86 --- .../zaqar-config-ref/autogen/zaqar.headers | 10 - .../zaqar-config-ref/common/tables/zaqar-api.xml | 38 -- .../common/tables/zaqar-auth_token.xml | 145 ----- .../common/tables/zaqar-authentication.xml | 26 - .../common/tables/zaqar-drivers.xml | 30 - .../common/tables/zaqar-logging.xml | 102 ---- .../common/tables/zaqar-mongodb.xml | 70 --- .../common/tables/zaqar-pooling.xml | 26 - .../common/tables/zaqar-storage.xml | 34 -- .../common/tables/zaqar-transport.xml | 66 --- .../zaqar-config-ref/common/tables/zaqar-wsgi.xml | 30 - .../zaqar-config-ref/os-zaqar-configRef.xml | 250 -------- etc/oslo-config-generator/zaqar.conf | 2 + etc/policy.json.sample | 1 + examples/websocket.html | 309 ---------- install-guide/source/conf.py | 301 ++++++++++ install-guide/source/get_started.rst | 65 +++ install-guide/source/index.rst | 22 + install-guide/source/install-rdo.rst | 545 +++++++++++++++++ install-guide/source/install.rst | 33 ++ install-guide/source/next-steps.rst | 9 + install-guide/source/verify.rst | 36 ++ ...dd-a-notifier-using-trust-271d9cd1d2b4cdeb.yaml | 10 + .../notes/deprecate-v11-976cccc1b56a28e7.yaml | 6 + ..._auth_issue_for_root_path-b15e1c4e92e4e8b1.yaml | 7 + .../fix_subscription_limit-c3cdc9385825285a.yaml | 5 + ...y-queues-in-subscriptions-6bade4a1b8eca3e5.yaml | 8 + ...ault_attributes_for_queue-3d87333752484c87.yaml | 8 + ...rnoff-deprecated-versions-44656aeb8ebb8881.yaml | 7 + .../notes/user_ipv6_sockets-1e1b436de6b81ae3.yaml | 7 + ...subscription_confirmation-883cb7f325885ef0.yaml | 9 + releasenotes/source/conf.py | 12 +- releasenotes/source/index.rst | 3 +- releasenotes/source/mitaka.rst | 6 + requirements.txt | 20 +- .../receive_message/JsonDecoder.java | 43 ++ .../receive_message/SampleZaqarEndpoint.java | 57 ++ .../send_message/SampleZaqarEndpoint.java | 45 ++ samples/javascript/receive_message/zaqar_sample.js | 32 + samples/javascript/send_message/zaqar_sample.js | 25 + samples/javascript/websocket.html | 309 ++++++++++ .../jaxrs/receive_message/SampleZaqarServlet.java | 55 ++ samples/jaxrs/send_message/SampleZaqarServlet.java | 52 ++ samples/nodejs/receive_message/zaqar_sample.js | 34 ++ samples/nodejs/send_message/zaqar_sample.js | 27 + .../receive_message/zaqar_sample.py | 30 + .../send_message/zaqar_sample.py | 27 + samples/zaqar/subscriber_service_sample.py | 79 +++ setup.cfg | 20 +- test-requirements.txt | 15 +- tox.ini | 27 +- zaqar/api/handler.py | 27 +- zaqar/api/v1/request.py | 61 +- zaqar/api/v1/response.py | 15 +- zaqar/api/v1_1/request.py | 41 +- zaqar/api/v1_1/response.py | 19 +- zaqar/api/v2/endpoints.py | 23 +- zaqar/api/v2/request.py | 17 +- zaqar/api/v2/response.py | 19 +- zaqar/bootstrap.py | 20 +- zaqar/cmd/gc.py | 4 +- zaqar/cmd/server.py | 20 +- zaqar/common/api/request.py | 15 +- zaqar/common/auth.py | 95 +++ zaqar/common/configs.py | 11 +- zaqar/common/consts.py | 115 ++++ zaqar/common/decorators.py | 35 ++ zaqar/common/pipeline.py | 2 +- zaqar/common/transport/wsgi/helpers.py | 17 +- zaqar/common/urls.py | 14 +- zaqar/context.py | 8 +- zaqar/locale/es/LC_MESSAGES/zaqar.po | 22 +- zaqar/locale/fr/LC_MESSAGES/zaqar-log-error.po | 15 +- zaqar/locale/zaqar-log-error.pot | 54 -- zaqar/locale/zaqar.pot | 604 ------------------- zaqar/notification/notifier.py | 113 +++- zaqar/notification/task/__init__.py | 0 zaqar/notification/task/mailto.py | 53 -- zaqar/notification/task/webhook.py | 38 -- zaqar/notification/tasks/__init__.py | 0 zaqar/notification/tasks/mailto.py | 56 ++ zaqar/notification/tasks/trust.py | 63 ++ zaqar/notification/tasks/webhook.py | 49 ++ zaqar/storage/configuration.py | 48 ++ zaqar/storage/mongodb/claims.py | 2 +- zaqar/storage/mongodb/messages.py | 19 +- zaqar/storage/mongodb/subscriptions.py | 59 +- zaqar/storage/pipeline.py | 6 +- zaqar/storage/pooling.py | 11 + zaqar/storage/redis/claims.py | 15 +- zaqar/storage/redis/messages.py | 11 +- zaqar/storage/redis/models.py | 10 +- zaqar/storage/redis/scripts/claim_messages.lua | 45 +- zaqar/storage/redis/subscriptions.py | 22 +- zaqar/storage/redis/utils.py | 2 +- zaqar/storage/sqlalchemy/catalogue.py | 12 +- zaqar/storage/sqlalchemy/driver.py | 8 +- zaqar/storage/sqlalchemy/flavors.py | 14 +- zaqar/storage/sqlalchemy/pools.py | 20 +- zaqar/storage/utils.py | 7 +- .../api_schema/response/v2/queues.py | 4 +- .../services/messaging/json/messaging_client.py | 64 +- .../unit/transport/websocket/v2/test_claims.py | 51 +- .../unit/transport/websocket/v2/test_messages.py | 70 +-- .../transport/websocket/v2/test_queue_lifecycle.py | 47 +- .../transport/websocket/v2/test_subscriptions.py | 126 +++- .../unit/transport/wsgi/v1/test_validation.py | 16 + .../unit/transport/wsgi/v1_1/test_validation.py | 39 ++ .../transport/wsgi/v2_0/test_queue_lifecycle.py | 67 ++- .../unit/transport/wsgi/v2_0/test_subscriptions.py | 98 +++- zaqar/transport/auth.py | 5 + zaqar/transport/validation.py | 18 +- zaqar/transport/websocket/driver.py | 2 +- zaqar/transport/websocket/factory.py | 3 + zaqar/transport/websocket/protocol.py | 24 +- zaqar/transport/wsgi/app.py | 4 +- zaqar/transport/wsgi/driver.py | 46 +- zaqar/transport/wsgi/v1_0/__init__.py | 9 + zaqar/transport/wsgi/v1_0/metadata.py | 14 +- zaqar/transport/wsgi/v1_1/__init__.py | 17 +- zaqar/transport/wsgi/v1_1/queues.py | 16 +- zaqar/transport/wsgi/v2_0/__init__.py | 11 +- zaqar/transport/wsgi/v2_0/pools.py | 2 +- zaqar/transport/wsgi/v2_0/queues.py | 50 +- zaqar/transport/wsgi/v2_0/subscriptions.py | 103 +++- zaqar/transport/wsgi/v2_0/urls.py | 4 +- 254 files changed, 8872 insertions(+), 2902 deletions(-) Requirements updates -------------------- diff --git a/requirements.txt b/requirements.txt index 0f828f5..5b913db 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6 +6 @@ pbr>=1.6 # Apache-2.0 -Babel>=1.3 # BSD +Babel>=2.3.4 # BSD @@ -9,2 +9,2 @@ jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT -iso8601>=0.1.9 # MIT -keystonemiddleware!=4.1.0,>=4.0.0 # Apache-2.0 +iso8601>=0.1.11 # MIT +keystonemiddleware!=4.1.0,!=4.5.0,>=4.0.0 # Apache-2.0 @@ -14 +14 @@ WebOb>=1.2.3 # MIT -stevedore>=1.5.0 # Apache-2.0 +stevedore>=1.16.0 # Apache-2.0 @@ -17,2 +17,2 @@ oslo.cache>=1.5.0 # Apache-2.0 -oslo.config>=3.7.0 # Apache-2.0 -oslo.context>=0.2.0 # Apache-2.0 +oslo.config>=3.14.0 # Apache-2.0 +oslo.context>=2.9.0 # Apache-2.0 @@ -22,2 +22,2 @@ oslo.serialization>=1.10.0 # Apache-2.0 -oslo.utils>=3.5.0 # Apache-2.0 -oslo.policy>=0.5.0 # Apache-2.0 +oslo.utils>=3.16.0 # Apache-2.0 +oslo.policy>=1.9.0 # Apache-2.0 @@ -28,2 +28,2 @@ autobahn>=0.10.1 # MIT License -requests!=2.9.0,>=2.8.1 # Apache-2.0 -futurist>=0.11.0 # Apache-2.0 +requests>=2.10.0 # Apache-2.0 +futurist!=0.15.0,>=0.11.0 # Apache-2.0 diff --git a/test-requirements.txt b/test-requirements.txt index a2fb1cc..93fe258 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -8 +8 @@ hacking<0.11,>=0.10.0 -mock>=1.2 # BSD +mock>=2.0 # BSD @@ -18 +18 @@ ddt>=1.0.1 # MIT -fixtures>=1.3.1 # Apache-2.0/BSD +fixtures>=3.0.0 # Apache-2.0/BSD @@ -24 +24 @@ testtools>=1.4.0 # MIT -sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2 # BSD +sphinx!=1.3b1,<1.3,>=1.2.1 # BSD @@ -26 +26 @@ oslosphinx!=3.4.0,>=2.5.0 # Apache-2.0 -openstack-doc-tools>=0.23 # Apache-2.0 +openstackdocstheme>=1.5.0 # Apache-2.0 @@ -28 +28,2 @@ oslotest>=1.10.0 # Apache-2.0 -reno>=0.1.1 # Apache2 +reno>=1.8.0 # Apache2 +os-api-ref>=1.0.0 # Apache-2.0 @@ -30,2 +31,2 @@ reno>=0.1.1 # Apache2 -# Tempest Plugin -tempest-lib>=0.14.0 # Apache-2.0 +# Tempest +tempest>=12.1.0 # Apache-2.0