[openstack-dev] taskflow 0.6.0 released
Joshua Harlow
harlowja at outlook.com
Fri Dec 19 02:47:19 UTC 2014
The Oslo team is pleased to announce the release of:
taskflow 0.6.0: Taskflow structured state management library.
For more details, please see the git log history below and:
http://launchpad.net/taskflow/+milestone/0.6.0
Please report issues through launchpad:
http://bugs.launchpad.net/taskflow/
Noteable changes
----------------
* Dialects are now correctly supported in the persistence
backends (mainly the ability to use sqlalchemy dialects; for
example 'mysql+pymysql').
* Task *local* notification mechanism (autobind, trigger,
bind, unbind, listeners_iter methods) have been replaced with direct
usage of the notification type. A transition will be needed (if
you are currently using these functions) to the newer property and
equivalent methods/functions that the notification type
provides.
To help perform this transition the *nearly* equivalent method
mapping is/are the following:
=================== ============================
Old method New property/method/function
=================== ============================
task.autobind notifier.register_deregister
task.bind task.notifier.register
task.unbind task.notifier.deregister
task.trigger task.notifier.notify
task.listeners_iter task.notifier.listeners_iter
=================== ============================
* The 'EngineBase', 'ListenerBase' have been renamed to 'Engine'
and 'Listener' (removal of the 'Base' post-fix); these existing
classes (with the 'Base' post-fix) are marked as deprecated and
will be subject to removal in a future version.
* The existing listeners in taskflow/listeners now take an new
constructor parameter; 'retry_listen_for' which specifies what
notifications to listen for/recieve (by default ANY or '*'),
existing derivatives of this class have been updated to pass this
along (subclasses that have been created can omit it if they
choose to).
* A new 'blather()/BLATHER' logging level has been added and used to
avoid the low-level scope and runtime information that is being emitted
from taskflow during compilation time and at engine runtime. This
should reduce the amount of noise that is generated (and really only
useful by taskflow developers).
* This log level is currently set at number 5 (all log levels have
equivalent numbers) which appears to be a common pattern shared
by the multiprocessing logger and kazoo (the library) which use it
for a similar purpose.
* The engine helper run/load 'engine_conf' dictionary keyword argument
has been marked as deprecated and should now be replaced with usage
of the 'engine=<URI>' format (where the URI contains the engine type
to use and any specific engine options in the URIs parameters). The
'engine_conf' keyword argument will be subject to removal in a
future version.
* Tasks can now be copied via a copy() method (this will be useful in an
upcoming ability for an engine to run tasks using the multiprocess
library; therefore creating a nice median between thread based engines
and remote work based engines).
* A claims listener that can be used to connect jobboards to engines as
well as a dynamic/useful logging listener that can adjust the logging
levels based on notifications received.
* The engine 'task_notifier' has been renamed to its more general
name of 'atom_notifier' and 'task_notifier' has been marked as
deprecated and it will be subject to removal in a future version.
* The greenthread executor (previously in a utility module) has been
moved to the types/futures.py module where it should now be acceptable
to use this as a first-class type (previously it is/was not accepted
to use internal utility classes/modules externally).
* A new types folder + useful helper modules aid taskflow (and likely
can aid other projects as well); some of these modules are splitting
off into there own projects for more general usage (this is
happening on a as needed/ongoing basis).
* Storage methods 'ensure_retry' and 'ensure_task' have been replaced
with the type agnostic 'ensure_atom' (which is now the internally
supported and used API for ensuring the storage unit has allocated
the details about a given atom, retry or task...).
* New and improved symbol scoping/finding support!
* New and improved documentation and examples!
Changes in /homes/harlowja/dev/os/taskflow 0.5.0..0.6.0
-------------------------------------------------------
NOTE: Skipping requirement commits...
4e514f4 Move over to using oslo.utils [reflection, uuidutils]
6520b9c Add a basic map/reduce example to show how this can be done
cafa3b2 Add a parallel table mutation example
1b06183 Add a 'can_be_registered' method that checks before notifying
97b4e18 Base task executor should provide 'wait_for_any'
e9ecdc7 Replace autobind with a notifier module helper function
aa8d55d Cleanup some doc warnings/bad/broken links
1f4dd72 Use the notifier type in the task class/module directly
cdfd8ec Use a tiny clamp helper to clamp the 'on_progress' value
624d966 Retain the existence of a 'EngineBase' until 0.7 or later
f5060ff Remove the base postfix from the internal task executor
b4e4e21 Remove usage of listener base postfix
a440ec4 Add a moved_inheritable_class deprecation helper
1c7d242 Avoid holding the lock while scanning for existing jobs
79ff9e7 Remove the base postfix for engine abstract base class
880f7d2 Avoid popping while another entity is iterating
fda6fde Use explict 'attr_dict' when adding provider->consumer edge
eaf4995 Properly handle and skip empty intermediary flows
f1457a0 Ensure message gets processed correctly
b275c51 Just assign a empty collection instead of copy/clear
f333e1b Remove rtype from task clone() doc
14431bc Add and use a new simple helper logging module
a113368 Have the sphinx copyright date be dynamic
2f78ecf Add appropriate links into README.rst
4eb0ca2 Use condition variables using 'with'
50b866c Use an appropriate ``extract_traceback`` limit
c4d3279 Allow all deprecation helpers to take a stacklevel
cd664bd Correctly identify stack level in ``_extract_engine``
5f0b514 Stop returning atoms from execute/revert methods
dc4262e Have tasks be able to provide copy() methods
e978eca Allow stopwatches to be restarted
af62f4c Ensure that failures can be pickled
e168f44 Rework pieces of the task callback capability
dc39351 Just use 4 spaces for classifier indents
4707ac7 Move atom action handlers to there own subfolder/submodule
2033d01 Workflow documentation is now in infra-manual
2f03736 Ensure frozen attribute is set in fsm clones/copies
8150553 Fix split on "+" for connection strings that specify dialects
b8e975e Update listeners to ensure they correctly handle all atoms
cf45a70 Allow for the notifier to provide a 'details_filter'
c698842 Be explicit about publish keyword arguments
6a6aa79 Some package additions and adjustments to the env_builder.sh
a692138 Cache immutable visible scopes in the runtime component
14ecaa4 Raise value errors instead of asserts
1de8bbd Add a claims listener that connects job claims to engines
1e8fabd Split the scheduler into sub-schedulers
35fcd90 Use a module level constant to provide the DEFAULT_LISTEN_FOR
178f279 Move the _pformat() method to be a classmethod
9675964 Add link to issue 17911
e07fb21 Avoid deepcopying exception values
95e94f7 Include documentation of the utility modules
265181f Use a metaclass to dynamically add testcases to example runner
cf85dd0 Remove default setting of 'mysql_traditional_mode'
bb8ea56 Move scheduler and completer classes to there own modules
2832d6e Ensure that the zookeeper backend creates missing atoms
148723b Use the deprecation utility module instead of warnings
49d7a51 Tweaks to setup.cfg
17fb4b4 Add a jobboard high level architecture diagram
487cc51 Mark 'task_notifier' as renamed to 'atom_notifier'
2f7e582 Revert wrapt usage until further notice
2a7ca47 Add a history retry object, makes retry histories easier to use
7d199e0 Format failures via a static method
c543dc2 When creating daemon threads use the bundled threading_utils
613af61 Ensure failure types contain only immutable items
b24656c Mark 'task_notifier' as renamed to 'atom_notifier'
f3e4bb0 Use wrapt to provide the deprecated class proxy
c8b0f25 Reduce the worker-engine joint testing time
543b6a0 Link bug in requirements so people understand why pbr is listed
34b358a Use standard threading locks in the cache types
edb9212 Handle the case where '_exc_type_names' is empty
5671868 Add pbr to installation requirements
3c9871d Remove direct usage of the deprecated failure location
1f12ab3 Fix the example 'default_provides'
ac8eefd Use constants for retry automatically provided kwargs
58f27fc Remove direct usage of the deprecated notifier location
7fe6bf0 Remove attrdict and just use existing types
ca101d1 Use the mock that finds a working implementation
b014fc7 Add a futures type that can unify our future functionality
ac77b4d Bump the deprecation version number
7ca6313 Use and verify event and latch wait() return using timeouts
d433a53 Deprecate `engine_conf` and prefer `engine` instead
3a8a78e Use constants for link metadata keys
d638c8f Bump up the sqlalchemy version for py26
bf84288 Hoist the notifier to its own module
f2ea4f1 Move failure to its own type specific module
a15e07a Use constants for revert automatically provided kwargs
8e66177 Improve some of the task docstrings
bcae66b We can now use PyMySQL in py3.x tests
c90e360 Add the database schema to the sqlalchemy docs
52494e7 Change messaging from handler connection timeouts -> operation
timeouts
b86b7e1 Switch to a custom NotImplementedError derivative
94b4b60 Allow the worker banner to be written to an arbitrary location
8d14318 Update engine class names to better reflect there usage
95b30d6 Refactor parts of the job lock/job condition zookeeper usage
cf1e468 Add a more dynamic/useful logging listener
be254ea Use timeutils functions instead of misc.wallclock
eedc335 Expose only `ensure_atom` from storage
dc688c1 Increase robustness of WBE message and request processing
7640b09 Bring in a newer optional eventlet
9537f52 Document more function/class/method params
7fe2f51 Remove no longer needed r/w lock interface base class
8bbc2fd Better handle the tree freeze method
c5c2211 Ensure state machine can be frozen
97e6bb1 Link a few of the classes to implemented features/bugs in python
6bbf85b Add a timing listener that also prints the results
c5aa2f9 Remove useless __exit__ return
26793dc Add a state machine copy() method
d98f23d Add a couple of scope shadowing test cases
d6ef687 Relax the graph flow symbol constraints
76641d8 Relax the unordered flow symbol constraints
2339bac Relax the linear flow symbol constraints
fa077c9 Revamp the symbol lookup mechanism
e68d72f Be smarter about required flow symbols
296e660 Have the dispatch_job function return a future
8dc6e4f Add a sample script that can be used to build a test environment
be4fac3 Extract the state changes from the ensure storage method
Diffstat (except docs and test files)
-------------------------------------
CONTRIBUTING.rst | 4 +-
README.rst | 5 +-
openstack-common.conf | 1 -
requirements-py2.txt | 7 +-
requirements-py3.txt | 7 +-
setup.cfg | 14 +-
taskflow/atom.py | 26 +-
taskflow/conductors/base.py | 26 +-
taskflow/conductors/single_threaded.py | 26 +-
taskflow/engines/action_engine/actions/retry.py | 124 +
taskflow/engines/action_engine/actions/task.py | 145 +
taskflow/engines/action_engine/compiler.py | 456 +-
taskflow/engines/action_engine/completer.py | 114 +
taskflow/engines/action_engine/engine.py | 53 +-
taskflow/engines/action_engine/executor.py | 84 +-
taskflow/engines/action_engine/retry_action.py | 86 -
taskflow/engines/action_engine/runner.py | 15 +-
taskflow/engines/action_engine/runtime.py | 228 +-
taskflow/engines/action_engine/scheduler.py | 114 +
taskflow/engines/action_engine/scopes.py | 113 +
taskflow/engines/action_engine/task_action.py | 116 -
taskflow/engines/base.py | 49 +-
taskflow/engines/helpers.py | 201 +-
taskflow/engines/worker_based/cache.py | 4 +-
taskflow/engines/worker_based/dispatcher.py | 6 +-
taskflow/engines/worker_based/endpoint.py | 15 +-
taskflow/engines/worker_based/engine.py | 30 +-
taskflow/engines/worker_based/executor.py | 52 +-
taskflow/engines/worker_based/protocol.py | 51 +-
taskflow/engines/worker_based/proxy.py | 55 +-
taskflow/engines/worker_based/server.py | 123 +-
taskflow/engines/worker_based/worker.py | 17 +-
taskflow/examples/calculate_in_parallel.py | 2 +-
taskflow/examples/create_parallel_volume.py | 13 +-
taskflow/examples/delayed_return.py | 6 +-
taskflow/examples/fake_billing.py | 12 +-
taskflow/examples/graph_flow.py | 4 +-
.../examples/jobboard_produce_consume_colors.py | 7 +-
taskflow/examples/parallel_table_multiply.py | 129 +
taskflow/examples/persistence_example.py | 13 +-
taskflow/examples/resume_vm_boot.py | 22 +-
taskflow/examples/resume_volume_create.py | 6 +-
taskflow/examples/run_by_iter.py | 13 +-
taskflow/examples/run_by_iter_enumerate.py | 8 +-
taskflow/examples/simple_linear_pass.py | 2 +-
taskflow/examples/simple_map_reduce.py | 115 +
taskflow/examples/timing_listener.py | 59 +
taskflow/examples/wbe_mandelbrot.py | 5 +-
taskflow/examples/wbe_simple_linear.py | 19 +-
taskflow/examples/wrapped_exception.py | 20 +-
taskflow/exceptions.py | 91 +-
taskflow/flow.py | 48 +-
taskflow/jobs/backends/__init__.py | 8 +-
taskflow/jobs/backends/impl_zookeeper.py | 90 +-
taskflow/jobs/job.py | 3 +-
taskflow/jobs/jobboard.py | 4 +-
taskflow/listeners/base.py | 199 +-
taskflow/listeners/claims.py | 100 +
taskflow/listeners/logging.py | 175 +-
taskflow/listeners/printing.py | 16 +-
taskflow/listeners/timing.py | 55 +-
taskflow/logging.py | 92 +
taskflow/openstack/common/__init__.py | 17 -
taskflow/openstack/common/uuidutils.py | 37 -
taskflow/patterns/graph_flow.py | 177 +-
taskflow/patterns/linear_flow.py | 50 +-
taskflow/patterns/unordered_flow.py | 61 +-
taskflow/persistence/backends/__init__.py | 13 +-
taskflow/persistence/backends/impl_dir.py | 2 +-
taskflow/persistence/backends/impl_memory.py | 3 +-
taskflow/persistence/backends/impl_sqlalchemy.py | 11 +-
taskflow/persistence/backends/impl_zookeeper.py | 8 +-
taskflow/persistence/backends/sqlalchemy/models.py | 2 +-
taskflow/persistence/logbook.py | 21 +-
taskflow/retry.py | 141 +-
taskflow/storage.py | 343 +-
taskflow/task.py | 205 +-
taskflow/test.py | 78 +
taskflow/types/cache.py | 25 +-
taskflow/types/failure.py | 342 +
taskflow/types/fsm.py | 42 +-
taskflow/types/futures.py | 206 +
taskflow/types/graph.py | 22 +
taskflow/types/latch.py | 17 +-
taskflow/types/notifier.py | 278 +
taskflow/types/timing.py | 41 +-
taskflow/types/tree.py | 24 +-
taskflow/utils/async_utils.py | 82 +-
taskflow/utils/deprecation.py | 257 +
taskflow/utils/eventlet_utils.py | 192 -
taskflow/utils/kazoo_utils.py | 10 +-
taskflow/utils/lock_utils.py | 67 +-
taskflow/utils/misc.py | 627 +-
taskflow/utils/persistence_utils.py | 4 +-
taskflow/utils/reflection.py | 251 -
taskflow/utils/threading_utils.py | 20 +
test-requirements.txt | 13 +-
tools/env_builder.sh | 126 +
tools/schema_generator.py | 83 +
tox.ini | 8 +-
158 files changed, 7662 insertions(+), 12475 deletions(-)
Requirements updates
--------------------
diff --git a/requirements-py2.txt b/requirements-py2.txt
index 6134877..e142007 100644
--- a/requirements-py2.txt
+++ b/requirements-py2.txt
@@ -4,0 +5,3 @@
+# See: https://bugs.launchpad.net/pbr/+bug/1384919 for why this is here...
+pbr>=0.6,!=0.7,<1.0
+
@@ -17 +20 @@ networkx>=1.8
-stevedore>=1.0.0 # Apache-2.0
+stevedore>=1.1.0 # Apache-2.0
@@ -26 +29 @@ jsonschema>=2.0.0,<3.0.0
-oslo.utils>=1.0.0 # Apache-2.0
+oslo.utils>=1.1.0 # Apache-2.0
diff --git a/requirements-py3.txt b/requirements-py3.txt
index ea30582..d827a17 100644
--- a/requirements-py3.txt
+++ b/requirements-py3.txt
@@ -4,0 +5,3 @@
+# See: https://bugs.launchpad.net/pbr/+bug/1384919 for why this is here...
+pbr>=0.6,!=0.7,<1.0
+
@@ -14 +17 @@ networkx>=1.8
-stevedore>=1.0.0 # Apache-2.0
+stevedore>=1.1.0 # Apache-2.0
@@ -20 +23 @@ jsonschema>=2.0.0,<3.0.0
-oslo.utils>=1.0.0 # Apache-2.0
+oslo.utils>=1.1.0 # Apache-2.0
diff --git a/test-requirements.txt b/test-requirements.txt
index 19856bb..96ab944 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -6 +6 @@ hacking>=0.9.2,<0.10
-oslotest>=1.1.0 # Apache-2.0
+oslotest>=1.2.0 # Apache-2.0
@@ -8 +8 @@ mock>=1.0
-testtools>=0.9.34
+testtools>=0.9.36,!=1.2.0
@@ -22 +22,6 @@ kazoo>=1.3.1
-alembic>=0.6.4
+#
+# Explict mysql drivers are also not listed here so that we can test
against
+# PyMySQL or MySQL-python depending on the python version the tests are
being
+# ran in (MySQL-python is currently preferred for 2.x environments, since
+# it has been used in openstack for the longest).
+alembic>=0.7.1
@@ -26 +31 @@ psycopg2
-sphinx>=1.1.2,!=1.2.0,<1.3
+sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
More information about the OpenStack-dev
mailing list