any idiomatic testtools / stestr patterns for asyncio?
hey gang -
I'm adding asyncio support to oslo.db. Fairly major block hit, which I can work around, is that testtools / stestr do not seem to have any provision for asyncio present.
An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase):
import unittest from oslotest import base
class MyTestSuite(unittest.IsolatedAsyncioTestCase, base.BaseTestCase): # ...
but that didn't work because the degree to which testtools modifies the run() method of unittest.TestCase prevents IsolatedAsyncioTestCase from running awaitable test cases.
Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools patterns):
class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ...
this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object.
final workaround, patch one on, here's my base:
class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result)
now I can run asyncio tests. But only by ditching testtools entirely.
What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?
On Mon, Jul 1, 2024, at 11:32 AM, Mike Bayer wrote:
hey gang -
I'm adding asyncio support to oslo.db. Fairly major block hit, which I can work around, is that testtools / stestr do not seem to have any provision for asyncio present.
An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase):
import unittest from oslotest import base
class MyTestSuite(unittest.IsolatedAsyncioTestCase, base.BaseTestCase): # ...
but that didn't work because the degree to which testtools modifies the run() method of unittest.TestCase prevents IsolatedAsyncioTestCase from running awaitable test cases.
Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools patterns):
class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ...
this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object.
final workaround, patch one on, here's my base:
class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result)
now I can run asyncio tests. But only by ditching testtools entirely.
What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?
I think the dependency on testtools was far more important back when we were trying to support python2.6, python2.7 and python3.something. In particular support for addCleanup() is/was important to ensure reliable cleanups of external resources. I think there were also problems with setUp failures maybe not failing test cases in old python unittest (don't quote me on that I don't remember all the details).
At this point I suspect the need to only support relatively modern python particularly when we talk about adding asyncio support to stuff means we may get away with stdlib unittest over testtools. Testtools was primarily there to bridge some behavior gaps and ensure consistent behavior between python2 and python3. That is no longer a concern.
On Mon, Jul 1, 2024, at 2:53 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 11:32 AM, Mike Bayer wrote:
hey gang -
I'm adding asyncio support to oslo.db. Fairly major block hit, which I can work around, is that testtools / stestr do not seem to have any provision for asyncio present.
An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase):
import unittest from oslotest import base
class MyTestSuite(unittest.IsolatedAsyncioTestCase, base.BaseTestCase): # ...
but that didn't work because the degree to which testtools modifies the run() method of unittest.TestCase prevents IsolatedAsyncioTestCase from running awaitable test cases.
Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools patterns):
class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ...
this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object.
final workaround, patch one on, here's my base:
class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result)
now I can run asyncio tests. But only by ditching testtools entirely.
What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?
I think the dependency on testtools was far more important back when we were trying to support python2.6, python2.7 and python3.something. In particular support for addCleanup() is/was important to ensure reliable cleanups of external resources. I think there were also problems with setUp failures maybe not failing test cases in old python unittest (don't quote me on that I don't remember all the details).
At this point I suspect the need to only support relatively modern python particularly when we talk about adding asyncio support to stuff means we may get away with stdlib unittest over testtools. Testtools was primarily there to bridge some behavior gaps and ensure consistent behavior between python2 and python3. That is no longer a concern.
OK. So...what about the stestr part ? That itself seems to be slightly incompatible with straight unittest at this point. Also what's the role of oslotest ? Noting I'm working on an actual oslo project.
On Mon, Jul 1, 2024, at 1:55 PM, Mike Bayer wrote:
On Mon, Jul 1, 2024, at 2:53 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 11:32 AM, Mike Bayer wrote:
hey gang -
I'm adding asyncio support to oslo.db. Fairly major block hit, which I can work around, is that testtools / stestr do not seem to have any provision for asyncio present.
An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase):
import unittest from oslotest import base
class MyTestSuite(unittest.IsolatedAsyncioTestCase, base.BaseTestCase): # ...
but that didn't work because the degree to which testtools modifies the run() method of unittest.TestCase prevents IsolatedAsyncioTestCase from running awaitable test cases.
Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools patterns):
class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ...
this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object.
final workaround, patch one on, here's my base:
class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result)
now I can run asyncio tests. But only by ditching testtools entirely.
What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?
I think the dependency on testtools was far more important back when we were trying to support python2.6, python2.7 and python3.something. In particular support for addCleanup() is/was important to ensure reliable cleanups of external resources. I think there were also problems with setUp failures maybe not failing test cases in old python unittest (don't quote me on that I don't remember all the details).
At this point I suspect the need to only support relatively modern python particularly when we talk about adding asyncio support to stuff means we may get away with stdlib unittest over testtools. Testtools was primarily there to bridge some behavior gaps and ensure consistent behavior between python2 and python3. That is no longer a concern.
OK. So...what about the stestr part ? That itself seems to be slightly incompatible with straight unittest at this point. Also what's the role of oslotest ? Noting I'm working on an actual oslo project.
The stestr issue appears to originate in python subunit [0]. Reading this pytest issue [1] addDuration is brand new in python 3.12. I suspect that subunit just needs to be updated to support the new API similar to how pytest was updated. But also it appears that this should only be a warning so shouldn't prevent things from working?
[0] https://github.com/testing-cabal/subunit [1] https://github.com/pytest-dev/pytest/issues/10875
On Mon, Jul 1, 2024, at 6:06 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 1:55 PM, Mike Bayer wrote:
On Mon, Jul 1, 2024, at 2:53 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 11:32 AM, Mike Bayer wrote:
hey gang -
I'm adding asyncio support to oslo.db. Fairly major block hit, which I can work around, is that testtools / stestr do not seem to have any provision for asyncio present.
An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase):
import unittest from oslotest import base
class MyTestSuite(unittest.IsolatedAsyncioTestCase, base.BaseTestCase): # ...
but that didn't work because the degree to which testtools modifies the run() method of unittest.TestCase prevents IsolatedAsyncioTestCase from running awaitable test cases.
Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools patterns):
class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ...
this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object.
final workaround, patch one on, here's my base:
class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result)
now I can run asyncio tests. But only by ditching testtools entirely.
What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?
I think the dependency on testtools was far more important back when we were trying to support python2.6, python2.7 and python3.something. In particular support for addCleanup() is/was important to ensure reliable cleanups of external resources. I think there were also problems with setUp failures maybe not failing test cases in old python unittest (don't quote me on that I don't remember all the details).
At this point I suspect the need to only support relatively modern python particularly when we talk about adding asyncio support to stuff means we may get away with stdlib unittest over testtools. Testtools was primarily there to bridge some behavior gaps and ensure consistent behavior between python2 and python3. That is no longer a concern.
OK. So...what about the stestr part ? That itself seems to be slightly incompatible with straight unittest at this point. Also what's the role of oslotest ? Noting I'm working on an actual oslo project.
The stestr issue appears to originate in python subunit [0]. Reading this pytest issue [1] addDuration is brand new in python 3.12. I suspect that subunit just needs to be updated to support the new API similar to how pytest was updated. But also it appears that this should only be a warning so shouldn't prevent things from working?
[0] https://github.com/testing-cabal/subunit [1] https://github.com/pytest-dev/pytest/issues/10875
OK that's fine, yes it's only a warning, was just not sure if this was an indication of a larger incompatibility with straight unittest that doesnt use testtools.TestCase. that was basically my question
On 7/2/24 1:11 AM, Mike Bayer wrote:
On Mon, Jul 1, 2024, at 6:06 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 1:55 PM, Mike Bayer wrote:
On Mon, Jul 1, 2024, at 2:53 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 11:32 AM, Mike Bayer wrote:
hey gang -
I'm adding asyncio support to oslo.db. Fairly major block hit, which I can work around, is that testtools / stestr do not seem to have any provision for asyncio present.
An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase):
import unittest from oslotest import base
class MyTestSuite(unittest.IsolatedAsyncioTestCase, base.BaseTestCase): # ...
but that didn't work because the degree to which testtools modifies the run() method of unittest.TestCase prevents IsolatedAsyncioTestCase from running awaitable test cases.
Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools patterns):
class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ...
this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object.
final workaround, patch one on, here's my base:
class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result)
now I can run asyncio tests. But only by ditching testtools entirely.
What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?
I think the dependency on testtools was far more important back when we were trying to support python2.6, python2.7 and python3.something. In particular support for addCleanup() is/was important to ensure reliable cleanups of external resources. I think there were also problems with setUp failures maybe not failing test cases in old python unittest (don't quote me on that I don't remember all the details).
At this point I suspect the need to only support relatively modern python particularly when we talk about adding asyncio support to stuff means we may get away with stdlib unittest over testtools. Testtools was primarily there to bridge some behavior gaps and ensure consistent behavior between python2 and python3. That is no longer a concern.
OK. So...what about the stestr part ? That itself seems to be slightly incompatible with straight unittest at this point. Also what's the role of oslotest ? Noting I'm working on an actual oslo project.
The stestr issue appears to originate in python subunit [0]. Reading this pytest issue [1] addDuration is brand new in python 3.12. I suspect that subunit just needs to be updated to support the new API similar to how pytest was updated. But also it appears that this should only be a warning so shouldn't prevent things from working?
[0] https://github.com/testing-cabal/subunit [1] https://github.com/pytest-dev/pytest/issues/10875
OK that's fine, yes it's only a warning, was just not sure if this was an indication of a larger incompatibility with straight unittest that doesnt use testtools.TestCase. that was basically my question
FWIW we have a few projects (e.g. metalsmith) that use stestr but neither testtools nor oslotest.
Dmitry
Thanks Mike for raising that point. I think I'll add a section about your observations in the goal proposal [1] to warn about this point.
[1] https://review.opendev.org/c/openstack/governance/+/902585
Le mar. 2 juil. 2024 à 12:13, Dmitry Tantsur dtantsur@protonmail.com a écrit :
On 7/2/24 1:11 AM, Mike Bayer wrote:
On Mon, Jul 1, 2024, at 6:06 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 1:55 PM, Mike Bayer wrote:
On Mon, Jul 1, 2024, at 2:53 PM, Clark Boylan wrote:
On Mon, Jul 1, 2024, at 11:32 AM, Mike Bayer wrote:
hey gang -
I'm adding asyncio support to oslo.db. Fairly major block hit,
which
I can work around, is that testtools / stestr do not seem to have any provision for asyncio present.
An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase):
import unittest from oslotest import base
class MyTestSuite(unittest.IsolatedAsyncioTestCase,
base.BaseTestCase):
# ...but that didn't work because the degree to which testtools modifies
the
run() method of unittest.TestCase prevents IsolatedAsyncioTestCase
from
running awaitable test cases.
Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools
patterns):
class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ...
this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object.
final workaround, patch one on, here's my base:
class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result)
now I can run asyncio tests. But only by ditching testtools
entirely.
What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?
I think the dependency on testtools was far more important back when
we
were trying to support python2.6, python2.7 and python3.something. In particular support for addCleanup() is/was important to ensure
reliable
cleanups of external resources. I think there were also problems with setUp failures maybe not failing test cases in old python unittest (don't quote me on that I don't remember all the details).
At this point I suspect the need to only support relatively modern python particularly when we talk about adding asyncio support to stuff means we may get away with stdlib unittest over testtools. Testtools was primarily there to bridge some behavior gaps and ensure consistent behavior between python2 and python3. That is no longer a concern.
OK. So...what about the stestr part ? That itself seems to be slightly incompatible with straight unittest at this point. Also what's the role of oslotest ? Noting I'm working on an actual oslo project.
The stestr issue appears to originate in python subunit [0]. Reading this pytest issue [1] addDuration is brand new in python 3.12. I suspect that subunit just needs to be updated to support the new API similar to how pytest was updated. But also it appears that this should only be a warning so shouldn't prevent things from working?
[0] https://github.com/testing-cabal/subunit [1] https://github.com/pytest-dev/pytest/issues/10875
OK that's fine, yes it's only a warning, was just not sure if this was
an indication of a larger incompatibility with straight unittest that doesnt use testtools.TestCase. that was basically my question
FWIW we have a few projects (e.g. metalsmith) that use stestr but neither testtools nor oslotest.
Dmitry
participants (4)
-
Clark Boylan
-
Dmitry Tantsur
-
Herve Beraud
-
Mike Bayer