Open Stack

On 01/07/2016 02:41 PM, Sean Dague wrote:
> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
>> Hi all,
>>
>> A change to global-requirements[1] introduces mimic, which is an http
>> server that can mock various APIs, including nova and ironic, including
>> control of error codes and timeouts. The ironic team plans to use this
>> for testing python-ironicclient without standing up a full ironic
>> environment.
>>
>> Here's the catch - mimic is built on twisted. I know twisted was
>> previously removed from OpenStack (or at least people said "pls no", I
>> don't know the full history). We didn't intend to stealth-introduce
>> twisted back into g-r, but it was pointed out to me that it may appear
>> this way, so here I am letting everyone know. lifeless pointed out that
>> when tests are failing, people may end up digging into mimic or twisted
>> code, which most people in this community aren't familiar with AFAIK,
>> which is a valid point though I hope it isn't required often.
>>
>> So, the primary question here is: do folks have a problem with adding
>> twisted here? We're holding off on Ironic changes that depend on this
>> until this discussion has happened, but aren't reverting the g-r change
>> until we decide one way or another.
>>
>> // jim
>>
>> [1] https://review.openstack.org/#/c/220268/
> What is the advantage of running another server like this over using
> requests-mock (which is used by other OpenStack projects for testing
> today)? The only difference here seems to be that you actually execute
> requests code in one case and not in the other.
>
> Requests-mock debugging when things go wrong seems a bit simpler.
>
> This is less about twisted and more about trying to not introduce yet
> another way to mock code in the tree that people need to understand.

As an outsider and lurker...this is a fair point. I also maintain
another project - StackInABox (and OpenStackInABox) - that resolves the
issue in a slightly different way that utilizes requests-mock to do
something similar, but closer to true unit tests.

The main advantage of mimic (https://pypi.python.org/pypi/mimic) is that
it can be independent of language framework (e.g you can use it for C++
or Java or Python2, Python3, or...) as its an independent process from
your testing, and the tasks can be handled very asynchronously so it can
mock out the responses for various tasks in a timely manner, and several
parts of a system can interact together during a test. You're also
utilizing a common mock of the entire system so multiple projects can
benefit from the same code base. It's kind of integration-test but not
quite - a good fit for projects where unit tests need multiple other
components working together, e.g services that use taskflow would
certainly benefit from using mimic.

The disadvantage of mimic from what I can tell is how the tests are
setup. But I'll leave that more to the mimic users and maintainers
(OpenStack Poppy, Rackspace Autoscale, etc) to explain.

For comparison: My StackInABox project provides a framework upon which
OpenStackInABox builds to provide a common model so multiple projects
benefit. I've used StackInABox on several of my own (non-OpenStack
projects) and my unit tests (run by tox) have been able to be nearly
complete integration tests. Aside from the slight use-case difference
(as exemplified by Rackspace AutoScale),  the main advantage of mimic
over my OpenStackInABox is the momentum behind it due to its use by
Rackspace AutoScale and OpenStack Poppy; I just don't have the resources
to build out the models in OpenStackInABox efficiently or quickly
(though I'd certainly welcome the help to do so). Advantage of the
StackInABox approach is every unit test is self-contained and you don't
have to change code to run the test in most cases (where you do, it's
minimal) - just like with using requests-mock; but you can't have two
services independently hit it - like with Rackspace AutoScale.

HTH,

Ben

P.S here's my two projects:
https://github.com/BenjamenMeyer/stackInABox ~
https://pypi.python.org/pypi/stackinabox
https://github.com/BenjamenMeyer/openstackinabox ~
https://pypi.python.org/pypi/openstackinabox