Open Stack

On Thu, Jan 07 2016, Jay Pipes wrote:

> OK, I just watched that. Sorry, still don't see the value that Mimic provides
> over unit testing the client interfaces and mocking out the HTTP payloads so
> you have strict control over the expectations.
>
> The problem that Glyph noted in the video about the unit test that mocked out
> os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
> find it odd that that example was used in discussing the value of Mimic. Bad
> mocks are just that: bad mocks. The same false positive failure could easily be
> introduced with a typo in the "metadata injection" that Mimic does to inject
> failures into the system.
>
> Mimic isn't testing anything on the server side at all so I'm not sure why
> folks call it "integration testing". It isn't testing the integration of
> anything at all. All it enables is multi-language client library testing, and
> see my response to Ben, the surface area it introduces for bugs in the test
> platform itself in my opinion outweigh the multi-language value it might have.

I completely agree with what Jay points out here.

To illustrate with what we do in the Telemetry team: in order test
gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
client against that freshly installed server. It works very well, and
we're sure we test against real data. That's what I would call
integration testing.
We could also easily test against older version of the Gnocchi server by
pip install-ing an older version if we wanted.

This has way more value than mocking the whole HTTP server and crossing
fingers we did a good job mimicking it. Oh, and it's also actually less
work. :)

¹  https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 800 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160108/301a079d/attachment.pgp>