Open Stack

On 08/01/16 04:49 AM, Julien Danjou wrote:
> On Thu, Jan 07 2016, Jay Pipes wrote:
>
>> OK, I just watched that. Sorry, still don't see the value that Mimic provides
>> over unit testing the client interfaces and mocking out the HTTP payloads so
>> you have strict control over the expectations.
>>
>> The problem that Glyph noted in the video about the unit test that mocked out
>> os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
>> find it odd that that example was used in discussing the value of Mimic. Bad
>> mocks are just that: bad mocks. The same false positive failure could easily be
>> introduced with a typo in the "metadata injection" that Mimic does to inject
>> failures into the system.
>>
>> Mimic isn't testing anything on the server side at all so I'm not sure why
>> folks call it "integration testing". It isn't testing the integration of
>> anything at all. All it enables is multi-language client library testing, and
>> see my response to Ben, the surface area it introduces for bugs in the test
>> platform itself in my opinion outweigh the multi-language value it might have.
> I completely agree with what Jay points out here.
>
> To illustrate with what we do in the Telemetry team: in order test
> gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
> client against that freshly installed server. It works very well, and
> we're sure we test against real data. That's what I would call
> integration testing.
> We could also easily test against older version of the Gnocchi server by
> pip install-ing an older version if we wanted.
>
> This has way more value than mocking the whole HTTP server and crossing
> fingers we did a good job mimicking it. Oh, and it's also actually less
> work. :)
>
> ¹  https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh
>

this happens with aodhclient as well. the one caveat i would mention is 
that it doesn't really allow for test isolation which requires you to 
put a bit more thought into your tests.

cheers,

-- 
gord