Open Stack

On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:
> On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
>> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
>>> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
>>>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
>>>>> Hi all,
>>>>>
>>>>> A change to global-requirements[1] introduces mimic, which is an http
>>>>> server that can mock various APIs, including nova and ironic, including
>>>>> control of error codes and timeouts. The ironic team plans to use this
>>>>> for testing python-ironicclient without standing up a full ironic
>>>>> environment.
>>>>>
>>>>> Here's the catch - mimic is built on twisted. I know twisted was
>>>>> previously removed from OpenStack (or at least people said "pls no", I
>>>>> don't know the full history). We didn't intend to stealth-introduce
>>>>> twisted back into g-r, but it was pointed out to me that it may appear
>>>>> this way, so here I am letting everyone know. lifeless pointed out that
>>>>> when tests are failing, people may end up digging into mimic or twisted
>>>>> code, which most people in this community aren't familiar with AFAIK,
>>>>> which is a valid point though I hope it isn't required often.
>>>>>
>>>>> So, the primary question here is: do folks have a problem with adding
>>>>> twisted here? We're holding off on Ironic changes that depend on this
>>>>> until this discussion has happened, but aren't reverting the g-r change
>>>>> until we decide one way or another.
>>>>>
>>>>> // jim
>>>>>
>>>>> [1] https://review.openstack.org/#/c/220268/
>>>>
>>>> What is the advantage of running another server like this over using
>>>> requests-mock (which is used by other OpenStack projects for testing
>>>> today)? The only difference here seems to be that you actually execute
>>>> requests code in one case and not in the other.
>>>>
>>>> Requests-mock debugging when things go wrong seems a bit simpler.
>>>>
>>>> This is less about twisted and more about trying to not introduce yet
>>>> another way to mock code in the tree that people need to understand.
>>>>
>>>> 	-Sean
>>>
>>> We'd be using this for functional tests, not unit, where we can't really
>>> inject mocks. The idea is that we could run a full functional suite
>>> against either mimic or a full ironic environment, just by changing a
>>> test setting.
>>
>> I don't really see the point of a separate project like Mimic that has a
>> whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
>> RAX-specific) API services. It's just a great way to introduce a larger
>> surface area for bugs to creep in -- since you have to keep the Mimic
>> interfaces up to date with the real interfaces. Better to keep something
>> like this -- if it is TRULY needed -- in-tree with the API service itself,
>> so that the chances of divergence are reduced. This is similar to the
>> fakevirt driver in Nova. It's in tree for good reason: when someone changes
>> the virt driver interface, the fakevirt driver goes boom and needs to be
>> changed in a corresponding fashion in the same patch.
>
> I tend to think our REST APIs are much more stable than internal python
> APIs, and so we can manage the divergence much easier.
>
> Also, mimic can simulate:
>
> * old versions (less needed with all the microversion stuff)
> * old bugs that were shipped
> * misconfigurations
> * networking errors
> * the passage of time (including timeouts)
>
> We probably don't want to keep a catalog of old bugs and misconfigurations in
> tree forever.
>
>> What value does a functional test against an HTTP API service that does
>> nothing (other than introduce greater surface area for bugs) actually offer
>> over unit tests anyway?
>
> Testing the full stack of the client instead of mocking the bottom
> half of requests is a big one.
>
> Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
> https://www.youtube.com/watch?v=HKUUQme3dwA

OK, I just watched that. Sorry, still don't see the value that Mimic 
provides over unit testing the client interfaces and mocking out the 
HTTP payloads so you have strict control over the expectations.

The problem that Glyph noted in the video about the unit test that 
mocked out os.chdir to improperly return a value isn't solved whatsoever 
by Mimic, so I find it odd that that example was used in discussing the 
value of Mimic. Bad mocks are just that: bad mocks. The same false 
positive failure could easily be introduced with a typo in the "metadata 
injection" that Mimic does to inject failures into the system.

Mimic isn't testing anything on the server side at all so I'm not sure 
why folks call it "integration testing". It isn't testing the 
integration of anything at all. All it enables is multi-language client 
library testing, and see my response to Ben, the surface area it 
introduces for bugs in the test platform itself in my opinion outweigh 
the multi-language value it might have.

Final question on this... if Mimic is *only* for testing clients, why 
not make it just a dependency for python-ironicclient and not ironic itself?

Best,
-jay