Open Stack

On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:
>> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:
>>> On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
>>>> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
>>>>> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
>>>>>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> A change to global-requirements[1] introduces mimic, which is an http
>>>>>>> server that can mock various APIs, including nova and ironic, including
>>>>>>> control of error codes and timeouts. The ironic team plans to use this
>>>>>>> for testing python-ironicclient without standing up a full ironic
>>>>>>> environment.
>>>>>>>
>>>>>>> Here's the catch - mimic is built on twisted. I know twisted was
>>>>>>> previously removed from OpenStack (or at least people said "pls no", I
>>>>>>> don't know the full history). We didn't intend to stealth-introduce
>>>>>>> twisted back into g-r, but it was pointed out to me that it may appear
>>>>>>> this way, so here I am letting everyone know. lifeless pointed out that
>>>>>>> when tests are failing, people may end up digging into mimic or twisted
>>>>>>> code, which most people in this community aren't familiar with AFAIK,
>>>>>>> which is a valid point though I hope it isn't required often.
>>>>>>>
>>>>>>> So, the primary question here is: do folks have a problem with adding
>>>>>>> twisted here? We're holding off on Ironic changes that depend on this
>>>>>>> until this discussion has happened, but aren't reverting the g-r change
>>>>>>> until we decide one way or another.
>>>>>>>
>>>>>>> // jim
>>>>>>>
>>>>>>> [1] https://review.openstack.org/#/c/220268/
>>>>>>
>>>>>> What is the advantage of running another server like this over using
>>>>>> requests-mock (which is used by other OpenStack projects for testing
>>>>>> today)? The only difference here seems to be that you actually execute
>>>>>> requests code in one case and not in the other.
>>>>>>
>>>>>> Requests-mock debugging when things go wrong seems a bit simpler.
>>>>>>
>>>>>> This is less about twisted and more about trying to not introduce yet
>>>>>> another way to mock code in the tree that people need to understand.
>>>>>>
>>>>>> 	-Sean
>>>>>
>>>>> We'd be using this for functional tests, not unit, where we can't really
>>>>> inject mocks. The idea is that we could run a full functional suite
>>>>> against either mimic or a full ironic environment, just by changing a
>>>>> test setting.
>>>>
>>>> I don't really see the point of a separate project like Mimic that has a
>>>> whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
>>>> RAX-specific) API services. It's just a great way to introduce a larger
>>>> surface area for bugs to creep in -- since you have to keep the Mimic
>>>> interfaces up to date with the real interfaces. Better to keep something
>>>> like this -- if it is TRULY needed -- in-tree with the API service itself,
>>>> so that the chances of divergence are reduced. This is similar to the
>>>> fakevirt driver in Nova. It's in tree for good reason: when someone changes
>>>> the virt driver interface, the fakevirt driver goes boom and needs to be
>>>> changed in a corresponding fashion in the same patch.
>>>
>>> I tend to think our REST APIs are much more stable than internal python
>>> APIs, and so we can manage the divergence much easier.
>>>
>>> Also, mimic can simulate:
>>>
>>> * old versions (less needed with all the microversion stuff)
>>> * old bugs that were shipped
>>> * misconfigurations
>>> * networking errors
>>> * the passage of time (including timeouts)
>>>
>>> We probably don't want to keep a catalog of old bugs and misconfigurations in
>>> tree forever.
>>>
>>>> What value does a functional test against an HTTP API service that does
>>>> nothing (other than introduce greater surface area for bugs) actually offer
>>>> over unit tests anyway?
>>>
>>> Testing the full stack of the client instead of mocking the bottom
>>> half of requests is a big one.
>>>
>>> Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
>>> https://www.youtube.com/watch?v=HKUUQme3dwA
>>
>> OK, I just watched that. Sorry, still don't see the value that Mimic
>> provides over unit testing the client interfaces and mocking out the HTTP
>> payloads so you have strict control over the expectations.
>>
>> The problem that Glyph noted in the video about the unit test that mocked
>> out os.chdir to improperly return a value isn't solved whatsoever by Mimic,
>> so I find it odd that that example was used in discussing the value of
>> Mimic. Bad mocks are just that: bad mocks. The same false positive failure
>> could easily be introduced with a typo in the "metadata injection" that
>> Mimic does to inject failures into the system.
>>
>> Mimic isn't testing anything on the server side at all so I'm not sure why
>> folks call it "integration testing". It isn't testing the integration of
>> anything at all. All it enables is multi-language client library testing,
>> and see my response to Ben, the surface area it introduces for bugs in the
>> test platform itself in my opinion outweigh the multi-language value it
>> might have.
>
> FWIW, I've only been calling it functional testing. I also don't care
> about the multi-language parts here. The sole purpose we have for mimic
> (so far) is to functionally test python-ironicclient without mocking the
> world in ironicclient.
>
>> Final question on this... if Mimic is *only* for testing clients, why not
>> make it just a dependency for python-ironicclient and not ironic itself?
>
> We haven't made it a dep for anything yet, only added to g-r.

According to Dims, not to g-r, but to u-c, right Dims? Not sure if that 
makes functionally any difference, though (pun intended).

> However, now that you mention that, a really ambitious goal would be to
> add a rabbit interface to mimic, and functionally test the API server
> (that it sends the right messages, etc). Another would be to mimic
> (Neutron, Glance) and test Ironic by itself.

So you would reimplement AMQP communication protocols using an in-memory 
data store for queues. Sounds like an even greater surface area for bugs 
to be introduced.

> Last, I frankly don't understand why there's
> such heavy opposition to the ironic team using an additional tool for
> testing.

Since you asked, I'll be blunt. This isn't a personal attack on you, 
Jim, though.

a) Because it fractures the testing and QA processes used by upstream 
contributors that work on OpenStack projects by requiring them to learn 
another system -- and one that potentially would require them to 
understand a whole new surface area for potential bugs

b) Because it represents yet another RAX-driven divergence in the QA 
space. CloudCafe took essentially all of the RAX folks that were at one 
point working on Tempest and upstream QA and siloed them into a totally 
different organization, in true RAX fashion. Instead of pulling the 
OpenStack QA community along together, RAX QA continues to just do its 
own thing and there's still bitterness on the tips of tongues.

<snip>

> There was *more* than sufficient time for "I don't think this is useful"
> to be posted on the review.

Sorry, this was the first I'd heard of it all.

 > If anyone has a hard technical reason why
> mimic should not be used to test an OpenStack project, I'm happy to
> listen. Otherwise I'd like to work on actually getting things done.

I've listed my hard technical reason: it introduces, IMHO, too large of 
a surface area for bugs to creep in to the testing process versus the 
benefit it provides.

Best,
-jay