Open Stack

yes Jay, i have -W on the bot proposed merge, pending this discussion:
https://review.openstack.org/#/c/263592/

-- Dims

On Fri, Jan 8, 2016 at 2:08 PM, Jay Pipes <jaypipes at gmail.com> wrote:
> On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
>>
>> On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:
>>>
>>> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:
>>>>
>>>> On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
>>>>>
>>>>> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
>>>>>>
>>>>>> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
>>>>>>>
>>>>>>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> A change to global-requirements[1] introduces mimic, which is an
>>>>>>>> http
>>>>>>>> server that can mock various APIs, including nova and ironic,
>>>>>>>> including
>>>>>>>> control of error codes and timeouts. The ironic team plans to use
>>>>>>>> this
>>>>>>>> for testing python-ironicclient without standing up a full ironic
>>>>>>>> environment.
>>>>>>>>
>>>>>>>> Here's the catch - mimic is built on twisted. I know twisted was
>>>>>>>> previously removed from OpenStack (or at least people said "pls no",
>>>>>>>> I
>>>>>>>> don't know the full history). We didn't intend to stealth-introduce
>>>>>>>> twisted back into g-r, but it was pointed out to me that it may
>>>>>>>> appear
>>>>>>>> this way, so here I am letting everyone know. lifeless pointed out
>>>>>>>> that
>>>>>>>> when tests are failing, people may end up digging into mimic or
>>>>>>>> twisted
>>>>>>>> code, which most people in this community aren't familiar with
>>>>>>>> AFAIK,
>>>>>>>> which is a valid point though I hope it isn't required often.
>>>>>>>>
>>>>>>>> So, the primary question here is: do folks have a problem with
>>>>>>>> adding
>>>>>>>> twisted here? We're holding off on Ironic changes that depend on
>>>>>>>> this
>>>>>>>> until this discussion has happened, but aren't reverting the g-r
>>>>>>>> change
>>>>>>>> until we decide one way or another.
>>>>>>>>
>>>>>>>> // jim
>>>>>>>>
>>>>>>>> [1] https://review.openstack.org/#/c/220268/
>>>>>>>
>>>>>>>
>>>>>>> What is the advantage of running another server like this over using
>>>>>>> requests-mock (which is used by other OpenStack projects for testing
>>>>>>> today)? The only difference here seems to be that you actually
>>>>>>> execute
>>>>>>> requests code in one case and not in the other.
>>>>>>>
>>>>>>> Requests-mock debugging when things go wrong seems a bit simpler.
>>>>>>>
>>>>>>> This is less about twisted and more about trying to not introduce yet
>>>>>>> another way to mock code in the tree that people need to understand.
>>>>>>>
>>>>>>>         -Sean
>>>>>>
>>>>>>
>>>>>> We'd be using this for functional tests, not unit, where we can't
>>>>>> really
>>>>>> inject mocks. The idea is that we could run a full functional suite
>>>>>> against either mimic or a full ironic environment, just by changing a
>>>>>> test setting.
>>>>>
>>>>>
>>>>> I don't really see the point of a separate project like Mimic that has
>>>>> a
>>>>> whole bunch of reimplementations (mocked out) of all sorts of OpenStack
>>>>> (and
>>>>> RAX-specific) API services. It's just a great way to introduce a larger
>>>>> surface area for bugs to creep in -- since you have to keep the Mimic
>>>>> interfaces up to date with the real interfaces. Better to keep
>>>>> something
>>>>> like this -- if it is TRULY needed -- in-tree with the API service
>>>>> itself,
>>>>> so that the chances of divergence are reduced. This is similar to the
>>>>> fakevirt driver in Nova. It's in tree for good reason: when someone
>>>>> changes
>>>>> the virt driver interface, the fakevirt driver goes boom and needs to
>>>>> be
>>>>> changed in a corresponding fashion in the same patch.
>>>>
>>>>
>>>> I tend to think our REST APIs are much more stable than internal python
>>>> APIs, and so we can manage the divergence much easier.
>>>>
>>>> Also, mimic can simulate:
>>>>
>>>> * old versions (less needed with all the microversion stuff)
>>>> * old bugs that were shipped
>>>> * misconfigurations
>>>> * networking errors
>>>> * the passage of time (including timeouts)
>>>>
>>>> We probably don't want to keep a catalog of old bugs and
>>>> misconfigurations in
>>>> tree forever.
>>>>
>>>>> What value does a functional test against an HTTP API service that does
>>>>> nothing (other than introduce greater surface area for bugs) actually
>>>>> offer
>>>>> over unit tests anyway?
>>>>
>>>>
>>>> Testing the full stack of the client instead of mocking the bottom
>>>> half of requests is a big one.
>>>>
>>>> Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
>>>> https://www.youtube.com/watch?v=HKUUQme3dwA
>>>
>>>
>>> OK, I just watched that. Sorry, still don't see the value that Mimic
>>> provides over unit testing the client interfaces and mocking out the HTTP
>>> payloads so you have strict control over the expectations.
>>>
>>> The problem that Glyph noted in the video about the unit test that mocked
>>> out os.chdir to improperly return a value isn't solved whatsoever by
>>> Mimic,
>>> so I find it odd that that example was used in discussing the value of
>>> Mimic. Bad mocks are just that: bad mocks. The same false positive
>>> failure
>>> could easily be introduced with a typo in the "metadata injection" that
>>> Mimic does to inject failures into the system.
>>>
>>> Mimic isn't testing anything on the server side at all so I'm not sure
>>> why
>>> folks call it "integration testing". It isn't testing the integration of
>>> anything at all. All it enables is multi-language client library testing,
>>> and see my response to Ben, the surface area it introduces for bugs in
>>> the
>>> test platform itself in my opinion outweigh the multi-language value it
>>> might have.
>>
>>
>> FWIW, I've only been calling it functional testing. I also don't care
>> about the multi-language parts here. The sole purpose we have for mimic
>> (so far) is to functionally test python-ironicclient without mocking the
>> world in ironicclient.
>>
>>> Final question on this... if Mimic is *only* for testing clients, why not
>>> make it just a dependency for python-ironicclient and not ironic itself?
>>
>>
>> We haven't made it a dep for anything yet, only added to g-r.
>
>
> According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> makes functionally any difference, though (pun intended).
>
>> However, now that you mention that, a really ambitious goal would be to
>> add a rabbit interface to mimic, and functionally test the API server
>> (that it sends the right messages, etc). Another would be to mimic
>> (Neutron, Glance) and test Ironic by itself.
>
>
> So you would reimplement AMQP communication protocols using an in-memory
> data store for queues. Sounds like an even greater surface area for bugs to
> be introduced.
>
>> Last, I frankly don't understand why there's
>> such heavy opposition to the ironic team using an additional tool for
>> testing.
>
>
> Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> though.
>
> a) Because it fractures the testing and QA processes used by upstream
> contributors that work on OpenStack projects by requiring them to learn
> another system -- and one that potentially would require them to understand
> a whole new surface area for potential bugs
>
> b) Because it represents yet another RAX-driven divergence in the QA space.
> CloudCafe took essentially all of the RAX folks that were at one point
> working on Tempest and upstream QA and siloed them into a totally different
> organization, in true RAX fashion. Instead of pulling the OpenStack QA
> community along together, RAX QA continues to just do its own thing and
> there's still bitterness on the tips of tongues.
>
> <snip>
>
>> There was *more* than sufficient time for "I don't think this is useful"
>> to be posted on the review.
>
>
> Sorry, this was the first I'd heard of it all.
>
>> If anyone has a hard technical reason why
>>
>> mimic should not be used to test an OpenStack project, I'm happy to
>> listen. Otherwise I'd like to work on actually getting things done.
>
>
> I've listed my hard technical reason: it introduces, IMHO, too large of a
> surface area for bugs to creep in to the testing process versus the benefit
> it provides.
>
> Best,
> -jay
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims