Open Stack

On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:
> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:
> >On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
> >>On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
> >>>On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
> >>>>On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
> >>>>>Hi all,
> >>>>>
> >>>>>A change to global-requirements[1] introduces mimic, which is an http
> >>>>>server that can mock various APIs, including nova and ironic, including
> >>>>>control of error codes and timeouts. The ironic team plans to use this
> >>>>>for testing python-ironicclient without standing up a full ironic
> >>>>>environment.
> >>>>>
> >>>>>Here's the catch - mimic is built on twisted. I know twisted was
> >>>>>previously removed from OpenStack (or at least people said "pls no", I
> >>>>>don't know the full history). We didn't intend to stealth-introduce
> >>>>>twisted back into g-r, but it was pointed out to me that it may appear
> >>>>>this way, so here I am letting everyone know. lifeless pointed out that
> >>>>>when tests are failing, people may end up digging into mimic or twisted
> >>>>>code, which most people in this community aren't familiar with AFAIK,
> >>>>>which is a valid point though I hope it isn't required often.
> >>>>>
> >>>>>So, the primary question here is: do folks have a problem with adding
> >>>>>twisted here? We're holding off on Ironic changes that depend on this
> >>>>>until this discussion has happened, but aren't reverting the g-r change
> >>>>>until we decide one way or another.
> >>>>>
> >>>>>// jim
> >>>>>
> >>>>>[1] https://review.openstack.org/#/c/220268/
> >>>>
> >>>>What is the advantage of running another server like this over using
> >>>>requests-mock (which is used by other OpenStack projects for testing
> >>>>today)? The only difference here seems to be that you actually execute
> >>>>requests code in one case and not in the other.
> >>>>
> >>>>Requests-mock debugging when things go wrong seems a bit simpler.
> >>>>
> >>>>This is less about twisted and more about trying to not introduce yet
> >>>>another way to mock code in the tree that people need to understand.
> >>>>
> >>>>	-Sean
> >>>
> >>>We'd be using this for functional tests, not unit, where we can't really
> >>>inject mocks. The idea is that we could run a full functional suite
> >>>against either mimic or a full ironic environment, just by changing a
> >>>test setting.
> >>
> >>I don't really see the point of a separate project like Mimic that has a
> >>whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
> >>RAX-specific) API services. It's just a great way to introduce a larger
> >>surface area for bugs to creep in -- since you have to keep the Mimic
> >>interfaces up to date with the real interfaces. Better to keep something
> >>like this -- if it is TRULY needed -- in-tree with the API service itself,
> >>so that the chances of divergence are reduced. This is similar to the
> >>fakevirt driver in Nova. It's in tree for good reason: when someone changes
> >>the virt driver interface, the fakevirt driver goes boom and needs to be
> >>changed in a corresponding fashion in the same patch.
> >
> >I tend to think our REST APIs are much more stable than internal python
> >APIs, and so we can manage the divergence much easier.
> >
> >Also, mimic can simulate:
> >
> >* old versions (less needed with all the microversion stuff)
> >* old bugs that were shipped
> >* misconfigurations
> >* networking errors
> >* the passage of time (including timeouts)
> >
> >We probably don't want to keep a catalog of old bugs and misconfigurations in
> >tree forever.
> >
> >>What value does a functional test against an HTTP API service that does
> >>nothing (other than introduce greater surface area for bugs) actually offer
> >>over unit tests anyway?
> >
> >Testing the full stack of the client instead of mocking the bottom
> >half of requests is a big one.
> >
> >Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
> >https://www.youtube.com/watch?v=HKUUQme3dwA
> 
> OK, I just watched that. Sorry, still don't see the value that Mimic
> provides over unit testing the client interfaces and mocking out the HTTP
> payloads so you have strict control over the expectations.
> 
> The problem that Glyph noted in the video about the unit test that mocked
> out os.chdir to improperly return a value isn't solved whatsoever by Mimic,
> so I find it odd that that example was used in discussing the value of
> Mimic. Bad mocks are just that: bad mocks. The same false positive failure
> could easily be introduced with a typo in the "metadata injection" that
> Mimic does to inject failures into the system.
> 
> Mimic isn't testing anything on the server side at all so I'm not sure why
> folks call it "integration testing". It isn't testing the integration of
> anything at all. All it enables is multi-language client library testing,
> and see my response to Ben, the surface area it introduces for bugs in the
> test platform itself in my opinion outweigh the multi-language value it
> might have.

FWIW, I've only been calling it functional testing. I also don't care
about the multi-language parts here. The sole purpose we have for mimic
(so far) is to functionally test python-ironicclient without mocking the
world in ironicclient.

> Final question on this... if Mimic is *only* for testing clients, why not
> make it just a dependency for python-ironicclient and not ironic itself?

We haven't made it a dep for anything yet, only added to g-r.

However, now that you mention that, a really ambitious goal would be to
add a rabbit interface to mimic, and functionally test the API server
(that it sends the right messages, etc). Another would be to mimic
(Neutron, Glance) and test Ironic by itself.

Last, I frankly don't understand why there's
such heavy opposition to the ironic team using an additional tool for
testing. Clearly we see some value in using mimic; I'd love to make
progress on exploring that and seeing if it brings the value we think it
will, rather than talking about why others think it's useless. The g-r
change is already merged; lifeless asked me to start this thread because
twisted is a touchy subject. I didn't start this thread to get buyoff
from the entire community on using a new test tool in a single project.

We have code review for a reason. This review went up on September 3. It
was frozen for Liberty final. In October it was blocked on Python 3
compatibility. The mimic team worked to make it Python 3 compatible
(including some of the dependencies), and released a compatible version
in December. People were plugging this patch in IRC on and off because
they want to go ahead and actually use the library.

There was *more* than sufficient time for "I don't think this is useful"
to be posted on the review. If anyone has a hard technical reason why
mimic should not be used to test an OpenStack project, I'm happy to
listen. Otherwise I'd like to work on actually getting things done.

// jim