Open Stack

On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
> >On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
> >>On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
> >>>Hi all,
> >>>
> >>>A change to global-requirements[1] introduces mimic, which is an http
> >>>server that can mock various APIs, including nova and ironic, including
> >>>control of error codes and timeouts. The ironic team plans to use this
> >>>for testing python-ironicclient without standing up a full ironic
> >>>environment.
> >>>
> >>>Here's the catch - mimic is built on twisted. I know twisted was
> >>>previously removed from OpenStack (or at least people said "pls no", I
> >>>don't know the full history). We didn't intend to stealth-introduce
> >>>twisted back into g-r, but it was pointed out to me that it may appear
> >>>this way, so here I am letting everyone know. lifeless pointed out that
> >>>when tests are failing, people may end up digging into mimic or twisted
> >>>code, which most people in this community aren't familiar with AFAIK,
> >>>which is a valid point though I hope it isn't required often.
> >>>
> >>>So, the primary question here is: do folks have a problem with adding
> >>>twisted here? We're holding off on Ironic changes that depend on this
> >>>until this discussion has happened, but aren't reverting the g-r change
> >>>until we decide one way or another.
> >>>
> >>>// jim
> >>>
> >>>[1] https://review.openstack.org/#/c/220268/
> >>
> >>What is the advantage of running another server like this over using
> >>requests-mock (which is used by other OpenStack projects for testing
> >>today)? The only difference here seems to be that you actually execute
> >>requests code in one case and not in the other.
> >>
> >>Requests-mock debugging when things go wrong seems a bit simpler.
> >>
> >>This is less about twisted and more about trying to not introduce yet
> >>another way to mock code in the tree that people need to understand.
> >>
> >>	-Sean
> >
> >We'd be using this for functional tests, not unit, where we can't really
> >inject mocks. The idea is that we could run a full functional suite
> >against either mimic or a full ironic environment, just by changing a
> >test setting.
> 
> I don't really see the point of a separate project like Mimic that has a
> whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
> RAX-specific) API services. It's just a great way to introduce a larger
> surface area for bugs to creep in -- since you have to keep the Mimic
> interfaces up to date with the real interfaces. Better to keep something
> like this -- if it is TRULY needed -- in-tree with the API service itself,
> so that the chances of divergence are reduced. This is similar to the
> fakevirt driver in Nova. It's in tree for good reason: when someone changes
> the virt driver interface, the fakevirt driver goes boom and needs to be
> changed in a corresponding fashion in the same patch.

I tend to think our REST APIs are much more stable than internal python
APIs, and so we can manage the divergence much easier.

Also, mimic can simulate:

* old versions (less needed with all the microversion stuff)
* old bugs that were shipped
* misconfigurations
* networking errors
* the passage of time (including timeouts)

We probably don't want to keep a catalog of old bugs and misconfigurations in
tree forever.

> What value does a functional test against an HTTP API service that does
> nothing (other than introduce greater surface area for bugs) actually offer
> over unit tests anyway?

Testing the full stack of the client instead of mocking the bottom
half of requests is a big one.

Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
https://www.youtube.com/watch?v=HKUUQme3dwA

// jim