[openstack-dev] [infra]Requesting consideration of httmock package for test-requirements in Juno
Jamie Lennox
jamielennox at redhat.com
Tue Apr 15 00:09:32 UTC 2014
On Fri, 2014-04-11 at 13:29 +0000, Paul Michali (pcm) wrote:
> See inline @PCM…
>
>
> On Apr 9, 2014, at 5:56 PM, Jamie Lennox <jamielennox at redhat.com>
> wrote:
>
>
> >
> >
> > ----- Original Message -----
> > > From: "Paul Michali (pcm)" <pcm at cisco.com>
> > > To: "OpenStack Development Mailing List (not for usage questions)"
> > > <openstack-dev at lists.openstack.org>
> > > Sent: Wednesday, April 9, 2014 6:31:09 AM
> > > Subject: Re: [openstack-dev] [infra]Requesting consideration of
> > > httmock package for test-requirements in Juno
> > >
> > > On Apr 8, 2014, at 3:04 PM, Jamie Lennox < jamielennox at redhat.com
> > > > wrote:
> > >
> > >
> > >
> > >
> > >
> > >
> > > ----- Original Message -----
> > >
> > >
> > > From: "Paul Michali (pcm)" < pcm at cisco.com >
> > > To: "OpenStack Development Mailing List (not for usage questions)"
> > > <
> > > openstack-dev at lists.openstack.org >
> > > Cc: jamielennox at gmail.com
> > > Sent: Wednesday, April 9, 2014 12:09:58 AM
> > > Subject: [openstack-dev] [infra]Requesting consideration of
> > > httmock package
> > > for test-requirements in Juno
> > >
> > > Reposting this, after discussing with Sean Dague…
> > >
> > > For background, I have developed a REST client lib to talk to a
> > > H/W device
> > > with REST server for VPNaaS in Neutron. To support unit testing of
> > > this, I
> > > created a UT module and a mock REST server module and used the
> > > httmock
> > > package. I found it easy to use, and was able to easily create a
> > > sub-class
> > > of my UT to run the same test cases with real H/W, instead of the
> > > mock REST
> > > server. See the original email below, for links of the UT and REST
> > > mock to
> > > see how I used it.
> > >
> > >
> > > I created a bug under requirements, to propose adding httmock to
> > > the
> > > test-requirements. Sean mentioned that there is an existing mock
> > > package,
> > > called httpretty , which I found is used in keystone client UTs),
> > > and should
> > > petition to see if httmock should replace httpretty, since the two
> > > appear to
> > > overlap in functionality.
> > >
> > > I found this link, with a brief comparison of the two:
> > > http://marekbrzoska.wordpress.com/2013/08/28/mocking-http-requests-in-python/
> > >
> > > So… I’m wondering if the community is interested in adopting this
> > > package
> > > (with the goal of deprecating the httpretty package). Otherwise, I
> > > will work
> > > on reworking the UT code I have to try to use httpretty.
> > >
> > > Would be interested in peoples’ thoughts, especially those who
> > > have worked
> > > with httpretty.
> > >
> > > Thanks in advance!
> > >
> > > So I introduced HTTPretty into the requirements and did the work
> > > around
> > > keystoneclient and am well aware that it has a few warts.
> > >
> > > PCM: Great, I grabbed your name from keystone client logs and was
> > > hoping you
> > > had some knowledge of httpretty.
> > >
> > >
> > >
> > >
> > >
> > >
> > > At the time we were going through the changeover from httplib to
> > > requests and
> > > httpretty gave a good way to change over the library and ensure
> > > that we
> > > hadn't actually changed the issued requests at all. If we had
> > > already been
> > > on requests i don't know if i'd have made the same choice.
> > >
> > > In general I am in favour of mocking the response layer rather
> > > than the
> > > client layer - whether we do this with httpretty or httmock
> > > doesn't bother
> > > me that much. Honestly I don't think a global patch of the
> > > requests Session
> > > object is that much safer that a global patch of the socket
> > > interface, if
> > > anything requests is under development and so this interface is
> > > less
> > > defined.
> > >
> > > PCM: Not sure that httmock can be considered a global patch. It is
> > > a context
> > > lib that intercepts the call through various decorators where the
> > > request
> > > can be filtered/processed and if not, will fall through and call
> > > the actual
> > > library.
> > >
> > > So, with the context lib, you can define several handlers for the
> > > request(s).
> > > When the call is made, it will try each handler and if they all
> > > return None,
> > > will call the original function, otherwise they return the value
> > > of the mock
> > > routine. Here’s an example front he test cases I cerated:
> > >
> > > with httmock.HTTMock(csr_request.token, csr_request.put,
> > > csr_request.normal_get):
> > > keepalive_info = {'interval': 60, 'retry': 4}
> > > self.csr.configure_ike_keepalive (keepalive_info)
> > > self.assertEqual(requests.codes.NO_CONTENT, self.csr.status)
> > > content = self.csr.get_request ('vpn-svc/ike/keepalive')
> > > self.assertEqual(requests.codes.OK, self.csr.status)
> > > expected = {'periodic': False}
> > > expected.update(keepalive_info)
> > > self.assertDictContainsSubset(expected, content)
> > >
> > > The client code (red) does a POST with authentication info to get
> > > token, does
> > > a PUT with the setting, and then a GET to verify the value. The
> > > mock module
> > > has these methods created:
> > >
> > > @httmock.urlmatch ( netloc =r'localhost')
> > > def token( url , request):
> > > if ' auth /token-services' in url.path:
> > > return {'status_code': requests.codes.OK,
> > > 'content': {'token-id': 'dummy-token'}}
> > >
> > >
> > > @httmock.urlmatch ( netloc =r'localhost')
> > > def normal_get( url , request):
> > > if request.method != 'GET':
> > > return
> > > if not request.headers.get('X- auth -token', None):
> > > return {'status_code': requests.codes.UNAUTHORIZED}
> > > …
> > > if 'vpn-svc/ike/keepalive' in url.path:
> > > content = {u'interval': 60,
> > > u'retry': 4,
> > > u'periodic': True}
> > > return httmock.response(requests.codes.OK, content=content)
> > >
> > > @httmock.urlmatch(netloc=r'localhost')
> > > def put(url, request):
> > > if request.method != 'PUT':
> > > return
> > > if not request.headers.get('X-auth-token', None):
> > > return {'status_code': requests.codes.UNAUTHORIZED}
> > > return {'status_code': requests.codes.NO_CONTENT}
> > >
> > >
> > > Just a few notes….
> > > A) Could have created separate context lib for put vs get.
> > > B) Could have a method for specific URI request matches (I do that
> > > in some
> > > places, in others I catch all requests).
> > > C) Can filter the requests based on request type or URI.
> > > D) Can catch all URLs or filter.
> > > E) Additional decorators can be created for these handlers.
> > > F) There is lots of flexibility with manipulating the response
> > > data.
> > >
> > > For (C) I’ve done things like this:
> > >
> > > @filter_request(['post'], 'vpn-svc/site-to-site')
> > > @httmock.urlmatch(netloc=r'localhost')
> > > def post_missing_ipsec_policy(url, request):
> > > if not request.headers.get('X-auth-token', None):
> > > return {'status_code': requests.codes.UNAUTHORIZED}
> > > return {'status_code': requests.codes.BAD_REQUEST}
> > >
> > > For (D), I have all my handlers set for localhost, and then I have
> > > another
> > > test module that creates a subclass of the test class, sets the
> > > host to an
> > > IP of a live system, and runs the same tests, only this time
> > > against a live
> > > router, instead the mock module. This saves a lot of coding and
> > > allows me to
> > > reuse the test code (and make sure it REALLY works with real
> > > hardware, in my
> > > case).
> > >
> > > For (E), I did this to simulate a timeout in a request and then
> > > latter
> > > success on retry:
> > >
> > > @filter_request (['get'], 'global/host-name')
> > > @repeat(1)
> > > @httmock.urlmatch ( netloc =r'localhost')
> > > def expired_request( url , request):
> > > """Simulate access denied failure on first request for this
> > > resource.
> > >
> > > Intent here is to simulate that the token has expired, by failing
> > > the first request to the resource. Because of the repeat=1, this
> > > will only be called once, and subsequent calls will not be handled
> > > by this function, but instead will access the normal handler and
> > > will pass. Currently configured for a GET request, but will work
> > > with POST and PUT as well. For DELETE, would need to
> > > filter_request on a
> > > different resource (e.g. 'global/local-users')
> > > """
> > >
> > > return {'status_code': requests.codes.UNAUTHORIZED}
> > >
> > > Ref:
> > > https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py
> > > Ref:
> > > https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/cisco_csr_mock.py
> > >
> >
> > The reason that i call this a global mock is that behind the context
> > manager it is still patching the class method not an individual
> > instance of a requests.Session object. see:
> > https://github.com/patrys/httmock/blob/master/httmock.py#L142
>
>
> @PCM Good point. I forgot that it did that.
>
>
>
> >
> > Everything you have specified is doable with httpretty.
> >
> > @httpretty.activate does the mocking of the library (socket rather
> > than requests.Session) then you use
> >
> > httpretty.register_uri(httpretty.GET,
> > 'http://testurl/testpath',
> > body=json.dumps({'hello': 'world'},
> > status=200)
> >
> > You can do the same multiple queued responses, responses via callbck
> > and everything else. Also the thing i find a use a lot that i don't
> > see provided by httmock is things like:
> >
> > self.assertEqual(httpretty.last_request().headers['X-Auth-Token'],
> > expected_token)
> >
> > In which you test that the request that you send is actually what
> > you expect as well.
>
>
> @PCM So this is testing that what was sent to the code under test is
> what it sends out in the HTTP request? I’m not sure I see the
> advantage in testing that. I usually just test that the response is
> correct, given the request data I send to the code under test.
>
>
>
> >
> > Anyway there is a pretty long readme on the github page:
> > https://github.com/gabrielfalcao/HTTPretty that explains these sort
> > of features.
>
>
> @PCM Thanks. I’ll take a look at this.
>
>
>
> >
> >
> > > What i would like to see though is this mocking transferred into
> > > fixtures
> > > like in https://review.openstack.org/#/c/77961/ and have the
> > > actual choice
> > > in mock library hidden behind those fixtures. Is this a pattern
> > > that httmock
> > > can handle? or something else again?
> > >
> > > PCM I see your point - it would be nice to be able to have the
> > > flexibility to
> > > swap the underlying mock libraries. Would have to think about it
> > > more, to
> > > see if a common fixture can be used for httpretty and httmock. I’m
> > > not sure
> > > if they are different enough (one monkey patching, one using
> > > context lib)
> > > that it would be possible (and I don’t have a strong understanding
> > > of
> > > httpretty yet).
> > >
> > > Would love to hear peoples’ thoughts no this.
> >
> > Again, i'm not that strongly in favour of HTTPretty over httmock
> > (though missing last_request will be a problem).
>
>
> @PCM I’ll take a look and see if I can change all my UTs to use
> HTTPretty. I’ll be curious to see if it can handle all the cases I
> use. In particular, I have two test modules, one which has test cases
> that, by using a URL directed to the localhost, makes use of a mock
> REST server, to exercise my client REST module.
>
>
> I have a second test module, which has subclasses for the test classes
> in the first module, only in setup, it specifies the URL of a “real”
> REST server. When I run this module, which is very small, it leverages
> off all the test cases, but verifies with a real device, which is the
> true proof that the REST client under test actually works. Granted,
> it wouldn’t be used during a UT TOX run, but it could be used with a
> Tempest run.
>
>
> From an httmock standpoint there is no change, as the handler methods
> have filters to only catch requests to a URL directed to localhost.
> The “live” test will end up falling through and making a REST request
> to a real server.
This is the same as HTTPretty, if you don't catch the URL then it will
pass through as normal (IMO this is the wrong default for unit testing)
so you should be able to do the same thing.
>
> Likewise, I have tests for checking timeouts, where I have the desired
> call map to the mock, so the request fails, but other requests flow
> through to the live server.
>
>
> Any thoughts on how I would be able to do these things with HTTPretty?
>
I am not sure about timeout exceptions, as a last resort you can specify
a callback for a response and if you raised the appropriate exception
there it should filter up.
>
>
>
> > My point is just that the two libraries can both do the same job and
> > i'm wondering what the advantage of the new one is.
>
>
> @PCM One thing is that I think the context manager technique of
> handling the mocking is (subjectively) a really clean way to do
> things. The other is this duality of calls passing through to a real
> (non-mock) server without altering the test cases.
>
>
>
>
>
> >
> > As mentioned i would really like to see this moved into fixtures -
> > httpretty will allow that somewhat, i don't see that httmock will,
>
>
> @PCM My guess is that probably either method could be wrapped by a
> fixture, but I’m also guessing that they are different enough such
> that it may be very hard to come up with a fixture to accommodate
> both.
>
>
>
> > and if i had the time i think i would just rewrite a library like
> > this making use of the tools we already have like mock, fixtures,
> > and testresources. (My other thought is to just subclass a
> > requests.Session object with an overridden request method -rather
> > than a global patch- and allow that to be passed into clients but i
> > don't think they all support that).
>
>
> @PCM I’ll try to see if I can adapt my UTs to use HTTPretty and think
> a little bit on how a test fixture could be used.
I actually had a play the other day to see just how hard it would be to
do my own requests mocking library that provided the kind of interfaces
i was looking for. Answer is quite easy:
https://github.com/jamielennox/requests-mock
I'm not sure i want to get into maintaining a library like that but
maybe we could get the httmock to expand there scope a little and
provide some hooks that would adapt to the fixture pattern more easily.
Jamie
> Regards,
>
>
> PCM (Paul Michali)
>
>
> MAIL …..…. pcm at cisco.com
> IRC ……..… pcm_ (irc.freenode.com)
> TW ………... @pmichali
> GPG Key … 4525ECC253E31A83
> Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
>
>
>
>
> >
> >
> > Jamie
> >
> >
> >
> > >
> > > PCM (Paul Michali)
> > >
> > > MAIL …..…. pcm at cisco.com
> > > IRC ……..… pcm_ ( irc.freenode.com )
> > > TW ………... @pmichali
> > > GPG Key … 4525ECC253E31A83
> > > Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Jamie
> > >
> > >
> > >
> > >
> > > PCM (Paul Michali)
> > >
> > > MAIL …..…. pcm at cisco.com
> > > IRC ……..… pcm_ ( irc.freenode.com )
> > > TW ………... @pmichali
> > > GPG Key … 4525ECC253E31A83
> > > Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
> > >
> > >
> > >
> > > On Apr 4, 2014, at 10:44 AM, Paul Michali (pcm) < pcm at cisco.com >
> > > wrote:
> > >
> > >
> > >
> > >
> > > I’d like to get this added to the test-requirements for Neutron.
> > > It is a very
> > > flexible HTTP mock module that works with the Requests package. It
> > > is a
> > > decorator that wraps the Request’s send() method and allows easy
> > > mocking of
> > > responses, etc (w/o using a web server).
> > >
> > > The bug is: https://bugs.launchpad.net/neutron/+bug/1282855
> > >
> > > Initially I had requested both httmock and newer requests, but was
> > > requested
> > > to separate them, so this is to target httmock as it is more
> > > important (to
> > > me :) to get approval,
> > >
> > >
> > > The review request is: https://review.openstack.org/#/c/75296/
> > >
> > > An example of code that would use this:
> > >
> > > https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py
> > > https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/cisco_csr_mock.py
> > >
> > > Looking forward to hearing whether or not we can include this
> > > package into
> > > Juno.
> > >
> > > Thanks in advance!
> > >
> > >
> > > PCM (Paul Michali)
> > >
> > > MAIL …..…. pcm at cisco.com
> > > IRC ……..… pcm_ ( irc.freenode.com )
> > > TW ………... @pmichali
> > > GPG Key … 4525ECC253E31A83
> > > Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
> > >
> > >
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list