Open Stack

----- Original Message -----
> From: "Paul Michali (pcm)" <pcm at cisco.com>
> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
> Sent: Wednesday, April 9, 2014 6:31:09 AM
> Subject: Re: [openstack-dev] [infra]Requesting consideration of httmock package for test-requirements in Juno
> 
> On Apr 8, 2014, at 3:04 PM, Jamie Lennox < jamielennox at redhat.com > wrote:
> 
> 
> 
> 
> 
> 
> ----- Original Message -----
> 
> 
> From: "Paul Michali (pcm)" < pcm at cisco.com >
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org >
> Cc: jamielennox at gmail.com
> Sent: Wednesday, April 9, 2014 12:09:58 AM
> Subject: [openstack-dev] [infra]Requesting consideration of httmock package
> for test-requirements in Juno
> 
> Reposting this, after discussing with Sean Dague…
> 
> For background, I have developed a REST client lib to talk to a H/W device
> with REST server for VPNaaS in Neutron. To support unit testing of this, I
> created a UT module and a mock REST server module and used the httmock
> package. I found it easy to use, and was able to easily create a sub-class
> of my UT to run the same test cases with real H/W, instead of the mock REST
> server. See the original email below, for links of the UT and REST mock to
> see how I used it.
> 
> 
> I created a bug under requirements, to propose adding httmock to the
> test-requirements. Sean mentioned that there is an existing mock package,
> called httpretty , which I found is used in keystone client UTs), and should
> petition to see if httmock should replace httpretty, since the two appear to
> overlap in functionality.
> 
> I found this link, with a brief comparison of the two:
> http://marekbrzoska.wordpress.com/2013/08/28/mocking-http-requests-in-python/
> 
> So… I’m wondering if the community is interested in adopting this package
> (with the goal of deprecating the httpretty package). Otherwise, I will work
> on reworking the UT code I have to try to use httpretty.
> 
> Would be interested in peoples’ thoughts, especially those who have worked
> with httpretty.
> 
> Thanks in advance!
> 
> So I introduced HTTPretty into the requirements and did the work around
> keystoneclient and am well aware that it has a few warts.
> 
> PCM: Great, I grabbed your name from keystone client logs and was hoping you
> had some knowledge of httpretty.
> 
> 
> 
> 
> 
> 
> At the time we were going through the changeover from httplib to requests and
> httpretty gave a good way to change over the library and ensure that we
> hadn't actually changed the issued requests at all. If we had already been
> on requests i don't know if i'd have made the same choice.
> 
> In general I am in favour of mocking the response layer rather than the
> client layer - whether we do this with httpretty or httmock doesn't bother
> me that much. Honestly I don't think a global patch of the requests Session
> object is that much safer that a global patch of the socket interface, if
> anything requests is under development and so this interface is less
> defined.
> 
> PCM: Not sure that httmock can be considered a global patch. It is a context
> lib that intercepts the call through various decorators where the request
> can be filtered/processed and if not, will fall through and call the actual
> library.
> 
> So, with the context lib, you can define several handlers for the request(s).
> When the call is made, it will try each handler and if they all return None,
> will call the original function, otherwise they return the value of the mock
> routine. Here’s an example front he test cases I cerated:
> 
> with httmock.HTTMock(csr_request.token, csr_request.put,
> csr_request.normal_get):
> keepalive_info = {'interval': 60, 'retry': 4}
> self.csr.configure_ike_keepalive (keepalive_info)
> self.assertEqual(requests.codes.NO_CONTENT, self.csr.status)
> content = self.csr.get_request ('vpn-svc/ike/keepalive')
> self.assertEqual(requests.codes.OK, self.csr.status)
> expected = {'periodic': False}
> expected.update(keepalive_info)
> self.assertDictContainsSubset(expected, content)
> 
> The client code (red) does a POST with authentication info to get token, does
> a PUT with the setting, and then a GET to verify the value. The mock module
> has these methods created:
> 
> @httmock.urlmatch ( netloc =r'localhost')
> def token( url , request):
> if ' auth /token-services' in url.path:
> return {'status_code': requests.codes.OK,
> 'content': {'token-id': 'dummy-token'}}
> 
> 
> @httmock.urlmatch ( netloc =r'localhost')
> def normal_get( url , request):
> if request.method != 'GET':
> return
> if not request.headers.get('X- auth -token', None):
> return {'status_code': requests.codes.UNAUTHORIZED}
> …
> if 'vpn-svc/ike/keepalive' in url.path:
> content = {u'interval': 60,
> u'retry': 4,
> u'periodic': True}
> return httmock.response(requests.codes.OK, content=content)
> 
> @httmock.urlmatch(netloc=r'localhost')
> def put(url, request):
> if request.method != 'PUT':
> return
> if not request.headers.get('X-auth-token', None):
> return {'status_code': requests.codes.UNAUTHORIZED}
> return {'status_code': requests.codes.NO_CONTENT}
> 
> 
> Just a few notes….
> A) Could have created separate context lib for put vs get.
> B) Could have a method for specific URI request matches (I do that in some
> places, in others I catch all requests).
> C) Can filter the requests based on request type or URI.
> D) Can catch all URLs or filter.
> E) Additional decorators can be created for these handlers.
> F) There is lots of flexibility with manipulating the response data.
> 
> For (C) I’ve done things like this:
> 
> @filter_request(['post'], 'vpn-svc/site-to-site')
> @httmock.urlmatch(netloc=r'localhost')
> def post_missing_ipsec_policy(url, request):
> if not request.headers.get('X-auth-token', None):
> return {'status_code': requests.codes.UNAUTHORIZED}
> return {'status_code': requests.codes.BAD_REQUEST}
> 
> For (D), I have all my handlers set for localhost, and then I have another
> test module that creates a subclass of the test class, sets the host to an
> IP of a live system, and runs the same tests, only this time against a live
> router, instead the mock module. This saves a lot of coding and allows me to
> reuse the test code (and make sure it REALLY works with real hardware, in my
> case).
> 
> For (E), I did this to simulate a timeout in a request and then latter
> success on retry:
> 
> @filter_request (['get'], 'global/host-name')
> @repeat(1)
> @httmock.urlmatch ( netloc =r'localhost')
> def expired_request( url , request):
> """Simulate access denied failure on first request for this resource.
> 
> Intent here is to simulate that the token has expired, by failing
> the first request to the resource. Because of the repeat=1, this
> will only be called once, and subsequent calls will not be handled
> by this function, but instead will access the normal handler and
> will pass. Currently configured for a GET request, but will work
> with POST and PUT as well. For DELETE, would need to filter_request on a
> different resource (e.g. 'global/local-users')
> """
> 
> return {'status_code': requests.codes.UNAUTHORIZED}
> 
> Ref:
> https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py
> Ref:
> https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/cisco_csr_mock.py
> 

The reason that i call this a global mock is that behind the context manager it is still patching the class method not an individual instance of a requests.Session object. see: https://github.com/patrys/httmock/blob/master/httmock.py#L142 

Everything you have specified is doable with httpretty.

@httpretty.activate does the mocking of the library (socket rather than requests.Session) then you use 

    httpretty.register_uri(httpretty.GET, 
                           'http://testurl/testpath', 
                           body=json.dumps({'hello': 'world'},
                           status=200)

You can do the same multiple queued responses, responses via callbck and everything else. Also the thing i find a use a lot that i don't see provided by httmock is things like: 

    self.assertEqual(httpretty.last_request().headers['X-Auth-Token'],
                     expected_token)

In which you test that the request that you send is actually what you expect as well. 

Anyway there is a pretty long readme on the github page: https://github.com/gabrielfalcao/HTTPretty that explains these sort of features.

 
> What i would like to see though is this mocking transferred into fixtures
> like in https://review.openstack.org/#/c/77961/ and have the actual choice
> in mock library hidden behind those fixtures. Is this a pattern that httmock
> can handle? or something else again?
> 
> PCM I see your point - it would be nice to be able to have the flexibility to
> swap the underlying mock libraries. Would have to think about it more, to
> see if a common fixture can be used for httpretty and httmock. I’m not sure
> if they are different enough (one monkey patching, one using context lib)
> that it would be possible (and I don’t have a strong understanding of
> httpretty yet).
> 
> Would love to hear peoples’ thoughts no this.

Again, i'm not that strongly in favour of HTTPretty over httmock (though missing last_request will be a problem). My point is just that the two libraries can both do the same job and i'm wondering what the advantage of the new one is.

As mentioned i would really like to see this moved into fixtures - httpretty will allow that somewhat, i don't see that httmock will, and if i had the time i think i would just rewrite a library like this making use of the tools we already have like mock, fixtures, and testresources. (My other thought is to just subclass a requests.Session object with an overridden request method -rather than a global patch- and allow that to be passed into clients but i don't think they all support that).


Jamie 



> 
> PCM (Paul Michali)
> 
> MAIL …..…. pcm at cisco.com
> IRC ……..… pcm_ ( irc.freenode.com )
> TW ………... @pmichali
> GPG Key … 4525ECC253E31A83
> Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
> 
> 
> 
> 
> 
> 
> 
> 
> Jamie
> 
> 
> 
> 
> PCM (Paul Michali)
> 
> MAIL …..…. pcm at cisco.com
> IRC ……..… pcm_ ( irc.freenode.com )
> TW ………... @pmichali
> GPG Key … 4525ECC253E31A83
> Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
> 
> 
> 
> On Apr 4, 2014, at 10:44 AM, Paul Michali (pcm) < pcm at cisco.com > wrote:
> 
> 
> 
> 
> I’d like to get this added to the test-requirements for Neutron. It is a very
> flexible HTTP mock module that works with the Requests package. It is a
> decorator that wraps the Request’s send() method and allows easy mocking of
> responses, etc (w/o using a web server).
> 
> The bug is: https://bugs.launchpad.net/neutron/+bug/1282855
> 
> Initially I had requested both httmock and newer requests, but was requested
> to separate them, so this is to target httmock as it is more important (to
> me :) to get approval,
> 
> 
> The review request is: https://review.openstack.org/#/c/75296/
> 
> An example of code that would use this:
> 
> https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py
> https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/cisco_csr_mock.py
> 
> Looking forward to hearing whether or not we can include this package into
> Juno.
> 
> Thanks in advance!
> 
> 
> PCM (Paul Michali)
> 
> MAIL …..…. pcm at cisco.com
> IRC ……..… pcm_ ( irc.freenode.com )
> TW ………... @pmichali
> GPG Key … 4525ECC253E31A83
> Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>