<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
See inline @PCM…
<div><br>
</div>
<div>On Apr 9, 2014, at 5:56 PM, Jamie Lennox <<a href="mailto:jamielennox@redhat.com">jamielennox@redhat.com</a>> wrote:<br>
<div apple-content-edited="true">
<div>
<div><br>
</div>
</div>
</div>
<div>
<blockquote type="cite"><br>
<br>
----- Original Message -----<br>
<blockquote type="cite">From: "Paul Michali (pcm)" <<a href="mailto:pcm@cisco.com">pcm@cisco.com</a>><br>
To: "OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
Sent: Wednesday, April 9, 2014 6:31:09 AM<br>
Subject: Re: [openstack-dev] [infra]Requesting consideration of httmock package for test-requirements in Juno<br>
<br>
On Apr 8, 2014, at 3:04 PM, Jamie Lennox < <a href="mailto:jamielennox@redhat.com">
jamielennox@redhat.com</a> > wrote:<br>
<br>
<br>
<br>
<br>
<br>
<br>
----- Original Message -----<br>
<br>
<br>
From: "Paul Michali (pcm)" < <a href="mailto:pcm@cisco.com">pcm@cisco.com</a> ><br>
To: "OpenStack Development Mailing List (not for usage questions)" <<br>
<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a> ><br>
Cc: <a href="mailto:jamielennox@gmail.com">jamielennox@gmail.com</a><br>
Sent: Wednesday, April 9, 2014 12:09:58 AM<br>
Subject: [openstack-dev] [infra]Requesting consideration of httmock package<br>
for test-requirements in Juno<br>
<br>
Reposting this, after discussing with Sean Dague…<br>
<br>
For background, I have developed a REST client lib to talk to a H/W device<br>
with REST server for VPNaaS in Neutron. To support unit testing of this, I<br>
created a UT module and a mock REST server module and used the httmock<br>
package. I found it easy to use, and was able to easily create a sub-class<br>
of my UT to run the same test cases with real H/W, instead of the mock REST<br>
server. See the original email below, for links of the UT and REST mock to<br>
see how I used it.<br>
<br>
<br>
I created a bug under requirements, to propose adding httmock to the<br>
test-requirements. Sean mentioned that there is an existing mock package,<br>
called httpretty , which I found is used in keystone client UTs), and should<br>
petition to see if httmock should replace httpretty, since the two appear to<br>
overlap in functionality.<br>
<br>
I found this link, with a brief comparison of the two:<br>
<a href="http://marekbrzoska.wordpress.com/2013/08/28/mocking-http-requests-in-python/">http://marekbrzoska.wordpress.com/2013/08/28/mocking-http-requests-in-python/</a><br>
<br>
So… I’m wondering if the community is interested in adopting this package<br>
(with the goal of deprecating the httpretty package). Otherwise, I will work<br>
on reworking the UT code I have to try to use httpretty.<br>
<br>
Would be interested in peoples’ thoughts, especially those who have worked<br>
with httpretty.<br>
<br>
Thanks in advance!<br>
<br>
So I introduced HTTPretty into the requirements and did the work around<br>
keystoneclient and am well aware that it has a few warts.<br>
<br>
PCM: Great, I grabbed your name from keystone client logs and was hoping you<br>
had some knowledge of httpretty.<br>
<br>
<br>
<br>
<br>
<br>
<br>
At the time we were going through the changeover from httplib to requests and<br>
httpretty gave a good way to change over the library and ensure that we<br>
hadn't actually changed the issued requests at all. If we had already been<br>
on requests i don't know if i'd have made the same choice.<br>
<br>
In general I am in favour of mocking the response layer rather than the<br>
client layer - whether we do this with httpretty or httmock doesn't bother<br>
me that much. Honestly I don't think a global patch of the requests Session<br>
object is that much safer that a global patch of the socket interface, if<br>
anything requests is under development and so this interface is less<br>
defined.<br>
<br>
PCM: Not sure that httmock can be considered a global patch. It is a context<br>
lib that intercepts the call through various decorators where the request<br>
can be filtered/processed and if not, will fall through and call the actual<br>
library.<br>
<br>
So, with the context lib, you can define several handlers for the request(s).<br>
When the call is made, it will try each handler and if they all return None,<br>
will call the original function, otherwise they return the value of the mock<br>
routine. Here’s an example front he test cases I cerated:<br>
<br>
with httmock.HTTMock(csr_request.token, csr_request.put,<br>
csr_request.normal_get):<br>
keepalive_info = {'interval': 60, 'retry': 4}<br>
self.csr.configure_ike_keepalive (keepalive_info)<br>
self.assertEqual(requests.codes.NO_CONTENT, self.csr.status)<br>
content = self.csr.get_request ('vpn-svc/ike/keepalive')<br>
self.assertEqual(requests.codes.OK, self.csr.status)<br>
expected = {'periodic': False}<br>
expected.update(keepalive_info)<br>
self.assertDictContainsSubset(expected, content)<br>
<br>
The client code (red) does a POST with authentication info to get token, does<br>
a PUT with the setting, and then a GET to verify the value. The mock module<br>
has these methods created:<br>
<br>
@httmock.urlmatch ( netloc =r'localhost')<br>
def token( url , request):<br>
if ' auth /token-services' in url.path:<br>
return {'status_code': requests.codes.OK,<br>
'content': {'token-id': 'dummy-token'}}<br>
<br>
<br>
@httmock.urlmatch ( netloc =r'localhost')<br>
def normal_get( url , request):<br>
if request.method != 'GET':<br>
return<br>
if not request.headers.get('X- auth -token', None):<br>
return {'status_code': requests.codes.UNAUTHORIZED}<br>
…<br>
if 'vpn-svc/ike/keepalive' in url.path:<br>
content = {u'interval': 60,<br>
u'retry': 4,<br>
u'periodic': True}<br>
return httmock.response(requests.codes.OK, content=content)<br>
<br>
@httmock.urlmatch(netloc=r'localhost')<br>
def put(url, request):<br>
if request.method != 'PUT':<br>
return<br>
if not request.headers.get('X-auth-token', None):<br>
return {'status_code': requests.codes.UNAUTHORIZED}<br>
return {'status_code': requests.codes.NO_CONTENT}<br>
<br>
<br>
Just a few notes….<br>
A) Could have created separate context lib for put vs get.<br>
B) Could have a method for specific URI request matches (I do that in some<br>
places, in others I catch all requests).<br>
C) Can filter the requests based on request type or URI.<br>
D) Can catch all URLs or filter.<br>
E) Additional decorators can be created for these handlers.<br>
F) There is lots of flexibility with manipulating the response data.<br>
<br>
For (C) I’ve done things like this:<br>
<br>
@filter_request(['post'], 'vpn-svc/site-to-site')<br>
@httmock.urlmatch(netloc=r'localhost')<br>
def post_missing_ipsec_policy(url, request):<br>
if not request.headers.get('X-auth-token', None):<br>
return {'status_code': requests.codes.UNAUTHORIZED}<br>
return {'status_code': requests.codes.BAD_REQUEST}<br>
<br>
For (D), I have all my handlers set for localhost, and then I have another<br>
test module that creates a subclass of the test class, sets the host to an<br>
IP of a live system, and runs the same tests, only this time against a live<br>
router, instead the mock module. This saves a lot of coding and allows me to<br>
reuse the test code (and make sure it REALLY works with real hardware, in my<br>
case).<br>
<br>
For (E), I did this to simulate a timeout in a request and then latter<br>
success on retry:<br>
<br>
@filter_request (['get'], 'global/host-name')<br>
@repeat(1)<br>
@httmock.urlmatch ( netloc =r'localhost')<br>
def expired_request( url , request):<br>
"""Simulate access denied failure on first request for this resource.<br>
<br>
Intent here is to simulate that the token has expired, by failing<br>
the first request to the resource. Because of the repeat=1, this<br>
will only be called once, and subsequent calls will not be handled<br>
by this function, but instead will access the normal handler and<br>
will pass. Currently configured for a GET request, but will work<br>
with POST and PUT as well. For DELETE, would need to filter_request on a<br>
different resource (e.g. 'global/local-users')<br>
"""<br>
<br>
return {'status_code': requests.codes.UNAUTHORIZED}<br>
<br>
Ref:<br>
https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py<br>
Ref:<br>
https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/cisco_csr_mock.py<br>
<br>
</blockquote>
<br>
The reason that i call this a global mock is that behind the context manager it is still patching the class method not an individual instance of a requests.Session object. see:
<a href="https://github.com/patrys/httmock/blob/master/httmock.py#L142">https://github.com/patrys/httmock/blob/master/httmock.py#L142</a>
<br>
</blockquote>
<div><br>
</div>
@PCM Good point. I forgot that it did that.</div>
<div><br>
</div>
<div><br>
<blockquote type="cite"><br>
Everything you have specified is doable with httpretty.<br>
<br>
@httpretty.activate does the mocking of the library (socket rather than requests.Session) then you use
<br>
<br>
httpretty.register_uri(httpretty.GET, <br>
'<a href="http://testurl/testpath'">http://testurl/testpath'</a>,
<br>
body=json.dumps({'hello': 'world'},<br>
status=200)<br>
<br>
You can do the same multiple queued responses, responses via callbck and everything else. Also the thing i find a use a lot that i don't see provided by httmock is things like:
<br>
<br>
self.assertEqual(httpretty.last_request().headers['X-Auth-Token'],<br>
expected_token)<br>
<br>
In which you test that the request that you send is actually what you expect as well.
<br>
</blockquote>
<div><br>
</div>
@PCM So this is testing that what was sent to the code under test is what it sends out in the HTTP request? I’m not sure I see the advantage in testing that. I usually just test that the response is correct, given the request data I send to the code under test.</div>
<div><br>
</div>
<div> <br>
<blockquote type="cite"><br>
Anyway there is a pretty long readme on the github page: <a href="https://github.com/gabrielfalcao/HTTPretty">
https://github.com/gabrielfalcao/HTTPretty</a> that explains these sort of features.<br>
</blockquote>
<div><br>
</div>
@PCM Thanks. I’ll take a look at this.</div>
<div><br>
</div>
<div><br>
<blockquote type="cite"><br>
<br>
<blockquote type="cite">What i would like to see though is this mocking transferred into fixtures<br>
like in <a href="https://review.openstack.org/#/c/77961/">https://review.openstack.org/#/c/77961/</a> and have the actual choice<br>
in mock library hidden behind those fixtures. Is this a pattern that httmock<br>
can handle? or something else again?<br>
<br>
PCM I see your point - it would be nice to be able to have the flexibility to<br>
swap the underlying mock libraries. Would have to think about it more, to<br>
see if a common fixture can be used for httpretty and httmock. I’m not sure<br>
if they are different enough (one monkey patching, one using context lib)<br>
that it would be possible (and I don’t have a strong understanding of<br>
httpretty yet).<br>
<br>
Would love to hear peoples’ thoughts no this.<br>
</blockquote>
<br>
Again, i'm not that strongly in favour of HTTPretty over httmock (though missing last_request will be a problem).
</blockquote>
<div><br>
</div>
<div>
<div>@PCM I’ll take a look and see if I can change all my UTs to use HTTPretty. I’ll be curious to see if it can handle all the cases I use. In particular, I have two test modules, one which has test cases that, by using a URL directed to the localhost, makes
use of a mock REST server, to exercise my client REST module.</div>
<div><br>
</div>
<div>I have a second test module, which has subclasses for the test classes in the first module, only in setup, it specifies the URL of a “real” REST server. When I run this module, which is very small, it leverages off all the test cases, but verifies with
a real device, which is the true proof that the REST client under test actually works. Granted, it wouldn’t be used during a UT TOX run, but it could be used with a Tempest run.</div>
<div><br>
</div>
<div>From an httmock standpoint there is no change, as the handler methods have filters to only catch requests to a URL directed to localhost. The “live” test will end up falling through and making a REST request to a real server.</div>
<div><br>
</div>
<div>Likewise, I have tests for checking timeouts, where I have the desired call map to the mock, so the request fails, but other requests flow through to the live server.</div>
<div><br>
</div>
<div>Any thoughts on how I would be able to do these things with HTTPretty?</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<blockquote type="cite">My point is just that the two libraries can both do the same job and i'm wondering what the advantage of the new one is.<br>
</blockquote>
<div><br>
</div>
</div>
<div>@PCM One thing is that I think the context manager technique of handling the mocking is (subjectively) a really clean way to do things. The other is this duality of calls passing through to a real (non-mock) server without altering the test cases.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<blockquote type="cite"><br>
As mentioned i would really like to see this moved into fixtures - httpretty will allow that somewhat, i don't see that httmock will,</blockquote>
<div><br>
</div>
@PCM My <b>guess</b> is that probably either method could be wrapped by a fixture, but I’m also guessing that they are different enough such that it may be very hard to come up with a fixture to accommodate both.</div>
<div><br>
</div>
<div><br>
<blockquote type="cite">and if i had the time i think i would just rewrite a library like this making use of the tools we already have like mock, fixtures, and testresources. (My other thought is to just subclass a requests.Session object with an overridden
request method -rather than a global patch- and allow that to be passed into clients but i don't think they all support that).<br>
</blockquote>
<div><br>
</div>
@PCM I’ll try to see if I can adapt my UTs to use HTTPretty and think a little bit on how a test fixture could be used.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>
<div>
<div>
<div>PCM (Paul Michali)</div>
<div><br>
</div>
<div>MAIL …..…. <a href="mailto:pcm@cisco.com">pcm@cisco.com</a></div>
<div>IRC ……..… pcm_ (<a href="http://irc.freenode.com">irc.freenode.com</a>)</div>
<div>TW ………... @pmichali</div>
<div>GPG Key … 4525ECC253E31A83</div>
<div>Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83</div>
</div>
<div><br>
</div>
<br class="Apple-interchange-newline">
</div>
<blockquote type="cite"><br>
<br>
Jamie <br>
<br>
<br>
<br>
<blockquote type="cite"><br>
PCM (Paul Michali)<br>
<br>
MAIL …..…. <a href="mailto:pcm@cisco.com">pcm@cisco.com</a><br>
IRC ……..… pcm_ ( <a href="http://irc.freenode.com">irc.freenode.com</a> )<br>
TW ………... @pmichali<br>
GPG Key … 4525ECC253E31A83<br>
Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Jamie<br>
<br>
<br>
<br>
<br>
PCM (Paul Michali)<br>
<br>
MAIL …..…. <a href="mailto:pcm@cisco.com">pcm@cisco.com</a><br>
IRC ……..… pcm_ ( <a href="http://irc.freenode.com">irc.freenode.com</a> )<br>
TW ………... @pmichali<br>
GPG Key … 4525ECC253E31A83<br>
Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83<br>
<br>
<br>
<br>
On Apr 4, 2014, at 10:44 AM, Paul Michali (pcm) < <a href="mailto:pcm@cisco.com">
pcm@cisco.com</a> > wrote:<br>
<br>
<br>
<br>
<br>
I’d like to get this added to the test-requirements for Neutron. It is a very<br>
flexible HTTP mock module that works with the Requests package. It is a<br>
decorator that wraps the Request’s send() method and allows easy mocking of<br>
responses, etc (w/o using a web server).<br>
<br>
The bug is: <a href="https://bugs.launchpad.net/neutron/+bug/1282855">https://bugs.launchpad.net/neutron/+bug/1282855</a><br>
<br>
Initially I had requested both httmock and newer requests, but was requested<br>
to separate them, so this is to target httmock as it is more important (to<br>
me :) to get approval,<br>
<br>
<br>
The review request is: <a href="https://review.openstack.org/#/c/75296/">https://review.openstack.org/#/c/75296/</a><br>
<br>
An example of code that would use this:<br>
<br>
<a href="https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py">https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py</a><br>
https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/cisco_csr_mock.py<br>
<br>
Looking forward to hearing whether or not we can include this package into<br>
Juno.<br>
<br>
Thanks in advance!<br>
<br>
<br>
PCM (Paul Michali)<br>
<br>
MAIL …..…. pcm@cisco.com<br>
IRC ……..… pcm_ ( irc.freenode.com )<br>
TW ………... @pmichali<br>
GPG Key … 4525ECC253E31A83<br>
Fingerprint .. 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83<br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org<br>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org<br>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org<br>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org<br>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br>
<br>
</blockquote>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br>
</blockquote>
</div>
<br>
</div>
</body>
</html>