Open Stack

Hey everyone,

Based on our conversation in irc yesterday, I’m detailing a few proposals for the way we handle testing. Before that, I want to establish the terminology so we’re all on the same page. In a software project like this, there are generally three types of tests: unit tests, integration tests, end-to-end tests.


End-to-end testing is the full run-through of a service operation as an end-user would interact with it. You pass in your identity parameters, instantiate a service like Swift, and execute an operation. A real HTTP request is sent over the wire, and a real HTTP response from the server is received. In other words, it’s a live network test of all components from end to end. Right now, any time we’re communicating with the server API in our test suite, it’s an end-to-end test. There doesn’t need to be many of these - just enough to test that our API works for end-users. E2E tests will typically be slow (due to network calls) - but this does not matter.


Integration testing is like end-to-end testing except no network connections happen. All it does is test is the integration between modules of the application. So if we want to test Swift operation - we’d instantiate a context object with identity parameters, then instantiate a Swift service object, and then after we’ve done the setup, finally test the operation object. In an integration test, the flow of execution happens like it would an end-to-end test, but all we’re testing is that different components work together. This is useful for ensuring that contracts between interfaces are being satisfied, etc. I don’t think we need to worry about writing these.


Unit testing is very different from both of the above. Instead, you test extremely small “units” of behavior in a particular class. Each test needs to be fully isolated and have only 1 responsibility (i.e. test one unit). The class you’re testing should not collaborate with real objects; instead, you need to pass in mocks. So, if we’re unit testing a Swift operation, instead of using a real service or transport client - we mock them and use the mock objects in our tests. If our tested class invokes methods on this mock, we also need to explicitly define how it does so. For example, if we’re testing a method that calls `$this->client->foo()` internally and expects a response, we need to explicitly tell the mocked client object to return a value when its “foo” method is called. We can also be more granular and strict: we can say that the method should only be called with certain arguments, that the method should be called x number of times. With unit tests, you are defining and testing communication promises. The point of doing this is that you’re testing HOW your object communicates in a precise way. Because they’re isolated, there should be hundreds of unit tests, and they should be very quick.


Here’s an example of how you’d mock for unit tests in phpunit: https://gist.github.com/jamiehannaford/ad7f389466ac5dcafe7a

There was a proposal made that we should just inject a HTTP client that doesn’t make real transactions - but that is not feasible because it goes against what a unit test is. If you were to do this, your tested class would be executing UNMOCKED methods against a REAL object - making as many calls as it wants. It would not be isolated, and for that reason it would not be a unit test. It would be an integration test because you’re forcing your tested class to interact like it would in the wild. Instead, you should mock every collaborator and explicitly define which calls are made against them by your tested class.

There are amazing libraries out there like Prophecy and phpspec which mocking a whole load easier and more natural - but I assume nobody wants to move away from phpunit…


Proposal going forward

So, here are my proposals for our current library:

1. Refactor our unit tests to use mocking instead of real HTTP/network calls. If a class relies on dependencies or collaborators, they will need to be mocked.

2. As services are added (Swift, Nova, Keystone), end-to-end tests are added for each. We’d therefore ensure that our SDK is interacting with the real API as expected.

3. Never use the @depends annotation on unit tests, because it makes them tightly coupled with each other and brittle. A unit test is supposed to be completely autonomous and independent - it should never depend on the output of another test.

4. Use the “setUp” and “tearDown” helper methods to easily set up test fixtures

5. In our source code, we need to make use of Dependency Injection AS MUCH AS POSSIBLE because it’s easier to test. If we don’t (choosing to directly instantiate objects in our code), it introduces tight coupling and is extremely hard to mock.



Does anybody have any major disagreements with my above proposal?


Jamie



Jamie Hannaford
Software Developer III - CH     [experience Fanatical Support]

Tel:    +41434303908
Mob:    +41791009767
        [Rackspace]



Rackspace International GmbH a company registered in the Canton of Zurich, Switzerland (company identification number CH-020.4.047.077-1) whose registered office is at Pfingstweidstrasse 60, 8005 Zurich, Switzerland. Rackspace International GmbH privacy policy can be viewed at www.rackspace.co.uk/legal/swiss-privacy-policy
-
Rackspace Hosting Australia PTY LTD a company registered in the state of Victoria, Australia (company registered number ACN 153 275 524) whose registered office is at Suite 3, Level 7, 210 George Street, Sydney, NSW 2000, Australia. Rackspace Hosting Australia PTY LTD privacy policy can be viewed at www.rackspace.com.au/company/legal-privacy-statement.php
-
Rackspace US, Inc, 5000 Walzem Road, San Antonio, Texas 78218, United States of America
Rackspace US, Inc privacy policy can be viewed at www.rackspace.com/information/legal/privacystatement
-
Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ.
Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy
-
Rackspace Benelux B.V. is a company registered in the Netherlands (company KvK nummer 34276327) whose registered office is at Teleportboulevard 110, 1043 EJ Amsterdam.
Rackspace Benelux B.V privacy policy can be viewed at www.rackspace.nl/juridisch/privacy-policy
-
Rackspace Asia Limited is a company registered in Hong Kong (Company no: 1211294) whose registered office is at 9/F, Cambridge House, Taikoo Place, 979 King's Road, Quarry Bay, Hong Kong.
Rackspace Asia Limited privacy policy can be viewed at www.rackspace.com.hk/company/legal-privacy-statement.php
-
This e-mail message (including any attachments or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse at rackspace.com and delete the original message. Your cooperation is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140522/88ad2b50/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagebd91e8.JPG
Type: image/jpeg
Size: 3124 bytes
Desc: imagebd91e8.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140522/88ad2b50/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imageb655a1.JPG
Type: image/jpeg
Size: 990 bytes
Desc: imageb655a1.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140522/88ad2b50/attachment-0001.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagedf3249.JPG
Type: image/jpeg
Size: 6844 bytes
Desc: imagedf3249.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140522/88ad2b50/attachment-0002.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image0d9919.JPG
Type: image/jpeg
Size: 1074 bytes
Desc: image0d9919.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140522/88ad2b50/attachment-0003.jpe>