[openstack-dev] [openstack-php-sdk] Testing proposal

Matthew Farina matt at mattfarina.com
Wed May 28 18:10:46 UTC 2014


Jamie, thanks for taking the time to detail out your proposal.

I have no major disagreements. The proposal looks fine to me.


On Thu, May 22, 2014 at 6:21 AM, Jamie Hannaford <
jamie.hannaford at rackspace.com> wrote:

>  Hey everyone,
>
>  Based on our conversation in irc yesterday, I’m detailing a few
> proposals for the way we handle testing. Before that, I want to establish
> the terminology so we’re all on the same page. In a software project like
> this, there are generally three types of tests: unit tests, integration
> tests, end-to-end tests.
>
>
>  *End-to-end testing* is the full run-through of a service operation as
> an end-user would interact with it. You pass in your identity parameters,
> instantiate a service like Swift, and execute an operation. A real HTTP
> request is sent over the wire, and a real HTTP response from the server is
> received. In other words, it’s a live network test of *all components *from
> end to end. Right now, any time we’re communicating with the server API in
> our test suite, it’s an *end-to-end test*. There doesn’t need to be many
> of these - just enough to test that our API works for end-users. E2E tests
> will typically be slow (due to network calls) - but this does not matter.
>
>
>  *Integration testing *is like end-to-end testing except no network
> connections happen. All it does is test is the *integration *between
> modules of the application. So if we want to test Swift operation - we’d
> instantiate a context object with identity parameters, then instantiate a
> Swift service object, and then after we’ve done the setup, finally test the
> operation object. In an integration test, the flow of execution happens
> like it would an end-to-end test, but all we’re testing is that different
> components work together. This is useful for ensuring that contracts
> between interfaces are being satisfied, etc. I don’t think we need to worry
> about writing these.
>
>
>  *Unit testing* is very different from both of the above. Instead, you
> test extremely small “units” of behavior in a particular class. Each test
> needs to be fully isolated and have only 1 responsibility (i.e. test one
> unit). The class you’re testing should not collaborate with real objects;
> instead, you need to pass in mocks. So, if we’re unit testing a Swift
> operation, instead of using a real service or transport client - we mock
> them and use the mock objects in our tests. If our tested class invokes
> methods on this mock, we also need to explicitly define how it does so. For
> example, if we’re testing a method that calls `$this->client->foo()`
> internally and expects a response, we need to explicitly tell the mocked
> client object to return a value when its “foo” method is called. We can
> also be more granular and strict: we can say that the method should only be
> called with certain arguments, that the method should be called x number of
> times. With unit tests, you are defining and testing communication
> promises. The point of doing this is that you’re testing HOW your object
> communicates in a precise way. Because they’re isolated, there should be
> hundreds of unit tests, and they should be very quick.
>
>
>  Here’s an example of how you’d mock for unit tests in phpunit:
> https://gist.github.com/jamiehannaford/ad7f389466ac5dcafe7a
>
>  There was a proposal made that we should just inject a HTTP client that
> doesn’t make real transactions - but that is not feasible because it goes
> against what a unit test is. If you were to do this, your tested class
> would be executing UNMOCKED methods against a REAL object - making as many
> calls as it wants. It would not be isolated, and for that reason it would
> not be a unit test. It would be an *integration test* because you’re
> forcing your tested class to interact like it would in the wild. Instead,
> you should mock every collaborator and explicitly define which calls are
> made against them by your tested class.
>
>  There are amazing libraries out there like Prophecy and phpspec which
> mocking a whole load easier and more natural - but I assume nobody wants to
> move away from phpunit…
>
>
>  *Proposal going forward*
>
>  So, here are my proposals for our current library:
>
>  1. Refactor our unit tests to use mocking instead of real HTTP/network
> calls. If a class relies on dependencies or collaborators, they will need
> to be mocked.
>
>  2. As services are added (Swift, Nova, Keystone), end-to-end tests are
> added for each. We’d therefore ensure that our SDK is interacting with the
> real API as expected.
>
>  3. Never use the @depends annotation on unit tests, because it makes
> them tightly coupled with each other and brittle. A unit test is supposed
> to be completely autonomous and independent - it should never depend on the
> output of another test.
>
>  4. Use the “setUp” and “tearDown” helper methods to easily set up test
> fixtures
>
>  5. In our source code, we need to make use of Dependency Injection AS
> MUCH AS POSSIBLE because it’s easier to test. If we don’t (choosing to
> directly instantiate objects in our code), it introduces tight coupling and
> is extremely hard to mock.
>
>
>
>  Does anybody have any major disagreements with my above proposal?
>
>
>  Jamie
>
>
>
>   Jamie Hannaford
> Software Developer III - CH [image: experience Fanatical Support] [image:
> LINE] Tel: +41434303908Mob: +41791009767 [image: Rackspace]
>
>
>
> Rackspace International GmbH a company registered in the Canton of Zurich,
> Switzerland (company identification number CH-020.4.047.077-1) whose
> registered office is at Pfingstweidstrasse 60, 8005 Zurich, Switzerland.
> Rackspace International GmbH privacy policy can be viewed at
> www.rackspace.co.uk/legal/swiss-privacy-policy
> -
> Rackspace Hosting Australia PTY LTD a company registered in the state of
> Victoria, Australia (company registered number ACN 153 275 524) whose
> registered office is at Suite 3, Level 7, 210 George Street, Sydney, NSW
> 2000, Australia. Rackspace Hosting Australia PTY LTD privacy policy can be
> viewed at www.rackspace.com.au/company/legal-privacy-statement.php
> -
> Rackspace US, Inc, 5000 Walzem Road, San Antonio, Texas 78218, United
> States of America
> Rackspace US, Inc privacy policy can be viewed at
> www.rackspace.com/information/legal/privacystatement
> -
> Rackspace Limited is a company registered in England & Wales (company
> registered number 03897010) whose registered office is at 5 Millington
> Road, Hyde Park Hayes, Middlesex UB3 4AZ.
> Rackspace Limited privacy policy can be viewed at
> www.rackspace.co.uk/legal/privacy-policy
> -
> Rackspace Benelux B.V. is a company registered in the Netherlands (company
> KvK nummer 34276327) whose registered office is at Teleportboulevard 110,
> 1043 EJ Amsterdam.
> Rackspace Benelux B.V privacy policy can be viewed at
> www.rackspace.nl/juridisch/privacy-policy
> -
> Rackspace Asia Limited is a company registered in Hong Kong (Company no:
> 1211294) whose registered office is at 9/F, Cambridge House, Taikoo Place,
> 979 King's Road, Quarry Bay, Hong Kong.
> Rackspace Asia Limited privacy policy can be viewed at
> www.rackspace.com.hk/company/legal-privacy-statement.php
> -
> This e-mail message (including any attachments or embedded documents) is
> intended for the exclusive and confidential use of the individual or entity
> to which this message is addressed, and unless otherwise expressly
> indicated, is confidential and privileged information of Rackspace. Any
> dissemination, distribution or copying of the enclosed material is
> prohibited. If you receive this transmission in error, please notify us
> immediately by e-mail at abuse at rackspace.com and delete the original
> message. Your cooperation is appreciated.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140528/adefe2f0/attachment.html>


More information about the OpenStack-dev mailing list