Open Stack

Hey all,

Sorry for the length of reply - but I want to provide as much information as possible about this topic.

Instead of enumerating pros and cons, I want to give a bit of context first (about what “feature stories” actually are), and then respond to some common misconceptions about them. This way, the pros/cons of Behat are a bit more substantiated.


Would Behat replace PHPUnit?

No - they’re completely different. We’d still use phpunit for unit testing because it’s way better at xunit-like assertions. We’d use behat instead for functional testing - making sure that features work against a production API.


Who’s using Behat and is it suitable for us?

From what I’ve heard, we’re using it for some projects at Rackspace and possibly some OpenStack projects - but I need to double check that. I’ve reached out to some folks about their experiences with it - so I’ll post the findings when I hear back.


What are BDD feature stories?

Here’s a link to a fantastic article which explains the benefits of BDD feature stories: http://dannorth.net/whats-in-a-story/

tl;dr:

BDD takes the position that you can turn an idea for a requirement into implemented, tested, production-ready code simply and effectively, as long as the requirement is specific enough that everyone knows what’s going on. To do this, we need a way to describe the requirement such that everyone – end-user, contributor, manager, technical lead (in short, anyone interested in using our SDK in their business) – have a common understanding of the scope of the work. You are showing them, in human-readable language, the features of the SDK and what it offers them. The result is that everyone — regardless of proficiency, skill level and familiarity with the codebase — is on the same level of understanding. From this they can agree a common definition of “done”, and we escape the dual gumption traps of “that’s not what I asked for” or “I forgot to tell you about this other thing”.

This, then, is the role of a Story. It is a description of a requirement and a set of criteria by which we all agree that it is “done”. It helps us understand and satisfy customer use-cases in a well expressed and clear way. It also helps us track project progress by having well-established acceptance criteria for feature sets.


3 misconceptions about BDD

(Inspired by http://www.thoughtworks.com/insights/blog/3-misconceptions-about-bdd)

1. End-users don’t care about this! They want code

This is actually a completely misdirected point. The purpose of behat is not to serve as a public-facing repository of sample code. Its actual purpose is twofold: to serve as a functional test suite (i.e. make sure our SDK works against an API), and secondly to serve as a communication device - to codify features in a human-readable way.

It’s the role of documentation to explain the concepts of the SDK with detailed code samples. Another good idea is to provide a “samples” folder that contains standalone scripts for common use-cases - this is what we offer for our current SDK, and users appreciate it. Both of these will allow developers to copy and paste working code for their requirements.

2. Contributors don’t want to write these specifications!

My response is this: how can you implement a piece of functionality if you have no idea what it’s supposed to do? You’re not communicating the scope of the feature, you’re not defining the criteria that makes the feature “done” - you’re just slinging code. Without feature stories, if somebody has a major problem with an approach you’re taking in your development of a feature, you’ll only find out afterwards in Gerrit. You’d end up wasting hours on stuff that does not deliver the feature as expected - it’s a wasteful and inefficient example of miscommunication that could easily be avoided.

If you write the acceptance criteria beforehand - you know exactly what you have to contribute. Everybody knows when the feature will be “done” because you’ve settled all the defined customer use-cases. Writing feature stories is all about collaboration and communication.

3. You can achieve the same with normal code!

Again, this is a misdirected point because comparing them is like comparing apples and oranges - they’re completely different things.

Behat is for communication. The whole reason for using the Gherkin DSL is for its readability. Its aim is to improve communication and help people understand features.  PHP code does not offer this - you’d end up trying to communicate features to people who are not proficient with our codebase, who may not be strong coders, and who might not even know PHP. We can’t assume that people want to traverse a labyrinth of code just to understand how a feature works.

Behat is for defining end-user expectations. Writing normal code doesn’t offer us a way to “test” functionality. With Behat we’re explicitly testing expectations: does this scenario return the right thing? Does it return this value? What about if I was a different user with different data? You’re testing scenarios. How easy would that be using normal PHP code? You’d have hundreds of lines and probably lots of messy exceptions. In short, it's messy and massively inefficient. Behat allows us to avoid this clumsiness and duplication by centralizing this type of expectation checking.


Ugh, another dependency. What happens if we switch to another BDD framework?

Absolutely nothing. The beauty of the Gherkin syntax is that it’s framework independent. This means we could even take the Feature files from another non-PHP project, such as an OpenStack one, and it’d be fine. We’d have to re-configure our context classes (the things which links human-readable DSL into SDK calls), but that’d be it. It’s completely interoperable :)


Jamie


On June 6, 2014 at 12:22:54 PM, Choi, Sam (sam.choi at hp.com<mailto:sam.choi at hp.com>) wrote:
Hello all,
During our most recent php sdk team meeting, a suggestion to use the PHP framework Behat (http://behat.org/) was brought up. I’d like to begin a discussion around the pros/cons of using Behat to create tests. For those not familiar with the php sdk or Behat, we are currently using PHPUnit for all of our tests and are considering Behat for writing human-readable stories that can potentially generate tests that can be run against the sdk.

A couple general questions for all devs

-        Is a similar framework being used for any OpenStack CLI/SDK project? I’d love to see an example.

-        What kind of pros/cons have you all seen when using Behat or similar frameworks for testing?

Jamie

-        Would the suggested Behat framework be intended to supplement our existing PHPUnit tests? In effect, giving us a set of Behat tests and PHPUnit tests. Or was the suggestion to have all tests rely on the Behat framework?

After reviewing Behat and some related technologies, I do have a few concerns at the moment. Any thoughts of the concerns would be appreciated.


-        We are introducing another dependency for the PHP SDK. What happens if there is a reason for us to switch to another BDD framework or ditch it to go back to PHPUnit? It seems like we would likely be faced with significant test refactoring.

-        Contributors will have to be proficient with Behat/BDD in order to write tests for the PHP SDK. This adds to what may already be a steep learning curve for a new contributor who has to learn our code base, Guzzle for the transport layer, how to use various/develop various cloud services etc.

-        For more complicated tests, writing out features, scenarios, step definitions, and testing logic would almost certainly take longer than writing a traditional test in pure PHP.


Thanks,
--
Sam Choi
Hewlett-Packard Co.
HP Cloud Services
+1 650 316 1652 / Office




Jamie Hannaford
Software Developer III - CH     [experience Fanatical Support]

Tel:    +41434303908
Mob:    +41791009767
        [Rackspace]



Rackspace International GmbH a company registered in the Canton of Zurich, Switzerland (company identification number CH-020.4.047.077-1) whose registered office is at Pfingstweidstrasse 60, 8005 Zurich, Switzerland. Rackspace International GmbH privacy policy can be viewed at www.rackspace.co.uk/legal/swiss-privacy-policy
-
Rackspace Hosting Australia PTY LTD a company registered in the state of Victoria, Australia (company registered number ACN 153 275 524) whose registered office is at Suite 3, Level 7, 210 George Street, Sydney, NSW 2000, Australia. Rackspace Hosting Australia PTY LTD privacy policy can be viewed at www.rackspace.com.au/company/legal-privacy-statement.php
-
Rackspace US, Inc, 5000 Walzem Road, San Antonio, Texas 78218, United States of America
Rackspace US, Inc privacy policy can be viewed at www.rackspace.com/information/legal/privacystatement
-
Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ.
Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy
-
Rackspace Benelux B.V. is a company registered in the Netherlands (company KvK nummer 34276327) whose registered office is at Teleportboulevard 110, 1043 EJ Amsterdam.
Rackspace Benelux B.V privacy policy can be viewed at www.rackspace.nl/juridisch/privacy-policy
-
Rackspace Asia Limited is a company registered in Hong Kong (Company no: 1211294) whose registered office is at 9/F, Cambridge House, Taikoo Place, 979 King's Road, Quarry Bay, Hong Kong.
Rackspace Asia Limited privacy policy can be viewed at www.rackspace.com.hk/company/legal-privacy-statement.php
-
This e-mail message (including any attachments or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse at rackspace.com and delete the original message. Your cooperation is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140606/fac4b903/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image82de36.JPG
Type: image/jpeg
Size: 3124 bytes
Desc: image82de36.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140606/fac4b903/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagefb2f83.JPG
Type: image/jpeg
Size: 990 bytes
Desc: imagefb2f83.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140606/fac4b903/attachment-0001.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagee4c271.JPG
Type: image/jpeg
Size: 6844 bytes
Desc: imagee4c271.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140606/fac4b903/attachment-0002.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image5fae94.JPG
Type: image/jpeg
Size: 1074 bytes
Desc: image5fae94.JPG
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140606/fac4b903/attachment-0003.jpe>