Testing boundaries
September 24, 2012 — 6 years agoSometimes we need to test our interaction with a certain dependency, but doing so proves difficult. But using a certain trick, we can get nearly the same assurance we're seeking without all the hassle.
The problem
Let's say you want to test that when you lock your bedroom door, it cannot be opened from the outside. The problem is that there's only one door, and you're all by yourself. So if you actually step outside the door, lock it, and shut it, you'll be locked out of your own bedroom. Getting back in may be difficult, expensive or even impossible.
One approach to testing this could be to stand on the inside, lock it, and try to pull it open. But first of all that won't work with many types of locks, which automatically unlock themselves when turned from the inside. But more importantly, this doesn't give us the assurance that it can't be opened from the outside when locked, which is precisely what we want to test.
This isn't that different from testing software. Sometimes we want to test an interaction with something like a database or the file system. If we tried to use the real thing in our tests, it would likely be problematic. Setting up the environment for tests and ensuring it's taken down correctly is tricky and often slows down tests. Plus it can easily become a bottleneck to making the test suite portable between different machines.
The solution
The key is to make use of known principles, and test around them instead of through them.
Let's look at our door example. We know that whether the handle can be turned is independent of whether the door is closed. So while the door is open, we can test that the outside handle can't be turned when the lock is enabled. This gives us assurance that the lock makes the handle unable to be turned from the outside.
We already have half our assurance. The only behavior left to test is that when the door is pushed at from the outside, it doesn't open. We can test this with the door unlocked so we don't lock ourselves out, since we've just proven that it's impossible to turn the handle when it's locked anyway. So pushing at the door is equivalent to trying to open it properly when locked.
These are the only two tests we need to give us nearly complete assurance that nobody can open the door from the outside when it's locked. It may not be literally the same as locking yourself out, but it's as close as we'll get without having a locksmith on call. All we had to do was design the right tests taking into account the relevant principles at work.
In practice
Let's say we want to test that our object-under-test searches our database for any names beginning with "Bob"
. We don't need to load up a database and fire up a real query just to generate this kind of assurance. Instead we can rely on the documentation of our database which indicates that a query using the =
operator may use wildcards (asterisks) in the string.
Our first test would state that our object-under-test calls the function FindNamesBeginningWith()
passing in the string "Bob"
. Our second test states that this function sends a SQL query to the database containing "WHERE NAME = '?*'"
and the string argument. (In real life each database has it's own unique API for doing this, but that's beside the point.)
One extra benefit of this technique is that if either behavior breaks in our production code, only one test will fail. For example if we call the function just right, but the function doesn't generate the right SQL query, our test that says we call the function will still pass. If we were to use the real database throughout our test suite, then every test whose execution path included that function would also fail, disguising the real source of the problem with an avalanche of mysteriously failing tests with potentially obfuscating output.
Caveats
There are a few caveats with this technique. For one thing, these principles are implied, so they're virtually hidden. If we were to apply this principle in some tests, either the name of the test or the comments should indicate the principle assumed to be at work in the test. Otherwise the tests become fragile. Communication between teammates is the key here, especially some kind of longer-term communication.
Another caveat is that, unlike with physics, the laws of a given software library may not match up with another library, or even a future version of the selfsame library. In the above example, this test might pass with MySQL but fail if we switched to PostgreSQL. Or, it may pass with MySQL 2, but tomorrow a teammate may upgrade the system to MySQL 3, and although your tests would pass, the functionality would break in production.
This technique isn't a panacea, and it may be the wrong solution in many circumstances. But as with all tools in our generalized tool-belts, it has a time and place, and you may one day find this is the best solution for the task at hand. On that day, I hope this analogy serves well as an alternative perspective when deciding what kind of tests to write.
About me

My name is Steven Degutis, and I've been writing software professionally for a decade. During that time, I've written many apps and websites, quite a few technical articles, and kept up-to-date with the rapidly evolving software industry.
If you have software needs for web, mobile, or desktop, and are looking for a seasoned software professional, please reach out to me at sbdegutis@gmail.com to set up a phone call.
Work Experience
- Self-employed – present
- Clean Coders – 5 years
- 8th Light – 2 years
- Big Nerd Ranch – 1 year
- Self-employed - 1 year
Platforms
- Web: full-stack
- iOS (UIKit)
- macOS (Cocoa)
- REST APIs
- AWS / EC2 / ELB
Languages
- JavaScript
- HTML5 / CSS
- Swift
- Objective-C
- Clojure
Frameworks
- Node.js
- Express.js
- React
- Vue.js
- Electron
Technical articles
Over the past decade, I've written a total of 169 technical articles on various programming languages, frameworks, best practices, and my own projects, as I kept up-to-date and active in the software industry.
Subscribe via RSS / Atom.
Featured
- 2017 — "Clean code" isn't actually clean
- 2017 — Passion in your field is overrated
- 2017 — What I learned in 5 days of writing an experimental website
- 2014 — Age of the Polyglot
- 2013 — How to Program
- 2013 — Ignore the Naysayers
- 2013 — Writing Clearly
- 2012 — Reinvent the wheel
- 2010 — Good usability
- 2009 — Twitter is the wrong tool
- 2009 — We're all pretty bad at driving
- 2008 — Why I Code
Chronological
2017
2016
2015
March | Notes on Haskell Extensions |
February | Second thoughts on front-end tools |
February | First thoughts on front-end tools |
February | Some thoughts on GUIs |
February | First thoughts on OCaml |
February | Ways to compile Haskell to JavaScript |
February | First thoughts on Haskell |
2014
2013
2012
September | Testing boundaries |
April | Ruby Accessors Considered Pernicious |
March | Reinvent the wheel |
2011
September | Go interfaces and DIP |
August | The future of Lisp |
2010
2009
2008
December | Cocoa Development is like Bureaucracy |
October | Why I Code |
October | Open at Login, for menu-bar apps |
October | NSLog(@"testing"); |
Portfolio
Here are some of the projects I'm most proud of. They were created using a variety of technologies, running on several different platforms and OSes. They're all finished products, and many of them are open source.
CleanCoders.com
Website - Online Video StoreI wrote this web store for Robert "Uncle Bob" Martin, using Clojure for the back-end, and JavaScript for the front-end. Over the course of 5 years, I took the site from a simple three-page website to a full enterprise-ready business solution, with nearly 100% test coverage.
Leviathan
macOS app - Clojure IDEWhile working on CleanCoders.com, a website written completely in Clojure, I increased my productivity by building a custom IDE for macOS designed specifically for Clojure projects.
Zephyros
macOS app - Hackable AutomationThis began as an experiment to see how many languages I could use to script a custom macOS window manager using our custom TCP protocol. Eventually it had bindings for Clojure, Ruby, Python, Go, JavaScript, CoffeeScript, Node.js, Chicken Sceme, and Racket, as well as other community additions.
Bubble Maker
iOS app - Bubble simulatorThis toy was made in a weekend to entertain my 1 year old daughter. It lets you create bubbles with your fingers, which then simulate physics by bumping into each other and falling.
Quick List
iOS app - Todo list appWhen I couldn't find an app in the App Store that let me make very simple lists extremely quickly, I made one myself. I use it almost every day to organize and track my activities.
sdegutis.com
Website - Personal PortfolioThis very site itself was written from scratch in about a day. It uses best practices for modern responsive web design, and a custom build phase to compile the sources into a single HTML file.
2048
Java app - GameThe game 2048 (created by Gabriele Cirulli) is so fun that my kids wanted their own copy. So I wrote this version in Java 8, using JavaFx for attractive graphics and silky smooth animations.
Mjolnir
macOS app - Window ManagerI created this app to increase my productivity by letting me move windows around in macOS using keyboard shortcuts. It grew into a community-driven highly extensible app, using Lua for its plugin system.
AffluentConfidante.com
Website - Social NetworkImplementing this elite social network gave me experience integrating both Apple Pay and credit card payments (via Stripe.com) seamlessly into web apps, for a frictionless and pain-free payment experience.
HyperChat
Website - Live ChatroomThis isn't just any chatroom. In this web app, you can see what everyone is typing while they type it. I made this in order to scratch my itch for making real-time apps and games, and learned how to use WebSockets in the process.
Bahamut
macOS app - Music PlayerAs iTunes went through many user interface changes, I wanted an app that was consistent, intuitive, and easy to use. So I created Bahamut, a minimal music player for macOS with a custom user interface.
Chatter
macOS app - Chat (IRC) ClientThis was written in 2009, before the time of Slack, when IRC was the main way for programmers to get short-term assistance from each other. Its purpose was to be a beautiful app with an emphasis on simplicity and usability over technical power.
AppGrid
macOS app - Window ManagerThis is an app I actually use every single day. It lets you move windows with global keyboard shortcuts. Since it uses Vim-like key bindings, it should feel pretty natural to any programmer. There's no configuration needed; it Just Works™.
Hydra
macOS app - Lua window managerAs an evolution of Phoenix, Hydra was my first attempt at embedding a full Lua virtual machine into an Objective-C app, to make a lightweight and efficient window manager that focused on speed, low memory usage, low CPU usage, and overall being gentle on laptop batteries.
Phoenix
macOS app - JavaScript window managerAs an evolution of Zephyros, Phoenix was my attempt to use Cocoa's native JavaScript bindings to make a more lightweight and efficient window manager, that focused on speed, low memory usage, low CPU usage, and overall being gentle on laptop batteries.
Smaller projects
These may be tiny, but they're interesting technical feats.
Lua4Swift | Swift framework for embedding Lua with a native Swift API. |
choose | Command line fuzzy-matching tool for macOS that uses a GUI |
music | Command line music daemon for macOS that only speaks JSON |
hecto | Command line text editor with an embedded Lua plugin system |
ZephSharp | Window manager for Windows using Clojure for scripting |
management | Minimalist EC2 configuration & deployment tool in Ruby. |
go.assert | Assertion helper package for writing tests in Go. |
go.shattr | Go library for printing shell-attributed strings to stdout. |
OCDSpec2 | Objective-C based testing framework with Xcode integration. |