As the co-founder of a “142-proof” software development agency in the Silicon Valley catering to pre- and post-money companies, I’ve observed quite a bit of variance when it comes to the development process.
The capabilities range wildly, from hack-and-slash “oh shit commits” all the way to buttoned-down tests and infrastructure that makes us all giddy when we see it. It’s pretty much The Good, The Bad and The Fugly.
It boils down to the core competencies of the team and the experience of the technical leadership. What amazes me is how many of these companies start up doing it wrong. Granted, startup companies are under the gun to hurry up and build a great product and find relevance in the market. Never mind about the shiny bells and whistles involved with how it got produced.
In the first few chapters of The Lean Startup, Eric Ries explains how they honed their process such that any developer could commit code to production on their first day at work. This does not mean they gave the new geek the keys to the kingdom and the power to destroy the service. It means the tech team has enough process in place, as well as checks-and-balances, that nobody can break things — not even a newbie.
The challenge that many enterprises face is a lack of innovation and a slow response to change. Some large companies are trying to adopt “agility at scale” in the enterprise, or set up “intrapreneuer” programs to innovate. However, these enterprises tend to be the exception.
The upper hand startup companies have is that they have a chance to do it right the first time… but the reality is that many do not. The economics of time/talent/capital for startups means that most are running 5,000 MPH toward a goal — and leaving a wake of technical debt behind them.
“But what is this ‘doing it right’?” you ask. Simply put, it means code that is continuously tested and integrated whenever an engineer commits code to the source repository. The most popular alternative is some mix of:
- developer just tests it and/or,
- throw it over the wall to a QA team and/or,
- have a “house party” where the entire company tries to break the build on a staging server.
The adopted solution to house parties is automated user interface testing, aka: end-to-end, or functional testing. This is best achieved by building these tests from Day 1 of a project instead of waiting and back-filling them at a later point in time. Requirements or user stories are translated into tests that exercise each feature, under each possible user persona. Needless to say, I’m bullishly anti-house-party when it comes to testing for quality.
A challenge with startups is that the product continually changes.
You can have all of these awesome tests, and developers can certainly run them in their working environment, but the key is to continuously run them in a test server environment or a continuous integration (CI) server. Each time an engineer checks code into the repository, the CI server wakes up and gets to work.
Ideally, unit tests are run first, because they run much faster, and then the functional tests are run. For large applications, the CI server can break up and ideally parallelize the functional tests to save time. An automated test cycle that takes too long is not of use; I’ve put the upper bound at 15-20 minutes for a max test cycle. Obviously, shorter is always more productive for the team.
By building these tests, an automated web browser can drive the user interface and make sure it still works as intended. This keeps the codebase from moving backwards in production, and stops the presses until it’s resolved, or until they “fix the build.”
Many dev shops I’ve visited have a special monitor in the room that turns red, and I’ve seen a USB traffic light that is evidence of the build’s health.
Besides environmental indicators of the build, I’ve seen other variations on maintaining quality — such as passing around a My Little Pony and placing it shamefully on the desk of the engineer who broke the build, such as the geeks do at Say Media.
I prefer positive reinforcement — after the sprint demo, a medium-sized trophy adorned with Mardi Gras beads is awarded to the geek with the best feature of the week, such as the geeks do at Lyft.
The benefit of emphatic testing at the user-interface level is an increase in product quality. This is because each UI test is akin to a contract that cannot be broken. If the tests are evolving with the requirements, they serve as a watchdog to make sure we’re not rolling backwards.
In chatting with geeks about where to spend the most time testing early on in a company or project, I always tilt toward the user interface because it inevitably exercises all the other layers. If the UI works, the underlying layers are probably doing it right.
The exception is where the product is an API, where unit tests are going to be the way to go. Really, in a perfect world, functional and unit tests should be in place from the get-go.
A challenge with startups is that the product continually changes; that is often the byproduct of the Build/Measure/Learn cycle. Therefore, at what stage is it right to overlay end-to-end tests?
This is an open-ended question that I still ponder. When a company really has no idea of what it should build, it seems of limiting value because there probably are not enough users on the system where quality is of concern.
When the company knows what it needs to build, but the roadmap seems so daunting and talent so limited, it seems only natural to skip functional tests “for now.”
The payout is clearly down the road on the efforts of good testing.
But quality should ALWAYS be of concern, right? In the end, the trade-off is going to be a longer build cycle, but not 2x. I’d guesstimate it adds ~33 percent more to a project to have good functional and unit tests; therefore, on a two-week sprint this is at most days, not weeks.
So rather than “hurry up and get a product out there,” I favor “slow down and do it right, because we are probably living with the new features of this release for a long time.”
The payout is clearly down the road on the efforts of good testing. It creates a codebase that is delightful to work with for future engineers. It creates a codebase which lives longer, is easier to refactor and can stand the test of time.
The idea that I’d like to leave behind for startup founders and lead techies is to “Please! Please! Test Your User Interface.” Do this early in your progression and the payouts will be plentiful down the road.