Despite being slow, error-prone, and the bane of engineering teams everywhere, manual software testing is the industry norm. An astonishing 81% of software companies still rely on manual end-to-end testing as part of their current QA processes. Automated testing catches more bugs and speeds time to market but while every other aspect of the software development lifecycle gets automated, the old ways of end-to-end testing persist.
QA Wolf is betting that their new approach, what they call “QA as a Service,” will solve the problem that companies face trying to build comprehensive automated end-to-end test coverage: cost.
Whether software companies tried to build an in-house QA function or outsource test automation to an hourly contractor, the cost of one QA engineer for every five developers, plus additional infrastructure, made 80% test coverage impractical or impossible.
By bringing all of those services together — test creation, hosted parallel run infrastructure, and 24-hour maintenance — QA Wolf is able to leverage their scale and offer test coverage at a significantly lower cost. In case studies, their customers say that QA Wolf offsets more than $500,000 per year in ongoing QA engineering and infrastructure costs while increasing their test coverage to over 80%.
“For executives and technical leaders, QA Wolf ensures that their customers are getting the best possible user experience — free of bugs — for a fraction of the cost of how QA was historically done,” said Jon Perl, CEO of QA Wolf.
Understanding the true cost of automated testing
Over the years, companies have tried all different models to build, run, and maintain automated test suites — dedicated in-house QA functions, shifting left and having developers own test management, outsourcing to hourly contractors — but none of these approaches have given companies a solid return on their investment. The reason is that suite maintenance increases proportionally to its size; whoever is responsible can’t keep up with the ongoing and ever-expanding burden.
End-to-end tests are notoriously flaky — that is, they will frequently fail even if there isn’t a bug. Sometimes it’s an issue with the testing environment, sometimes the problem is that a developer updated the front-end without updating the test that’ll check it. Whatever the reason, each failure has to be investigated, broken tests need to be repaired, and bug tickets need to be filed and triaged.
With 10–15% of all test runs failing, and maintenance taking anywhere from a few minutes to a few hours, a full-time worker can only realistically manage 25–50 tests on their own. For a product engineer who’s also working on new features, it’s much less. To build more tests and maintain the larger suite, means more people and more money.
QA Wolf estimates an engineering team would need 1 QA engineer per every 3 developers to maintain high automated test coverage as new features are being built. According to Payscale, the average salary in 2023 for a QA engineer in the US is $75,972. Add in test infrastructure costs and QA Wolf estimates a company with 10 engineers would likely need to spend mid-six figures to manage a comprehensive test suite. For most companies, the investment needed makes automated test coverage unfeasible.
The other cost is infrastructure, and companies pay one of two ways: With additional compute power to run large suites in full parallel, or in developer downtime waiting for sequential test suites to clear through the CI/CD.
QA Wolf brings down the cost of testing and aligns pricing to value received
Whether companies choose to pay in QA engineering salaries, test runs, or hourly fees the critical failing is that they’re paying for time, not value.
Instead, QA Wolf charges per test under management. Because they include the building, running, and maintaining of the tests as part of their service, QA Wolf is better incentivized to create quality tests that require less maintenance moving forward, resulting in lower false positives. And with infrastructure and test runs included, their clients don’t need to worry about any unexpected fees.
Want more details on how QA Wolf compares to in-house and outsourced solutions? See their guide on automated end-to-end test coverage.
Skeptical they can test your web app? Talk to QA Wolf about your most complicated user flow and they’ll test it for free.