Post-prelude: I've written this during the 76 minutes I was waiting for a test suite to reach completion, so I could commit some changes and have lunch.

Testing takes time.

"- Thorough testing takes a lot of time and there's no escaping it."

"- Developers should not be concerned with the runtime of the tests, only the development time."

"- In every app I’ve worked on I’ve found that it’s not the testing that doesn’t scale, it’s the poor test implementation that doesn’t scale."

Reality check.

It's a fact that verifying software takes a lot of effort, and is costly. It's also a fact that every company has some (relatively?) bad employees. Appreciate their work all you want, but deep down inside you know their work is sub-par compared to the rest. We live and develop inside imperfect and unfair environments, and this is reflected on our work.

The attentiveness of agile teams on testing is bordering on OCD. For big products/projects, which are targeting mission critical environments, one wants to be extra-sure a piece of software will perform as asked. To do so, one needs to test a product on multiple levels over multiple axis:

• unit and entity testing
• integration testing
• functional testing
• security testing
• performance testing
• stability testing
• regression testing
• usability testing
• acceptance, compliance, compatibility, etc.

even when you skip or take out several of the above, for any reasonably large project your tests will surpass tens of thousands. There comes a point where the test suites grow large enough that they slow down the development cycle - I assume their effect on the testing cycle is apparent enough as it is.

I haven't worked on a lot of big projects, and I'm not a fan of exhaustive testing and ~90% code coverage, so take all this with a grain of salt. I hate waiting for 40 minute runs of unit tests just so I can commit my changes, and I hate having to keep an eye (OK, a script, but anyways) on 1-hour-long integration tests. Are there ways to cut down on test runtime?

Test fewer things

Sure, I don't need to run all unit tests before committing my changes. If I only run the unit tests which are relevant to my change, I have saved up a massive amount of time already, which I can spend creatively on something else.

In practice, however, the full tests during the integration stage still have to be run. Of course you will break builds because of unforeseen or untested dependencies. You will lose your confidence to your integration cycle, methods and developers (even though nothing might be wrong with them), and your testing will still be slow.

The alternative interpretation of "Test fewer things" is, "cut down on the code coverage". Cutting down on the testing scenarios is feasible (100% coverage is both impossible and unnecessary), until it hurts your deployments, your credibility to your customers, the flood of bug reports and frustrated developers which are continuously occupied with patching old holes. Until that point, there's no clear-cut solution; you can't leave stuff unattended..There are also testing areas where it would be considered unacceptable to leave other content untested (security testing, most likely).

Mock stuff

Mocking is a technique where objects are tested in isolation, and external dependencies (databases, LDAP servers, external signaling entities, etc.) are removed from the test scope. Testing those is delegated to another testing level; most commonly integration tests.

Mocking has a significant advantage: properly done, it can speed up development times tenfold. However, all it does is displace testing to a different level, because end-to-end integration testing is necessary to keep your sanity.

Mocking has a significant disadvantage, too: mocking code has to perform according to specifications of systems external to your project. Yes, you already have to know the end-to-end behaviour of those systems but the reality is somewhat more complicated. We often have to deal with systems that do not strictly conform to standards, proprietary equipment, poorly documented solutions (hi, slapd) and so forth. Try writing mocking code for those; it's a challenge in and of itself.

Some also express the concern that mocked code is complicated and difficult to maintain. Tests are code; you can do it right or wrong. I don't see why mocking should be any different.

Parallelization

For me, this is the most attractive (and potentially most effective) approach. The testing status quo for the product line I'm working on involves sequentially run hundreds of test cases on a cluster that, normally, can handle millions of concurrent users in parallel.Yet the approach we've taken with regard to testing is completely unintuitive: run each test case in complete isolation, exclusively reserving the whole cluster plus a testing box (if it's a proprietary hardware & software solution, you are royally screwed) for the whole duration. We just assign responsibility for a set of test suites to an arbitrary team and hope they'll run their integration/regression tests daily. And have the testers self-organize and schedule testing between themselves..

Parallelizing your workload works. It can be hard (not as hard as some would like you to think), but the benefits are immense. The challenges? Consistency, for one. Especially if your configuration setup is a monolithic beast. Blame will most certainly be put on parallelization and obscure race conditions instead of the actual issues in the codebase. Also, hardware is not free - and please, please try to avoid specialized, difficult-to-extend, testing solutions (at this point I'd bash on a certain company for a couple pages if I had enough € for the resulting lawsuit).

Less features

Yeah, I want to see you trying to pitch that angle to your customers. Be as agile as you want, your customers are rigid. That's not likely to change any time soon.

What not to do when a bird shits on you

Is testing really the twelve-headed hydra everyone makes it out to be? I don't think so. Despite all the open problems in software engineering we have lots of tools and methods to produce good software, and this doesn't seem to be the case with tests. I think that when it comes to testing, we conveniently overlook all the common system engineering principles and more - keep it simple, optimize for your hardware, have some coding guidelines, understand the system you're working with, and the list goes on. Maybe we should reason on testing the way we do with the "real" computer science problems. Maybe we should start taking our own advice before pointing fingers or trying to invent the end-all, be-all, "second-generation, outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology".