Testing software is hard?

Yeah, it’s hard. As much handwaving as there is in this article about code coverage (by a developer who works for a code coverage company, so get your grains of salt ready) about how simple Tetris is–it can be implemented in a single line of BBC Basic after all, and everyone knows “lines of BBC Basic” is becoming the hot new unusual unit of measurement–his article is actually about the many difficult-to-produce corner cases from the user interface’s perspective in an actually useful implementation of the game.

There are two primary paths people consider to gather code coverage data. On my current team at work, we typically use automated unit tests. Our coverage tool invokes the unit test console runner with arguments about which DLLs contain tests and what code to ignore, and we publish the resulting HTML reports and trend numbers on our CI server. The linked article is taking another popular approach: have actual humans run the application while the coverage tool watches and reports on what code they hit and what code they missed.

I’m not a huge fan of human-only testing–even when they are following a written script closely they might miss something or “do it wrong”, as we developers like to say about users, and even when they make videos of the entire testing session that just creates another problem in the form of huge volumes of material to review to find issues. That said, users or automated UI tests definitely answers some interesting questions about the code that unit tests cannot, like “is any of our code unreachable from the UI?”

For example, when there are known corner cases where code is extremely difficult to reach deterministically from the UI (as in the article), that would be great to know so you can do something like create a UI shortcut to manually invoke it, or otherwise increase your coverage with specific automated unit or integration tests.

All that said, I have to hand it to the author for making code coverage actually fun and engaging–this is a huge challenge by itself, so kudos to him!

This entry was posted in Coding, Continuous Integration and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s