As my parts of my current project mature, I realise that there are exactly two types of unit tests in my suite, to correspond to the two common types of change that I am making.<\/p>\n
The first is the fragile test. A typical example would have 60 carefully hand-calculated result values. When I tweak an unimportant threshold value that couldn’t possibly trigger any real bugs, it just tweaks performance (e.g. now run 15 threads in the thread pool rather than 10), each of those results must change and require recalculating by hand.<\/p>\n
The second is the robust test case. This is the type that passes despite the fact I have just made major improvements to the code without touching the corresponding test-case. If the old code and the new code both make the test case pass, what is the test case actually testing?<\/p>\n
Most commonly, it is because I have encountered an unexpected exception in later tests, or in the field, due to corrupt data being passed on by another component, and that provokes me to (a) improve the handling of the corrupt data nearer the source and (b) improve the handling of unexpected exceptions. Neither of these paths happen to be tested by the old test code. Until 100% branch coverage is suddenly discovered to be a sensible investment of time, they probably never will be.<\/p>\n
There is actually a third set of test cases – the ones that correspond to the times when I don’t make any changes. These are the ones that test some non-deterministic behaviour, and just fail randomly now and again (on perfectly good output) because the scheduler happened to take a different path to the one it did when the predicted output was determined. It might be possible to make these tests more robust, but the effort isn’t justified – especially if the component under test is stable and believed to be be bug-free (by manual inspection of the differing outputs).<\/p>\n
Of course, this is a very cynical view; it ignores the benefits I get from my tests. I should be happy with the revealed opportunities to improve the tests to make them both more robust and to have better coverage.<\/p>\n
But mainly, it helps get me annoyed at the people who evangelise automated unit-tests as though they are a magic panacea.<\/p>\n
I got very frustrated with the first book on Extreme Programming that I read. The author continually harped on about how exciting it was to have unit-tests, and that he would often press the button to run the whole suite when he sat down at his computer. <\/p>\n
I couldn’t understand what he was talking about; it made no sense to my understanding of development. <\/p>\n
It wasn’t until very late in the book he admitted it was really only practical if your unit tests ran in under 20 seconds. Twenty seconds? What project has unit tests running that quickly? <\/p>\n
I am on a small project, and my unit-tests for the main program alone take over five minutes. Several of the individual modules take far more than 20 seconds to tests. Running the unit-tests is a productivity killer, because it forces me to get distracted from my main development focus while I wait.<\/p>\n
At the time I read the book, I worked on a large, old project which, in my opinion, had woefully low unit-test coverage. Running all the automated unit-tests it did<\/em> have took hours.<\/p>\n It left me shaking my head at the real-world experience this author must have had.<\/p>\n To summarise in the vernacular, unit-tests suck – I wouldn’t bother with them if it wasn’t that the fraction of bugs they do manage to catch, suck even more.<\/p>\n","protected":false},"excerpt":{"rendered":" Wherein Julian bitches pointlessly about unit tests because it is late at night and he is bored of running them.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_s2mail":"","footnotes":""},"categories":[24,34],"tags":[],"class_list":["post-1194","post","type-post","status-publish","format-standard","hentry","category-cathartic-rant","category-software-development"],"_links":{"self":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/1194","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/comments?post=1194"}],"version-history":[{"count":4,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/1194\/revisions"}],"predecessor-version":[{"id":1198,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/1194\/revisions\/1198"}],"wp:attachment":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/media?parent=1194"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/categories?post=1194"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/tags?post=1194"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n