Unit Testing, Automatic Testing, TDD – Pros and Cons

U

I had a lot of recurrent conversations over why we should use automatic testing, unit testing or TDD. Through these conversations, I’ve heard a lot of arguments pro and con to these techniques, so I decided to summarize them.

To be honest, I am very passionate about TDD. I believe this technique is a must know for every developer who’s serious about his or her craft. Despite my personal preference, I will do my best to present both sides of the arguments as clear, pragmatic and even as I can. If you spot any mistakes or errors please let me know and I will update the article.

I will split the discussion in a few parts because I feel the arguments can be separated from general to specific.

Testing

Cons

Testing takes time and it doesn’t provide enough value.

(I’ve never actually heard this argument, but I can imagine people using it.)

Pros

The baseline value you can get from an application is people using it.

Code doesn’t just work. No matter how smart your programmers are, they will miss something.

If the application crashes, people won’t be using it.

If the application doesn’t do what it’s supposed to, people won’t be using it.

Discussion

Unless you want to make some crazy tests on humans (e.g. seeing how much they can stand using a non-working application), this is not the way to go. You need to verify that the application does what it’s supposed to.

Manual Testing

Cons

Manual testing is repetitive work and the computer does it much better.

The amount of manual testing required for an application increases over time. The only way to keep up is to add more testers, which then adds overhead on management.

Pros

Manual testing is not only repetitive work, but also discovery and critical thinking.

Our developers write clean code, review it thoroughly and test the application.

Discussion

The logical conclusion is that manual testing is excellent for discovering hidden issues. All the repetitive manual testing should be automated. However, testing automation doesn’t come without overhead. Adding more people to test also adds an overhead. Which one is larger? The problem is especially difficult when working with legacy code.

Testing at its core involves two things: understanding what it is that you are testing and actually performing the tests. The repetitive tasks from second part can be automated; the first part requires smart people and cannot be automated. So, is it better to ask from more people to learn or to manage the automatic tests?

From this analysis, it seems that learning is the bottleneck. If you remove all repetitive work, you only have to optimize learning.

In the case of developer testing, the problem is that they either test the whole application (which is time consuming) or only the parts that were changed (which is error prone). However, if you cannot do automatic testing, this is the next best thing because it creates the premises for catching and fixing most of the bugs early in the process. The risk you’re facing is that the bugs that escape this process can damage your product. Due to the side effects from the code, these bugs could be nasty. Or, if you’re lucky, they might also be trivial. You don’t know which ones you’re gonna get.

Removing repetitive work has beneficial effects for your team. It allows each and every team member to focus on doing the core activities needed for his/her work. In the case of testers, they learn and do “smart” testing, being suddenly freed from the worries of repeating the same scenario over and over again.

In the end, the choice depends on your context, but going towards automatic testing seems to be the best way.

Automatic Testing Done By Testers

Pros

Automatic testing eliminates repetitive work.

Cons

Developers should test the code they write. Testers should only do smart testing.

Discussion

Testers usually test applications from the user interface. Automating this part is possible with various tools – some of them incredibly expensive, some of them free, depending on what they’re used for.

The question here is why should testers find from the user interface issues that can be more easily found from the code? This is an overhead that can be moved towards programmers, leaving testers to worry about things like workflows through the application, user experience, interface design, hidden issues etc.

I think that most teams come to some sort of equilibrium between programmer automatic testing and tester automatic testing.

I’ve lately worked mostly with teams from startups, without testers so programmer automatic testing was a must. Please share your experiences.

Automatic Testing Done By Programmers

Pros

Saw the pros above

Cons

Tests add overhead to development. Whenever code changes, tests need to be changed as well.

Automatic programmer testing requires a testable design. Testable design is sometimes different from OOD.

Discussion

Writing tests is not enough. Programmers need to write tests that are extremely simple.

For example, some of the guidelines for writing tests are:

  • They should be very short
  • They should run very quickly
  • Each test should assert on one and only one thing

It’s fairly difficult to learn good programmer testing, and there are many traps that a team can fall into if tests are not simple enough. Some of the typical issues are:

  • Tests take a long time to run, so people don’t run them anymore
  • It’s very hard to change the code because you need to change a lot of tests each time

Some of the problems are alleviated by using powerful tools (ReSharper for Visual Studio or Eclipse are highly recommended) that make changes much easier. The other part of the problem – writing simple tests – takes practice and it’s an art in itself. The software craftsmanship movement shows how any programmer can learn these techniques.

Regarding testable design, the typical issues that arise are due to creating seams. Seams are ways to inject the code of collaborators for a class into that class. The most used one is through constructors and interfaces, and this means passing all collaborators in the constructor as an interface (thus creating lots of public interfaces). Other are possible, like having read/write properties for each collaborator, having read/write function properties (delegates, lambdas) etc. These kind of things contradict our intuition on OOD.

I agree with Roy Osherove when he says in “The Art of Unit Testing” that testable design is not OOD, but a different type of modular design. The main purpose of any design method is to produce code that does what it has to do and that is flexible. Testable design allows this in a slightly different way than OOD, with one advantage: it allows more validation. Pragmatically speaking, this advantage is much more important than violating our aesthetic senses.

Regarding the productivity, it’s true that writing tests slows down the development process. However, according to practitioners, empirical evidence and to a few studies (including one from Microsoft), the time for writing tests is more than made up for in the testing and maintenance phase so on the whole lifetime of the project, productivity is larger when writing tests. Bear in mind that neither of this is definite evidence that it always works; I believe that it’s however a very good reason to try it.

One thing is clear: doing automatic programmer testing in the right way is difficult, it requires lots of practice, continuous learning and re-training your intuition but creates value by validating your code.

Unit Testing After Writing the Code

Pros

We know exactly what to test because the code is written.

We design the system as we want before writing the code.

Cons

Your tests will be influenced by the code.

You never find the time to write all the tests if you write them after the code.

Upfront design is not needed. Emergent design does the work for you.

Discussion

The typical things I find out when programmers tell me that they write tests is

  • the tests are written after the code
  • there’s never enough time to write tests, so they get postponed indefinitely

We also need to take into account in this discussion the tendency towards overdesign and the tendency for analysis paralysis. I’ve known a lot of teams that started their projects by building a framework that could be used not only for that project but also for other similar projects. Most such projects failed.

Programmers love to prove they’re smart. Many of them do it by discussing about ideas rather than producing something. (Remember, I’m a programmer as well, so I can say these things.)

Since we know all this, anything that helps moving programmers towards just enough design is beneficial.

Test First Programming

Pros

I get to design before writing tests and write the tests that test the code matches the design.

The tests are written before the code.

Easier to do than TDD

Cons

Potential for overdesign, analysis paralysis.

Discussion

TFP solves one issue: you will write the tests. It doesn’t solve the others: overdesign, analysis paralysis. It allows however a focus on a small part of the code and thus less design, so it’s already better.

Test Driven Development

Pros

You always have tests for all the code.

Very little upfront design (not 0, but very little).

Good, simple design emerges from applying TDD.

Cons

It’s unnatural and hard to do

How do I know emergent design works?

Discussion

I believe that the unnatural part of TDD is due to our formation as programmers. We are trained to give solutions, whereas TDD makes us ask questions and teaches us that the solution will emerge. This is almost a blasphemy, it goes against everything we think we know and it requires a leap of faith.

If that’s not enough, we need to suspend our sense of design because sometimes we pass through a defactoring (making code look worse but doing the same thing) phase. It’s like all the world moves upside down.

Once you pass these psychological obstacles, you need to go even further by training yourself over and over to think simpler and to ask good questions. I found during my TDD practice that the bottleneck of TDD is understanding what tests to write.

And you always wonder: Does emergent design really work? What if it doesn’t work in some case? Does TDD really work? Are there any complex applications built using TDD? Are there studies that support TDD as a best practice?

My take on it all has been pretty simple. I’ve heard about TDD, I tried it, I saw its benefits, then I read some more about it, then I practiced some more and then I started using it in real applications and I had to learn some more and now I don’t want to work without doing TDD. My reasons are:

  • TDD makes me advance at a sustainable pace
  • TDD helps me avoid analysis paralysis and overdesign
  • TDD makes me focus on the thing I need to do
  • TDD forces me to work on small things only
  • The tests are always there to save me from my mistakes

If we look in the industry, there are companies using TDD extensively. Some of them may just say that they do it but we can’t be sure. However, there are some that we can be fairly certain that apply TDD: Hashrocket, Obtiva, 8th Light. The Ruby on Rails community is especially keen on doing TDD.

Regarding emergent design, we cannot have a proof it always works; from the collective experience, it seems to work in the majority of cases. Algorithms for example are an area where emergent design doesn’t seem to work very well. However, most applications nowadays use very few algorithms and in very specific places. The knowledge and experience of the developer is and will always remain very important.

In The End

I don’t think this is the end, but at least I hope I gathered a lot of useful information that can help you decide how you do testing. Let me know if this article helped you or not. If you need help learning unit testing or TDD contact me.

43 comments

Leave a Reply to Dan Sutton Cancel reply

  • Though not quite your phone ideal, there are a few android phones with keyboards. On them, Gentoo Linux can be installed along-side android, giving compilers and many other *nix tools, accessible through an android terminal program. Hope this helps!

    • Cool, I had no idea.

      I am now using a Nokia E71, and it has physical keyboard but none of the other qualities. I’ve seen that the Android drivers will get into a future kernel version, maybe it will make things easier.

      • i was just about to post about this same thing. i have a motorolla droid (still gen 1) it has a physical sliding keyboard, you can easily install any android build or i imagine a number of other OSes (you have root access you can do whatever you know how to do) i have just a stock build (didnt wanna risk voiding my warranty but i might root my next one) and i have an ssh terminal on it which i use with my linux box, i also have android scripting environment which, while not a compiler, allows for making scripts in a number of languages (been learning python on this baby)

    • Thank a lot.

      I knew about Hacker Spaces, but when I looked the last time there wasn’t any in Romania. Surprise, now it is! I’ll definitely check them out.

  • Are Macbooks really more difficult to repair? The source of the self-repair manifesto you provided is a repair guide for apple products.

    • I never used a MacBook extensively but in the company I work for about 10 people do use one. As a result, I never had to open or fix one. I may well be mistaken.

      I can tell you what I know. I know that MacBooks have a unibody. The people I know that use MacBooks would never dream of opening one up – it’s the perfect black box for them. Maybe the website contains instructions for fixing apple products because it’s harder to do so.

      But if apple products are just as easy to fix like the ones coming from other companies then it’s great. Does anyone have a better idea on this topic?

      • The unibody macbook is more of a problem – but it’s not impossible to open them.  All you need is a small head screwdriver (eyeglasses size) and the nerve to ignore the warning stickers. Many of the components are accessible by removing the bottom panel. 

        But it’s a slight disadvantage –  I usually get frustrated tinkering with laptops because everything is so small and cramped! 

        • You can tinker with them to an extent. But, there’s not much space to put in extra components, and to get at the screen / hinges you need a heat gun, glue and a suction cup. On the other hand, I’ve seen an RFID reader hacked into a laptop, and have upgraded the screen resolution on my own – which would be much harder on a MBP.

          • Have you documented any of your tinkering adventures? I have no idea how you can upgrade the screen resolution on a laptop. It would be interesting to learn.

  • I have not tinkered with hardware for over a decade, but there is something about the raspberry pi that makes me excited … the possibilities seem endless

    • Thank you for mentioning it. I took a look and it seems to be the tinkerer’s dropped from heaven device. I hope it will come to life.

  • Hi Alex,

    I think that we, people from most of post-communist countries, have the same experience like you. We were great in reparing of cars, great in computer (now so called) hacking, inventing how to do big things from out of blue because of lack of material or even the knowledge.

    And I am personally glad we have such experience.

    Thank you for writing this post remembering me our roots. Because that makes us different in the world…

    • I sometimes feel sad when I see the new generations in Romania missing all this fun and learning opportunities. In the case of electronic devices, it may be a sign that they are maturing. But I know that in the past innovation came from tinkering. We should encourage it and it should be part of our mindset if we want to keep moving forward.

  • Hi Alex,

    I think that we, people from most of post-communist countries, have the same experience like you. We were great in reparing of cars, great in computer (now so called) hacking, inventing how to do big things from out of blue because of lack of material or even the knowledge.

    And I am personally glad we have such experience.

    Thank you for writing this post remembering me our roots. Because that makes us different in the world…

  • Alex,

    Thank  you for putting into words what I have felt since my youth. I got started taking things apart when I was very young. My grandparents kept a very old telephone set that I would spend hours disassembling and re-assembling. As I got older, I have taken apart, repaired, modified countless radios, computers, vehicles, musical equipment, appliances. I have developed a reputation as one who will spend the time to fix something rather than go out and buy a new whatever.

    This is seen as strange for someone of my generation. Many of my peers have no idea how anything around them works and even less desire to explore the innards of their favorite device. My fear is that the current apathy that runs rampant in society will land us in a totalitarian environment very few are prepared to survive in.

    Starting a local Hackerspace is a great idea, one that I should explore as well.

    All the best, Drew

    • Thank you for your message. Knowing that you are a tinkerer fills me with great joy.

      My advice to you is: find people that share your passion. Meet with them. Take things apart as a group. Play around.

      It might look useless in the beginning but give it enough time and you might come up with something amazing. Even if you don’t, nobody can take your experience away and it will come in handy in whatever you decide to do.

    • I agree completely! But there is a reaction to all action: once a totalitarian society is established then tinkerers’ times come! I can assure you with that having myself lived for 13 years in a communist country – Bulgaria. Once they put limits and bans we always find ways to evade them and move forward although with much effort.

  • Hi Alex,

    I’ve been taking things apart to see how they work since I was 4.  I too am not afraid to remove a cover and see if I can fix a broken piece of equipment of any kind.  In todays world, where a bad component usually means a burnt out chip, I’m not always successful, but that doesn’t stop me from trying.

    I love learning how things work and am a firm believer in the “Hacker’s way”.  I think mankind’s ingenuity is astounding and am ashamed of the companies that try to prevent us from taking their device and finding new and useful ways to make it better.

    I joint the call “Hackers Unite!”

    • Wow, that’s impressive. I admit, after I wrote this article and got all this amazing feedback I realized that I am sometimes afraid of opening up the cover. Comments like yours have helped me decide to tinker more.

      I admit I am annoyed by some companies. It’s the natural reaction. However, that’s how companies work: when there’s a demand, they will provide a supply. We cannot change the world completely, but we can show that there is a demand for hackable devices

  • Hi,

    I was someone of those that has been repairing the things, but in agree with you, today everything is more complex, if you open a device such as phone or LED tv, most of time is waste of time, i open my car and i can’t see nothing to repair by myself. But just like you i enjoy do it, and i’ll never give up because as you say that make us learn and learn is the most rewarding for me. Your post is very good, thanks for sharing.

    • Thanks for your wonderful thoughts.

      I think that things are not as complex as we think they are. They are merely more opaque than they used to be.

      If you open a LED tv today, you will see a bunch of circuit boards, but as a user I want to have the right and the possibility to replace a defective board. Sometimes I won’t do it but I strongly believe in my right to have this option.

  • I think that taking things apart and tinkering with them (both hardware and software) is a function of intelligence: if you’re smart enough, then you must know how the things around you work, or else they annoy you: the act of taking them apart to see what they’re made of is a compulsion more than it is a choice… I also think that you can’t teach people to be intelligent… but if you have kids, then by all means, from their earliest moments, encourage them to do this: it’ll make them more curious and thus smarter: it’s the children who are not taught between the ages of zero and five to want to learn who grow up to be idiots.

    • One of my best moments in life is when I taught my niece physics. Seeing her eyes open in wonder when she realized the beautiful complexity of the world around us was immensely gratifying. And the thing is you just need to teach the children how to ask questions.

      If you haven’t seen Richard Feynman explaining the world, I strongly recommend you do it. There’s a series available on youtube.

      • Surely you meant “beautiful simplicity”, right? (N. Bohr said it: the world is simple in its essence, not in its manifestation.)

  • If you would like to build a phone with keyboard and compiler have a look at this http://www.sparkfun.com/products/10138 it is a GSM module for only $US50. Then purchase a RaspberryPI when they go into production later this month. (http://raspberrypi.org). The RaspberryPI is only $US35 approx, has USB inputs, runs Linux and has HDMI output. You could then write a driver for the GSM module and Walla… you have a linux powered GSM phone that displays on any HDMI screen. You could even run Android on it if you wanted.

  • Yep, necessity is mother of invention and when you have limited resource around you ..you have to learn harder to make the machine do the job…

    • That’s true and what if we turn the argument in our favour? It means that people learn best when they have limited resources. So why not introduce artificial constraints?

      That’s what I do when facilitating code retreats, when I train people and when I want to learn something. It’s the best way I know.

  • Check out the Nokia N900. It’s been a while now, but it’s a phone I know matching your description. Has a keyboard, terminal (xterm) & can have a compiler. Runs Linux natively (not just the kernel, initd, bash, sudo, passwd, X, sshd, /proc, /dev the whole works).

  • ALEX BOACA:

    BRAVO !!.  Some decades ago here in USA the Public Broadcasting System showed a series called “Connections” by James Burke.  This is a history of the development of technology.  One of his sub-theses is that technologies has experienced much of their advances because of  the activities of “tinkerers”. I don’t think he goes into pre-history but one might find that thesis embraced, at least until Octavian, in the evolution of early glass-making.   Burke’s Grande Thesis is not revealed until the last episode.  I am not going to tell you what his thesis is because I don’t want to give it away.

    Keep up the good work.

    terry huff.

    • I know the “Connections” series, it’s one of my favorite shows ever. You just gave me the incentive to watch it fully. Thank you.

alexbolboaca.ro Reflections on design, craft and software

A new home for merging ideas about design

It is my strong belief that software design can learn a lot from other design disciplines. I wrote blog posts, a book and did talks on this topic, and it was time to group them all together. These ideas have now a new home: https://codedesigner.eu. My plan is to add more blog posts there, and to involve other people doing work in this area.

Read My Book “Usable Software Design”

How UX techniques can be applied to software design to develop software better (given that the developer is the user of software design).

Read My Book “Coderetreat Hosting And Facilitating”

Learn how to facilitate and host a coderetreat from two of the most experienced coderetreat facilitators.