Beyond Red/Green/Refactor

Michael Feathers' post Making Too Much Of TDD. A must read.

Test Driven Wall Tiling

Last week I was tiling a kitchen backsplash and countertop and as I was using the usual tile alignment tricks, I was struck by the parallel with the Test-driven development techniques for application development.

This picture shows the wooden cleat (the dark brown area on the picture) that was temporarily screwed to the wall to obtain a perfect alignment for the second row of tiles above the countertop.

It made me think of TDD because :

• The screwed wooden cleat is a temporary built artifact that has nothing to do with what we want to construct (that's to say, a tiled backsplash), but we built it nonetheless, and with care, as it's going to direct the alignment of the tile. That's what we do with unit test, they are a programs designed to unit test other programs.
  
• The wooden cleat is attached to the wall using the same tools and technique that are used to build the main "noble" artifacts (wood, screw, you name it). A unit test is build with the same skills than the tested program.
  
• When the cleat is fixed we do not have to think anymore about the tile alignment. The alignment is forced by the wooden rule, we can forget about it and concentrate on the quality of the tiling. The unit test creates design or behavior constraints on the code that is written. You can then forget about these constraints (you'll get a red test anyway if you don't respect these constraints)
  
• The wooden cleat is set at the beginning of the tiling for the row, you do not verify the alignment at the end. Test first
  
• Nobody told you to screw a wooden cleat on the wall. Its an act of craftsmanship. We do it because we think that it will improve the quality of the tiling. You have to write unit tests, even when nobody tells you to do so.
  
• Sometimes, it is not perfectly clear where the tiling must start, so you just set the tiles on the countertop without adhesive, and then you know where the wooden cleat must be fixed. Can be seen as exploratory testing.
• I need the second row to be perfectly aligned since the first row above the contertop can not be aligned on the countertop itself (tiles thickness may vary). When parts of a system have good unit tests, the integration of these parts is easier.
  

So why do we still see large software systems that are written without any sort of unit tests whereas this seems to be a widespread technique for building physical artifacts (tiling, walls, etc...) ?

I think it is because programming is like tiling without adhesive. You may always be able to move the tiles. But even though we may try to align all the tiles at the end of the project, we would not completely succeed, and the quality of the alignment would suffer.

We should rather use all the available engineering techniques during the life of the project in order to be sure that the alignment is perfect, all the time. From start to end.

TDD and tomatoes

Tomatoes are like tests, it is easier to make a tomato go from green to red, than from red to green.

TDD : Can Shared Fixture test unexpected statefulness ?

When praticing TDD, the impact of the different types of fixture that can be used to setup the tests must be known in order to avoid to create Obscure Tests

The usual default fixture type is the Fresh Fixture

When the fresh fixture takes too much time a Shared Fixture can be used. Now, that being said, the shared fixture should usually be avoided (there's a risk of Erratic Test)

A few days ago, during a discussion about test fixture, some colleagues of mine and I were wondering if shared fixture could be used in order to test the statelessness of the objects used in the fixture as a side effect of the sharing.

Here is the initial reasoning : when an object is supposed to be stateless, but is buggy and not actually stateless , the use of this object will lead to some failing test, since the state left by some tests will make some subsequent tests fail. The statelessness is tested "for free" with a lot of different combination. Actually, if the object happens to be stateful it is a case of Interacting Tests and the reason why some tests fail could be quite hard to find.

I must admit that it may help to find some obscure statefulness interaction, but I do not really like this idea :

• I fear Interacting Tests far more than a single bug, as Interacting Tests in a test suite is hell on earth,
• A test is supposed to have One Assertion Per Test. Or, at most a few assertions. In this case, the statefulness test is spread in the whole test class.

Actually, the statefulness should be tested in its own test. Of course it means that only a few method call combination will be tested but it should be enough to prove the statelessness (or else it means that the code of the class under test must be refactored in order to make the stalessness explicit in the code structure)

To try to use the side effect of the shared fixture to test statelessness may seem to be worthwhile at first, but it creates Technical Debt. And in this case I think that the financial interest of the debt is pretty high.

"Unit Integrated Tests" for integrator objects

Whether I use interaction based testing or state based testing I usually write "real" unit tests in the sense that only one real object is involved (the SUT, System Under Test) and only stubs and mocks are used for the fixture. It usually is the only way to easily get a good code coverage (or at least to easily specifically cover some parts of the code )

However, Lately, when testing Facade or Adapter objects he seemed to me that the unit tests for these objects were :
- too complicated
- too...useless

For most Facade or Adapter objects :
- They're stateless, so let's forget state based testing, we must use mocks
- They're highly coupled to the contract of the adapted objects (by definition of a Facade or of an Adapter)

This kind of class does nothing clever, they basically chain one or multiples calls. They may have bug though if they do not respect the contracts of the adapted classes.

So a unit test for this kind of class depends more on the contract of the mocked objects, than on the behavior of the class under test. A very clear test smell.

Even worse, the fixtures for the mocks are going to closely mimick the code of the SUT (if a method of a Facade calls an object A and then an object B, the fixture for the test on mocks and stubs will tell the exact same thing).

In this specific case the unit test does not provide an orthogonal view of the behavior implemented by the code (the usual value of a unit test) ; it is the exact same view. If I make a wrong assumption on the contract of the adapted objects, I'm going to make the same exact mistake both in the test and then in the code (TDDly speaking)

The problem basically is that the job of this kind of objects is to integrate other objects in the system. So he seemed to me that it is very hard to try to test an integration behavior in isolation.

That's why, now, when I must deal with objects that heavily depend on the contract of the collaborating objects but do not do anything complicated, "integrator objects", I use integrated tests.

However when I encounter an object that heavily depend on the contract of the collaborating objects and does many complicated things, it just means that it is too hard to test, I refactor.

TDD : How I was overusing mocks

The path to TDD is hard. When you think that you begin to understand how to apply it, you read this great Jay Fields post, and you understand that you missed the point about stubs...

As explained in his post, I was overusing mocks. I knew that I had to improve a lot in my way of using mocks, but what I really had to do was to use fewer mocks and more stubs. Basically I was missing the point that stubs are still of great use and are not superseded by mocks.

It's been a long time since I learned such an important thing in a post about TDD. Now I'm really going to try to stick to "one assert per test" (including "one mock assert per test")

And trying to do that, I'm also going to try "one test class per test setup". See this presentation about BDD by Dave Astels

EasyMock and methods with side effects

I really like EasyMock, but was disappointed to not see any explanation in the documention about how to simulate the modification of an argument of a method with side effects.

I thought that maybe I could write a custom argument matcher that modifies the argument (even if it seems weird to implement a side effect on the matches method of the IArgumentMatcher)

Actually, it was confirmed by this brilliant article about a Generic Custom Argument Matching in EasyMock

TDD : state-based testings VS behavior-base testing

Using TDD to write my code I'm still trying to find when I should use mock objects or just test against the whole cluster of objects. That's the difference between state-based testing (no mocks) and behavior-based testing (using mocks) explained in this brilliant article by Martin Fowler : Mocks Aren't Stubs

I'm sure that both technics are useful and I switch from one technic to an other depending on the complexity of the objects I'm testing.

• State-based testing (no mock) tend to increase the amount of integration test between objects, so is better suited to tests clusters of object with complex interaction


• Behaviour-based testing (with mocks) tend to increase the test coverage on a given object as it is easier to control the test environment of the object under test

Using mocks objects tend to create more work, but the more I use them the more I detect bugs in my code that would very hard, or impossible to detect with state-based testing.

Last time it happened it allowed me to see that an external object was called and was not supposed to be called (because the mock was called and warned me that it was an unexpected call)

I would have never been able to detect this call with state base testing because :

• It did not change the state of the object under test


• It did not change the state of the external object

However the call to the real external object is very CPU intensive and would have harmed the performance. Behavior based testing allowed me to see this, it would have been undetected with state-based testing.

TDD, red-green-refactor : red as important as green

When I got started with TDD, I tended to often forget the first step : red

At first, it can seem silly to try to run a test when you did not write the matching functional code. You may think "Of course, it is going to be red ! There's nothing in my application for this feature !"

Of course... Unless there's a bug in your test. And it could be green and you're not going to see it. Then, you write the application code, with a bug too, you run your buggy test, and it is green even with your buggy functional code...

Basically the red step of red-green-refactor helps you debug your test before your code. The red step really is some sort of TDD applied to the test itself. When there is no functional code, a test must be red. It does not mean that the test is correct, but if it is green, surely, it must be fixed.

And even if your test is red and buggy, it is not a big deal, because when your functional code will be written, the test will be red too. The problem could come from your functional code, from your test, or both, but at least you will have to fix something to make the test pass.

TDD like all Agile techniques, is all about rapid feedback. Red gives you feedback about your test, green feedback about your functional code.