Andreas Frömer, Head of Software Development at Finanztip, shared his success story of using TDD to rewrite a legacy application, to replace the old unmaintainable code with the new tested one.
> Unfortunately, many teams make the mistake of jumping straight into Unit Tests. That is very dangerous - if we jump straight into Unit Tests (with no high level tests), then as we break dependencies and introduce unit tests, we could cause regression bugs at the system level!
Welp, I wish I thought about this or read this 2 years ago...
Thanks for clear explanation! Maybe I will be able to come back with a case study to you in a few months :D
Looking forward to hearing about your case study if you write it!
It was a big discovery too. I used to be very much into starting with Unit Tests, because that's also what I saw in refactoring books, and people like Uncle Bob also had big emphasis on unit tests... later I realized that Acceptance Tests is the low-hanging fruit, it's safest and faster to bring in Acceptance Tests and they give us highest ROI... then later we can move onto other tests...
P.S. Even books like "Working Effectively with Legacy Code" are mainly all about introducing Unit Tests.. no mention of Acceptance Tests... that's where I differ.
We can have Characterization Testing at any level. So yes, if we write System Tests based on the current behavior of the system (i.e., we take snapshots of how the system behaves, not how it should behave), then those are Characterization Tests at the System Level. We could also go to the Component Level, see how each component behaves, write tests - those are then Characterization Tests at the Component Level. Similarly, at Unit Level.
Even though Feathers mainly illustrates Characterization Tests at the Unit Level, I view them as existing at all levels.
P.S. I've found also similar case when I needed to do a rewrite - I went back to the requirements, and then created a new implementation that was much simpler than the old implementation.
If you put it that way it sounds really inside 🙈 It didn't really felt like it at all.
As Valentina already mentioned: As I added more implementation, it also came with a lot a new tests, so its not the "3 million business implementation" you could say half if not even more is just lines of code for tests.
I probably should take a deepdive and really compare the business logic before and after.
That would also be interesting to see alongside the overall 3M LoCs.
I'm a teach lead, and some years ago we start a Continuous Delivery transformation journey.
Our application Is a iiot gateway Linux embedded device
We change lots of things in our brhaviour, we had only manual test then we started to add automatic acceptance tests, now we have a suite of 4000 tests that rum in parallel in about 30 minutes.
If interested I can link a report of our transformation report.
We have started also with unit tests, but they are more difficult and less accepted by some team members...
"If interested I can link a report of our transformation report." -> yes, you can write the steps in your transformation journey, you can link your transformation report, I'm looking forward to reading it!
These are great results, so your automated Acceptance Tests run in 40 mins, which is good, it fits in within Dave Farley's recommended max 60 min threshold.
1. How did you introduce the acceptance tests? I see there's a lot of them (4,000), so interested to hear how you prioritized what scenarios to cover, what sequence, etc.
2. How did you handle the culture and behavioral changes?
3. What's the current approach your team uses in introducing unit tests? Which challenges are faced?
We a working on embedded systems, so we need to tun our acceptance test o the real HW, so we build an array of about 30 devices.
In the first our suite took about 14 hours to finish,
we work to reduce them under the 1 hour.
we added more devices to our test environment and we worked to make the test parallel.
1.As our device has a lot of different features we introduce few test each feature, staring from te most important,.
WE also check in our bug tracker in order to find the most problematic modules, and prioritized that tests.
Than we started to add new tests for each new feature as soon the feature was finished (I know this is not BDD or TDD, but great step comparing to manual test).
The culture was the hardest part, CD needs discipline so a change in culture and behaviour. At the beginning , when we introduced test in parallel with development , developer was not so happy to be interrupted by QA people when they found defect.
Than we start introducing automated test, I need to convince management that automated test was way better, faster and reliable that manual tests, they have big bias in automation tests: they was thinking: automation tests are software, who test the tests?
So I started automating few tests as an experiment, I choose the most frequent bugs from our bug tracker, And I demonstrate that the tests was able to intercept known bugs and also not already known bugs.
They started accepting automating tests. Then also some of my collegues (developer) that was not so happy about tests start to accept them.
Then at certain point we removed the QA team and merged with the dev teams, now tests, both unit and acceptance are reponsability of the devs teams.
In the team we decided that is our priority to take out test suite stable, so every morning in out stand up meeting we check the tests result and assign with priority the fix of broken or flaky tests and of course of defect found defect.
Our acceptance test uses a 4 layer approach (see Dave Farley)
Here we have some disagreement because I’d like to have small tests well written and well structured, so that we can use them as live documentation and easyer to maintain, but some devs think that this is too much for effort for test…
I’d like to have a more BDD approach, writing the scenarios (executable specification) before writing the code, but for now it is impossible, I made some dojo and experiment for demonstrating that is possible, but averybody tell me that they simply cant test something that isn’t implemntented.. I’m still workking on that.
Unitt test: we started also with unit test, but it was more difficult: the code was very difficult to test, we hade very low competence in testing and no idea about test double. It’s not easy to build a unittest framework in our C/C++ (cross compiled)environment.
Now we build up a great famework based on googletest, that make easy to create testdouble in C and C++, moking also system call..
the unit test built with cross compiler and runned in the dev PC via the ARM QEMU emulator.
SO we now have overcome the technological problems, bur we still have problem with units tests:
Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test.
Some time our code is still not easily testable.
I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped.
Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD.
I suppose I needs to be patient and persistent; changes are never easy.
(Part 2/2) Therefore, to zoom into Unit Testing, could you elaborate some further details about the following
1. "Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test."
What are the arguments they say?
2. "Some time our code is still not easily testable."
What are some examples of code that is not testable?
3. "I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped."
When you started the initiative what happened...
4. "Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD."
Is that for the existing codebase or new codebase, what caused the developers to write more unit tests, and to try TDD...
- Acceptance Testing is well accepted by the teams, and currently it's giving the teams good results
- Unit Testing is not yet accepted, or not understand, so I'd like to zoom further into that topic, because I find in so many companies there is aversion towards unit testing
When you refactor a legacy system and start with black-box tests first, I believe there will be a lot more of those tests in the test suite than there should be in the finished solution. Won't you get to a point where your test suite is distributed more as a ice cream cone instead of a pyramid ? Is there a step to "transfer" those E2E safety net tests to the lower stack of the pyramid ?
For one moment, we'll ignore the topic of legacy. My memory of Dave Farley (on some YouTube video) was talking about having thousands of Acceptance Tests (I wish I could find timestamp reference again, I may rewatch "Acceptance Testing for Continuous Delivery" or others to try to find it again), and that's why he even put in effort in optimizing system startup...
Now coming back to the topic of legacy system:
1. True, by starting out with blackbox tests first, yes, we are starting an ice cream cone. The benefit is that we get end user level feedback, but the feedback loop is very slow
2. I'd start with E2E tests (https://journal.optivem.com/p/e2e-tests-in-legacy-code), then can rewrite them as acceptance tests (that's what I'm having in upcoming posts in my posting schedule), and then after all our E2E tests have been converted into acceptance tests (and additionally we retain very few E2E tests), so we don't do double maintenance. We are still at the ice cream cone.
3. Then I'd go into the component tests. Here is where there is decision process:
3.a. Are the current component boundaries roughly ok? Then we proceed to write component tests. To do that, take a high priority acceptance test, decompose the behavior across components, write the respective comppnent tests. Thus, note we're not inventing component tests, we're getting them through decomposition of acceptance tests
3.b. Are the current component boundaries horrible, do we have intention of completely changing the component boundaries in a substantial way? Then I'd do that first (I then wouldn't write the component tests if we were to completely throw them away)
4. Now go on a component-by-component basis, choose a component with the highest business complexity, introduce HA, then introduce unit tests & narrow integration tests.
So yes, we could say:
In step 1. & step 2. we have the ice cream cone (direct transfer E2E -> acceptance), and only in step 3. & 4. we start forming some pyramid (decompose acceptance tests into component tests, then component tests into unit & narrow integration tests)
When I first started, there were, and i am not kidding, 0 tests in the whole application.
So we started with the black-box first and started reworking.
Everytime I found an error while testing with business after a change, I added another test and also checked if that had to be a black-box test anymore, or if I could get more specific.
But in the end I ended up with a lot of black-box tests, that also were slow. So obviously I also cleaned them up after wards. The more I got "newish" code in that was created with TDD, the less need I had to have such high level tests.
Achieving this in 6 months is impressive!
Yes, it's a very short time frame. We can also see the big reduction in the code size too.
> Unfortunately, many teams make the mistake of jumping straight into Unit Tests. That is very dangerous - if we jump straight into Unit Tests (with no high level tests), then as we break dependencies and introduce unit tests, we could cause regression bugs at the system level!
Welp, I wish I thought about this or read this 2 years ago...
Thanks for clear explanation! Maybe I will be able to come back with a case study to you in a few months :D
Looking forward to hearing about your case study if you write it!
It was a big discovery too. I used to be very much into starting with Unit Tests, because that's also what I saw in refactoring books, and people like Uncle Bob also had big emphasis on unit tests... later I realized that Acceptance Tests is the low-hanging fruit, it's safest and faster to bring in Acceptance Tests and they give us highest ROI... then later we can move onto other tests...
P.S. Even books like "Working Effectively with Legacy Code" are mainly all about introducing Unit Tests.. no mention of Acceptance Tests... that's where I differ.
Great & informative post!
I think Approval testing (Characterization testing) fall under the umbrella of system level tests and can be used in such cases, right?
We can have Characterization Testing at any level. So yes, if we write System Tests based on the current behavior of the system (i.e., we take snapshots of how the system behaves, not how it should behave), then those are Characterization Tests at the System Level. We could also go to the Component Level, see how each component behaves, write tests - those are then Characterization Tests at the Component Level. Similarly, at Unit Level.
Even though Feathers mainly illustrates Characterization Tests at the Unit Level, I view them as existing at all levels.
Amazing! 3M LoCs in 6 months? That's almost one line of code per second!
Yes, it's a great achievement.
P.S. I've found also similar case when I needed to do a rewrite - I went back to the requirements, and then created a new implementation that was much simpler than the old implementation.
If you put it that way it sounds really inside 🙈 It didn't really felt like it at all.
As Valentina already mentioned: As I added more implementation, it also came with a lot a new tests, so its not the "3 million business implementation" you could say half if not even more is just lines of code for tests.
I probably should take a deepdive and really compare the business logic before and after.
That would also be interesting to see alongside the overall 3M LoCs.
Hi all, and a Great thankyou to Valentina.
I'm a teach lead, and some years ago we start a Continuous Delivery transformation journey.
Our application Is a iiot gateway Linux embedded device
We change lots of things in our brhaviour, we had only manual test then we started to add automatic acceptance tests, now we have a suite of 4000 tests that rum in parallel in about 30 minutes.
If interested I can link a report of our transformation report.
We have started also with unit tests, but they are more difficult and less accepted by some team members...
Thanks for sharing your story!
"If interested I can link a report of our transformation report." -> yes, you can write the steps in your transformation journey, you can link your transformation report, I'm looking forward to reading it!
Hi Valentina, here our transformation process, it took a veru hay hard work,
the most difficult part was the culture and behaviour change:
We work to have automates test, we start with acceptance (4 layer): that test in parallel an array of 40 real devices.
gherking --> behave python
In the transformation we introduces also unit tests, but we had few of it we have an inverted pyramid.
In this òast year we have started to write more unit, we are learning to use test doubles and write a more modular code.
We had a long run transformation:
from Vmodel ---> Continuos delivery.
from late pahse manual test --> frequent and early manual test
from manual test --> continuous automatic accepting test
continuous automatic accepting test (14 hours)---> fast parallel accepting test (40 minutes)
see the full story here:
https://garygruver.com/docs/whitepapers/Carlo-Gavazzi-Controls-Black-Belt-Case-Study.pdf
These are great results, so your automated Acceptance Tests run in 40 mins, which is good, it fits in within Dave Farley's recommended max 60 min threshold.
1. How did you introduce the acceptance tests? I see there's a lot of them (4,000), so interested to hear how you prioritized what scenarios to cover, what sequence, etc.
2. How did you handle the culture and behavioral changes?
3. What's the current approach your team uses in introducing unit tests? Which challenges are faced?
Hi Valentina,
We a working on embedded systems, so we need to tun our acceptance test o the real HW, so we build an array of about 30 devices.
In the first our suite took about 14 hours to finish,
we work to reduce them under the 1 hour.
we added more devices to our test environment and we worked to make the test parallel.
1.As our device has a lot of different features we introduce few test each feature, staring from te most important,.
WE also check in our bug tracker in order to find the most problematic modules, and prioritized that tests.
Than we started to add new tests for each new feature as soon the feature was finished (I know this is not BDD or TDD, but great step comparing to manual test).
The culture was the hardest part, CD needs discipline so a change in culture and behaviour. At the beginning , when we introduced test in parallel with development , developer was not so happy to be interrupted by QA people when they found defect.
Than we start introducing automated test, I need to convince management that automated test was way better, faster and reliable that manual tests, they have big bias in automation tests: they was thinking: automation tests are software, who test the tests?
So I started automating few tests as an experiment, I choose the most frequent bugs from our bug tracker, And I demonstrate that the tests was able to intercept known bugs and also not already known bugs.
They started accepting automating tests. Then also some of my collegues (developer) that was not so happy about tests start to accept them.
Then at certain point we removed the QA team and merged with the dev teams, now tests, both unit and acceptance are reponsability of the devs teams.
In the team we decided that is our priority to take out test suite stable, so every morning in out stand up meeting we check the tests result and assign with priority the fix of broken or flaky tests and of course of defect found defect.
Our acceptance test uses a 4 layer approach (see Dave Farley)
Here we have some disagreement because I’d like to have small tests well written and well structured, so that we can use them as live documentation and easyer to maintain, but some devs think that this is too much for effort for test…
I’d like to have a more BDD approach, writing the scenarios (executable specification) before writing the code, but for now it is impossible, I made some dojo and experiment for demonstrating that is possible, but averybody tell me that they simply cant test something that isn’t implemntented.. I’m still workking on that.
Unitt test: we started also with unit test, but it was more difficult: the code was very difficult to test, we hade very low competence in testing and no idea about test double. It’s not easy to build a unittest framework in our C/C++ (cross compiled)environment.
Now we build up a great famework based on googletest, that make easy to create testdouble in C and C++, moking also system call..
the unit test built with cross compiler and runned in the dev PC via the ARM QEMU emulator.
SO we now have overcome the technological problems, bur we still have problem with units tests:
Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test.
Some time our code is still not easily testable.
I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped.
Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD.
I suppose I needs to be patient and persistent; changes are never easy.
I think this is way too long, sorry :-)
(Part 2/2) Therefore, to zoom into Unit Testing, could you elaborate some further details about the following
1. "Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test."
What are the arguments they say?
2. "Some time our code is still not easily testable."
What are some examples of code that is not testable?
3. "I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped."
When you started the initiative what happened...
4. "Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD."
Is that for the existing codebase or new codebase, what caused the developers to write more unit tests, and to try TDD...
1. The arguments against unit tests are:
They are useles
They slow down US
They are annoing, when I change the codebase we Need to change also the tests.
We Need to change the code only for test pourpose.
Other teams that write more FW level code (bare metal, no OS), they Just don't bother about test, they Just Say It Is impossible with Firmware.
2. Code not testable are code that interacts with the HW, but in general code that Is too coupled to be testable, both new and Legacy code.
3. When I've started the Dojo almost all my team members but 2 come with regularity to the sessions, plus we've got other Dev from other teams.
The 2 from my team that decide to not attendo are two Friends, a good developer with 3 year experience, and a junior.
They thought that unit tests and clean code are useless, they never attended.
After some month I was forced to stop from my manager, because of urgency on project milestones.
I'm anyway Happy because the people that attend was very Happy and we had a very joyfully sessions where we learned together.
I Hope to start again.
4.
The code Is mainly new code, plus some Legacy.
I think the real reasons against unit tests Is technically difficulties, luck of new skills.
In this last year we have worked hard to have a good unit tests framework for C/C++ , now we have a very Easy and flexible way to create test doubles.
(Part 1/2) Thanks for the detailed overview.
So my understanding is:
- Acceptance Testing is well accepted by the teams, and currently it's giving the teams good results
- Unit Testing is not yet accepted, or not understand, so I'd like to zoom further into that topic, because I find in so many companies there is aversion towards unit testing
When you refactor a legacy system and start with black-box tests first, I believe there will be a lot more of those tests in the test suite than there should be in the finished solution. Won't you get to a point where your test suite is distributed more as a ice cream cone instead of a pyramid ? Is there a step to "transfer" those E2E safety net tests to the lower stack of the pyramid ?
For one moment, we'll ignore the topic of legacy. My memory of Dave Farley (on some YouTube video) was talking about having thousands of Acceptance Tests (I wish I could find timestamp reference again, I may rewatch "Acceptance Testing for Continuous Delivery" or others to try to find it again), and that's why he even put in effort in optimizing system startup...
Now coming back to the topic of legacy system:
1. True, by starting out with blackbox tests first, yes, we are starting an ice cream cone. The benefit is that we get end user level feedback, but the feedback loop is very slow
2. I'd start with E2E tests (https://journal.optivem.com/p/e2e-tests-in-legacy-code), then can rewrite them as acceptance tests (that's what I'm having in upcoming posts in my posting schedule), and then after all our E2E tests have been converted into acceptance tests (and additionally we retain very few E2E tests), so we don't do double maintenance. We are still at the ice cream cone.
3. Then I'd go into the component tests. Here is where there is decision process:
3.a. Are the current component boundaries roughly ok? Then we proceed to write component tests. To do that, take a high priority acceptance test, decompose the behavior across components, write the respective comppnent tests. Thus, note we're not inventing component tests, we're getting them through decomposition of acceptance tests
3.b. Are the current component boundaries horrible, do we have intention of completely changing the component boundaries in a substantial way? Then I'd do that first (I then wouldn't write the component tests if we were to completely throw them away)
4. Now go on a component-by-component basis, choose a component with the highest business complexity, introduce HA, then introduce unit tests & narrow integration tests.
So yes, we could say:
In step 1. & step 2. we have the ice cream cone (direct transfer E2E -> acceptance), and only in step 3. & 4. we start forming some pyramid (decompose acceptance tests into component tests, then component tests into unit & narrow integration tests)
That was absolutely true.
When I first started, there were, and i am not kidding, 0 tests in the whole application.
So we started with the black-box first and started reworking.
Everytime I found an error while testing with business after a change, I added another test and also checked if that had to be a black-box test anymore, or if I could get more specific.
But in the end I ended up with a lot of black-box tests, that also were slow. So obviously I also cleaned them up after wards. The more I got "newish" code in that was created with TDD, the less need I had to have such high level tests.