I'm a teach lead, and some years ago we start a Continuous Delivery transformation journey.
Our application Is a iiot gateway Linux embedded device
We change lots of things in our brhaviour, we had only manual test then we started to add automatic acceptance tests, now we have a suite of 4000 tests…
I'm a teach lead, and some years ago we start a Continuous Delivery transformation journey.
Our application Is a iiot gateway Linux embedded device
We change lots of things in our brhaviour, we had only manual test then we started to add automatic acceptance tests, now we have a suite of 4000 tests that rum in parallel in about 30 minutes.
If interested I can link a report of our transformation report.
We have started also with unit tests, but they are more difficult and less accepted by some team members...
"If interested I can link a report of our transformation report." -> yes, you can write the steps in your transformation journey, you can link your transformation report, I'm looking forward to reading it!
These are great results, so your automated Acceptance Tests run in 40 mins, which is good, it fits in within Dave Farley's recommended max 60 min threshold.
1. How did you introduce the acceptance tests? I see there's a lot of them (4,000), so interested to hear how you prioritized what scenarios to cover, what sequence, etc.
2. How did you handle the culture and behavioral changes?
3. What's the current approach your team uses in introducing unit tests? Which challenges are faced?
We a working on embedded systems, so we need to tun our acceptance test o the real HW, so we build an array of about 30 devices.
In the first our suite took about 14 hours to finish,
we work to reduce them under the 1 hour.
we added more devices to our test environment and we worked to make the test parallel.
1.As our device has a lot of different features we introduce few test each feature, staring from te most important,.
WE also check in our bug tracker in order to find the most problematic modules, and prioritized that tests.
Than we started to add new tests for each new feature as soon the feature was finished (I know this is not BDD or TDD, but great step comparing to manual test).
The culture was the hardest part, CD needs discipline so a change in culture and behaviour. At the beginning , when we introduced test in parallel with development , developer was not so happy to be interrupted by QA people when they found defect.
Than we start introducing automated test, I need to convince management that automated test was way better, faster and reliable that manual tests, they have big bias in automation tests: they was thinking: automation tests are software, who test the tests?
So I started automating few tests as an experiment, I choose the most frequent bugs from our bug tracker, And I demonstrate that the tests was able to intercept known bugs and also not already known bugs.
They started accepting automating tests. Then also some of my collegues (developer) that was not so happy about tests start to accept them.
Then at certain point we removed the QA team and merged with the dev teams, now tests, both unit and acceptance are reponsability of the devs teams.
In the team we decided that is our priority to take out test suite stable, so every morning in out stand up meeting we check the tests result and assign with priority the fix of broken or flaky tests and of course of defect found defect.
Our acceptance test uses a 4 layer approach (see Dave Farley)
Here we have some disagreement because I’d like to have small tests well written and well structured, so that we can use them as live documentation and easyer to maintain, but some devs think that this is too much for effort for test…
I’d like to have a more BDD approach, writing the scenarios (executable specification) before writing the code, but for now it is impossible, I made some dojo and experiment for demonstrating that is possible, but averybody tell me that they simply cant test something that isn’t implemntented.. I’m still workking on that.
Unitt test: we started also with unit test, but it was more difficult: the code was very difficult to test, we hade very low competence in testing and no idea about test double. It’s not easy to build a unittest framework in our C/C++ (cross compiled)environment.
Now we build up a great famework based on googletest, that make easy to create testdouble in C and C++, moking also system call..
the unit test built with cross compiler and runned in the dev PC via the ARM QEMU emulator.
SO we now have overcome the technological problems, bur we still have problem with units tests:
Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test.
Some time our code is still not easily testable.
I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped.
Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD.
I suppose I needs to be patient and persistent; changes are never easy.
(Part 2/2) Therefore, to zoom into Unit Testing, could you elaborate some further details about the following
1. "Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test."
What are the arguments they say?
2. "Some time our code is still not easily testable."
What are some examples of code that is not testable?
3. "I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped."
When you started the initiative what happened...
4. "Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD."
Is that for the existing codebase or new codebase, what caused the developers to write more unit tests, and to try TDD...
- Acceptance Testing is well accepted by the teams, and currently it's giving the teams good results
- Unit Testing is not yet accepted, or not understand, so I'd like to zoom further into that topic, because I find in so many companies there is aversion towards unit testing
Hi all, and a Great thankyou to Valentina.
I'm a teach lead, and some years ago we start a Continuous Delivery transformation journey.
Our application Is a iiot gateway Linux embedded device
We change lots of things in our brhaviour, we had only manual test then we started to add automatic acceptance tests, now we have a suite of 4000 tests that rum in parallel in about 30 minutes.
If interested I can link a report of our transformation report.
We have started also with unit tests, but they are more difficult and less accepted by some team members...
Thanks for sharing your story!
"If interested I can link a report of our transformation report." -> yes, you can write the steps in your transformation journey, you can link your transformation report, I'm looking forward to reading it!
Hi Valentina, here our transformation process, it took a veru hay hard work,
the most difficult part was the culture and behaviour change:
We work to have automates test, we start with acceptance (4 layer): that test in parallel an array of 40 real devices.
gherking --> behave python
In the transformation we introduces also unit tests, but we had few of it we have an inverted pyramid.
In this òast year we have started to write more unit, we are learning to use test doubles and write a more modular code.
We had a long run transformation:
from Vmodel ---> Continuos delivery.
from late pahse manual test --> frequent and early manual test
from manual test --> continuous automatic accepting test
continuous automatic accepting test (14 hours)---> fast parallel accepting test (40 minutes)
see the full story here:
https://garygruver.com/docs/whitepapers/Carlo-Gavazzi-Controls-Black-Belt-Case-Study.pdf
These are great results, so your automated Acceptance Tests run in 40 mins, which is good, it fits in within Dave Farley's recommended max 60 min threshold.
1. How did you introduce the acceptance tests? I see there's a lot of them (4,000), so interested to hear how you prioritized what scenarios to cover, what sequence, etc.
2. How did you handle the culture and behavioral changes?
3. What's the current approach your team uses in introducing unit tests? Which challenges are faced?
Hi Valentina,
We a working on embedded systems, so we need to tun our acceptance test o the real HW, so we build an array of about 30 devices.
In the first our suite took about 14 hours to finish,
we work to reduce them under the 1 hour.
we added more devices to our test environment and we worked to make the test parallel.
1.As our device has a lot of different features we introduce few test each feature, staring from te most important,.
WE also check in our bug tracker in order to find the most problematic modules, and prioritized that tests.
Than we started to add new tests for each new feature as soon the feature was finished (I know this is not BDD or TDD, but great step comparing to manual test).
The culture was the hardest part, CD needs discipline so a change in culture and behaviour. At the beginning , when we introduced test in parallel with development , developer was not so happy to be interrupted by QA people when they found defect.
Than we start introducing automated test, I need to convince management that automated test was way better, faster and reliable that manual tests, they have big bias in automation tests: they was thinking: automation tests are software, who test the tests?
So I started automating few tests as an experiment, I choose the most frequent bugs from our bug tracker, And I demonstrate that the tests was able to intercept known bugs and also not already known bugs.
They started accepting automating tests. Then also some of my collegues (developer) that was not so happy about tests start to accept them.
Then at certain point we removed the QA team and merged with the dev teams, now tests, both unit and acceptance are reponsability of the devs teams.
In the team we decided that is our priority to take out test suite stable, so every morning in out stand up meeting we check the tests result and assign with priority the fix of broken or flaky tests and of course of defect found defect.
Our acceptance test uses a 4 layer approach (see Dave Farley)
Here we have some disagreement because I’d like to have small tests well written and well structured, so that we can use them as live documentation and easyer to maintain, but some devs think that this is too much for effort for test…
I’d like to have a more BDD approach, writing the scenarios (executable specification) before writing the code, but for now it is impossible, I made some dojo and experiment for demonstrating that is possible, but averybody tell me that they simply cant test something that isn’t implemntented.. I’m still workking on that.
Unitt test: we started also with unit test, but it was more difficult: the code was very difficult to test, we hade very low competence in testing and no idea about test double. It’s not easy to build a unittest framework in our C/C++ (cross compiled)environment.
Now we build up a great famework based on googletest, that make easy to create testdouble in C and C++, moking also system call..
the unit test built with cross compiler and runned in the dev PC via the ARM QEMU emulator.
SO we now have overcome the technological problems, bur we still have problem with units tests:
Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test.
Some time our code is still not easily testable.
I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped.
Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD.
I suppose I needs to be patient and persistent; changes are never easy.
I think this is way too long, sorry :-)
(Part 2/2) Therefore, to zoom into Unit Testing, could you elaborate some further details about the following
1. "Some dev don’t belive in unit tests and think that they are slowing them down so they prefer acceptance test."
What are the arguments they say?
2. "Some time our code is still not easily testable."
What are some examples of code that is not testable?
3. "I’ve started some internal dojo to show clean code, refactor, Unit test, TDD, but some dev are not interested, and managment does not sponsor that initiative . so I stopped."
When you started the initiative what happened...
4. "Anyway step by step things goes better also for unit tests, in current product we are writing a lot more unit test and some time some dev try also TDD."
Is that for the existing codebase or new codebase, what caused the developers to write more unit tests, and to try TDD...
1. The arguments against unit tests are:
They are useles
They slow down US
They are annoing, when I change the codebase we Need to change also the tests.
We Need to change the code only for test pourpose.
Other teams that write more FW level code (bare metal, no OS), they Just don't bother about test, they Just Say It Is impossible with Firmware.
2. Code not testable are code that interacts with the HW, but in general code that Is too coupled to be testable, both new and Legacy code.
3. When I've started the Dojo almost all my team members but 2 come with regularity to the sessions, plus we've got other Dev from other teams.
The 2 from my team that decide to not attendo are two Friends, a good developer with 3 year experience, and a junior.
They thought that unit tests and clean code are useless, they never attended.
After some month I was forced to stop from my manager, because of urgency on project milestones.
I'm anyway Happy because the people that attend was very Happy and we had a very joyfully sessions where we learned together.
I Hope to start again.
4.
The code Is mainly new code, plus some Legacy.
I think the real reasons against unit tests Is technically difficulties, luck of new skills.
In this last year we have worked hard to have a good unit tests framework for C/C++ , now we have a very Easy and flexible way to create test doubles.
3. Great to hear that most of your team members attended regularly! What kind of exercises did you use for the Dojo, and what made the people happy?
"After some month I was forced to stop from my manager, because of urgency on project milestones."
I'm interested if again you plan to again initiate the initiative, and if yes, how you plan to communicate it.
4. What kind of test doubles do you use? Which skillsets do you see as needed by developers?
(Part 1/2) Thanks for the detailed overview.
So my understanding is:
- Acceptance Testing is well accepted by the teams, and currently it's giving the teams good results
- Unit Testing is not yet accepted, or not understand, so I'd like to zoom further into that topic, because I find in so many companies there is aversion towards unit testing