Indeed, a common misunderstanding is people think CI/CD is for DevOps Engineers... not recognition that CI/CD is actually for the whole team. Automated Testing is an important part of CI/CD. The Pipeline is useful to coordinate the execution of Automated Tests on an independent machine.
We need to separate CI from CD. For effective testing we need a CI Pipeline. But to take the pipleline all the way to CD without a complete test suite is to continuously deliever crap.
Valentina, I see where you are going here. Some tests require some kind of deployment to a QA/Staging environment. If your pipeline includes these tests (and it certainly should), then your pipeline must include deployment. Are you calling this CD? I have always considered CD to be where the automated pipeline takes you to a point where you can deliver with flick of a switch.
Yes, to run Acceptance Tests (in the Acceptance Test Enviornment) it means we’d need to have automated deployment, then run the smoke tests, acceptance tests… If that passes, we can safely deliver to UAT, Production, etc…
“I have always considered CD to be where the automated pipeline takes you to a point where you can deliver with flick of a switch.“ - yes, that’s how I see it too.
Yes, it's the truth in terms of success in practice.
Unfortunately, some teams might be jumping into test automation without the pipeline first... and unfortunately, many books separate testing (books about testing) and pipeline (in books about CD). That's why I decided to integrate them in my articles, so that teams see we need both.
You can have lot of tests, but if you can't run them frequentely ana in a consistente way, you can't grow both code and tests with a clear quality signal
We have faster running tests that we can run locally (Unit Tests, Component Tests) but we still need to have assurance they can run independently on another machine, hence need them to be checked by the Pipeline.
We have slower running tests (Acceptance Tests) that we’d tend to therefore run on the Pipeline only, but rarely locally
Correct, we should aim for acceptance tests to be < 1hr duration (as recommended by Farley), which is possible when we write acceptance tests such that many can be executed in parallel (which needs tests that have data isolation).
In this way, we can get feedback relatively fast on the pipeline.
P.S. Now the difference of acceptance tests on the pipeline vs locally, is that on the pipeline we'd deploy the system & then run acceptance testing... whereas locally we'd use Docker Compose & run acceptance testing.
The part where I do see the value of running Acceptance Test locally is the following:
- Within ATDD cycle, running a specific Acceptance Test to see it RED
- If Pipeline is failing due to some Acceptance Test, I may run it locally too
CI/CD isn't just for DevOps!
Indeed, a common misunderstanding is people think CI/CD is for DevOps Engineers... not recognition that CI/CD is actually for the whole team. Automated Testing is an important part of CI/CD. The Pipeline is useful to coordinate the execution of Automated Tests on an independent machine.
We need to separate CI from CD. For effective testing we need a CI Pipeline. But to take the pipleline all the way to CD without a complete test suite is to continuously deliever crap.
What's the Pipeline structure that you current adopt, and which test types do you have in the different stages?
Valentina, I see where you are going here. Some tests require some kind of deployment to a QA/Staging environment. If your pipeline includes these tests (and it certainly should), then your pipeline must include deployment. Are you calling this CD? I have always considered CD to be where the automated pipeline takes you to a point where you can deliver with flick of a switch.
Yes, to run Acceptance Tests (in the Acceptance Test Enviornment) it means we’d need to have automated deployment, then run the smoke tests, acceptance tests… If that passes, we can safely deliver to UAT, Production, etc…
“I have always considered CD to be where the automated pipeline takes you to a point where you can deliver with flick of a switch.“ - yes, that’s how I see it too.
That's a unvarnished truth !
Yes, it's the truth in terms of success in practice.
Unfortunately, some teams might be jumping into test automation without the pipeline first... and unfortunately, many books separate testing (books about testing) and pipeline (in books about CD). That's why I decided to integrate them in my articles, so that teams see we need both.
I'd like to suggest a Great book by Gary Gruver: Engineering the digital Engineering!
It explain how to grow test automation at a sustemible pace from the processo point of view.
It's all about fast feedbacks.
You can have lot of tests, but if you can't run them frequentely ana in a consistente way, you can't grow both code and tests with a clear quality signal
Exactly, it’s about feedback.
We have faster running tests that we can run locally (Unit Tests, Component Tests) but we still need to have assurance they can run independently on another machine, hence need them to be checked by the Pipeline.
We have slower running tests (Acceptance Tests) that we’d tend to therefore run on the Pipeline only, but rarely locally
Acceptance Is slower thank unit so It Is right that they take longher.
But of they take too much for examples 8 hours, you see the effect of your modification After a day, this way Is very difficult to make them stabile.
You are right that running them on different environment help a lot
Correct, we should aim for acceptance tests to be < 1hr duration (as recommended by Farley), which is possible when we write acceptance tests such that many can be executed in parallel (which needs tests that have data isolation).
In this way, we can get feedback relatively fast on the pipeline.
P.S. Now the difference of acceptance tests on the pipeline vs locally, is that on the pipeline we'd deploy the system & then run acceptance testing... whereas locally we'd use Docker Compose & run acceptance testing.
The part where I do see the value of running Acceptance Test locally is the following:
- Within ATDD cycle, running a specific Acceptance Test to see it RED
- If Pipeline is failing due to some Acceptance Test, I may run it locally too