Legacy Code often doesn't have a Pipeline. That's why we have built a Pipeline, so that we have the foundation for automated test execution to support TDD.
(1) A developer might write some failing test, but to make the test pass, their quick and dirty implementation might be to use something from their local computer, perhaps referencing C:\JohnSmith\some_file.txt in their source code. Their test passes - on their computer. But it would fail everywhere else. Without a pipeline, they'd block their whole team, when everyone pulls this version of code.
(2) A developer might forget to run the full test suite before committing code. Without a pipeline, when their team members pull the code, the test suite would be failing on everyone else's computer.
Thus, yes, we could practice TDD without having a Pipeline, but we lose the assurance of having the independent assurance of running tests on a Build Server.
"UAT Stage - deploying Docker Images to UAT Environment, running Smoke Tests, so that then QA Engineers can do Manual QA Testing."
If I only run UAT stage on a docker image that has only passed acceptance stage. Then do we still need to do smoke test? because acceptance stage will do it anyway in acceptance environment. Then running them again in UAT stage or other stages is like running same test twice. Can we skip that?
Good question. Yes, for Smoke Tests we do it per environment. So this means:
- When we deploy to Acceptance Environment, we run Smoke Tests on Acceptance Environment
- When we deploy to UAT Environment, we run Smoke Tests on UAT Environment
- When we deploy to Production Environment, we run Smoke Tests on Production Environment
(If we have additional environments, such as Security Testing Environment, Performance Testing Environment, as soon as we deploy, we run Smoke Tests)
--------------
"If I only run UAT stage on a docker image that has only passed acceptance stage. Then do we still need to do smoke test? because acceptance stage will do it anyway in acceptance environment. Then running them again in UAT stage or other stages is like running same test twice."
Exactly, we are running the *same* test on *different* environments.
"Can we skip that?"
Cannot skip, because of environment differences. There might be non-intentional environmental differences, whereby the Smoke Test passes on Acceptance Environment, but fails on UAT Environment, or fails on Production Environment.
So just because the Smoke Test passes on one environment, it does *not* mean that it will pass on the other environments, that's why we run it per environment, to know that the application is up-and-running on each environment.
Hi Valentina, great series! I have a couple of pipeline-related questions:
Are the Acceptance stage and the Release stages (UAT, Production) each broken into separate levels—component-level and system-level—within the same stage?
Or do we run all component-level stages (Acceptance, UAT, Production) first, and then system-level stages afterward?
I think it's the second option because we manually select which “release candidate” components should move on to the system-level testing.
If component-level UAT and system-level UAT exist, do they use the same UAT environment?
Similarly, does each component in “Production” run in the same environment as the system-level Production deployment?
It would be great if you get a chance to make the Sandbox Project, where you'd have an opportunity to implement the pipeline architecture, then I'll be able to provide feedback to ensure that we're aligned (and it may spark even further discussion).
"Are the Acceptance stage and the Release stages (UAT, Production) each broken into separate levels—component-level and system-level—within the same stage?"
VJ: Commit Stage is component-level.
Acceptance Stage, UAT Stage and Production Stages are system-level
----------------------------
"Or do we run all component-level stages (Acceptance, UAT, Production) first, and then system-level stages afterward?"
VJ: *Only* the Commit Stage is component-level. The others (Acceptance, UAT, Production) are system-level.
----------------------------
"I think it's the second option because we manually select which “release candidate” components should move on to the system-level testing."
VJ: For each Component, its Component Commit Stage generates an artifact, which becomes a Component RC for the Acceptance Stage (i.e. for system-level testing).
----------------------------
"If component-level UAT and system-level UAT exist, do they use the same UAT environment?"
VJ: There is only system-level UAT that exists, because UAT is from end-user perspective (Acceptance, UAT, Production are all system-only). There is no component-level UAT because components are technical, they are not user-facing. Now regarding system-level testing:
- E2E Environment: Run Acceptance Stage - Smoke Tests, External System Contract Tests, E2E Tests
- UAT Environment: Run UAT Stage - Smoke Tests... (then QA Engineers do exploratory testing)
- Production Environment: Run Production Stage - Smoke Tests... (the Users can use the app)
----------------------------
"Similarly, does each component in “Production” run in the same environment as the system-level Production deployment?"
VJ: As part of the Production Stage, it means we deploy all components onto the Production Environment, and run Smoke Tests immediately after deployment.
1. In the COMMIT STAGE, we'll run the fast-running & isolated tests, including: Unit Level (Unit Tests & Narrow Integration Tests) and Component Level (Component Tests & Contract Tests). The Commit Stage is executed on *every* commit. It's at component-level, for each component we have an independent Commit Stage.
2. In the ACCEPTANCE STATE, we run system level tests: Acceptance Tests, External System Contract Tests, E2E Tests. These system tests are run on the deployed version of the system that contains the *latest* artifact versions of each component, i.e. the latest "release candidate" components.
The COMMIT STAGE is run in-memory.
The ACCEPTANCE STAGE is run as follows:
- Acceptance Tests are executed on Acceptance Environment
- External System Contract Tests & E2E Tests are executed on E2E Environment
Then, for the combination of the RC components that passed the ACCEPTANCE STAGE, they can be selected by QA Engineer to be released to UAT Environment, i.e. the UAT Stage. Then, they can select for those RC components to be deployed to Production, i.e. the Production Stage.
Is it possible to do TDD without a pipeline?
Suppose we practice TDD but there's no pipeline.
(1) A developer might write some failing test, but to make the test pass, their quick and dirty implementation might be to use something from their local computer, perhaps referencing C:\JohnSmith\some_file.txt in their source code. Their test passes - on their computer. But it would fail everywhere else. Without a pipeline, they'd block their whole team, when everyone pulls this version of code.
(2) A developer might forget to run the full test suite before committing code. Without a pipeline, when their team members pull the code, the test suite would be failing on everyone else's computer.
Thus, yes, we could practice TDD without having a Pipeline, but we lose the assurance of having the independent assurance of running tests on a Build Server.
Hi Valentina, a question :
"UAT Stage - deploying Docker Images to UAT Environment, running Smoke Tests, so that then QA Engineers can do Manual QA Testing."
If I only run UAT stage on a docker image that has only passed acceptance stage. Then do we still need to do smoke test? because acceptance stage will do it anyway in acceptance environment. Then running them again in UAT stage or other stages is like running same test twice. Can we skip that?
Hi MY,
Good question. Yes, for Smoke Tests we do it per environment. So this means:
- When we deploy to Acceptance Environment, we run Smoke Tests on Acceptance Environment
- When we deploy to UAT Environment, we run Smoke Tests on UAT Environment
- When we deploy to Production Environment, we run Smoke Tests on Production Environment
(If we have additional environments, such as Security Testing Environment, Performance Testing Environment, as soon as we deploy, we run Smoke Tests)
--------------
"If I only run UAT stage on a docker image that has only passed acceptance stage. Then do we still need to do smoke test? because acceptance stage will do it anyway in acceptance environment. Then running them again in UAT stage or other stages is like running same test twice."
Exactly, we are running the *same* test on *different* environments.
"Can we skip that?"
Cannot skip, because of environment differences. There might be non-intentional environmental differences, whereby the Smoke Test passes on Acceptance Environment, but fails on UAT Environment, or fails on Production Environment.
So just because the Smoke Test passes on one environment, it does *not* mean that it will pass on the other environments, that's why we run it per environment, to know that the application is up-and-running on each environment.
Hi Valentina, great series! I have a couple of pipeline-related questions:
Are the Acceptance stage and the Release stages (UAT, Production) each broken into separate levels—component-level and system-level—within the same stage?
Or do we run all component-level stages (Acceptance, UAT, Production) first, and then system-level stages afterward?
I think it's the second option because we manually select which “release candidate” components should move on to the system-level testing.
If component-level UAT and system-level UAT exist, do they use the same UAT environment?
Similarly, does each component in “Production” run in the same environment as the system-level Production deployment?
(Part 4/4) I released some parts of my diagram were perhaps confusing, so I updated it now.
(Part 3/4) I appreciated the deep questions you asked.
I gave my best to answer it, though the best possible answers, to ensure that we have a mutual understanding, is it you have a chance to create a Sandbox Project https://journal.optivem.com/p/legacy-code-sandbox-project
It would be great if you get a chance to make the Sandbox Project, where you'd have an opportunity to implement the pipeline architecture, then I'll be able to provide feedback to ensure that we're aligned (and it may spark even further discussion).
Thanks for your questions, and have a good day!
(Part 2/4) To answer your questions:
----------------------------
"Are the Acceptance stage and the Release stages (UAT, Production) each broken into separate levels—component-level and system-level—within the same stage?"
VJ: Commit Stage is component-level.
Acceptance Stage, UAT Stage and Production Stages are system-level
----------------------------
"Or do we run all component-level stages (Acceptance, UAT, Production) first, and then system-level stages afterward?"
VJ: *Only* the Commit Stage is component-level. The others (Acceptance, UAT, Production) are system-level.
----------------------------
"I think it's the second option because we manually select which “release candidate” components should move on to the system-level testing."
VJ: For each Component, its Component Commit Stage generates an artifact, which becomes a Component RC for the Acceptance Stage (i.e. for system-level testing).
----------------------------
"If component-level UAT and system-level UAT exist, do they use the same UAT environment?"
VJ: There is only system-level UAT that exists, because UAT is from end-user perspective (Acceptance, UAT, Production are all system-only). There is no component-level UAT because components are technical, they are not user-facing. Now regarding system-level testing:
- Acceptance Environment: Run Acceptance Stage - Smoke Tests, Acceptance Tests
- E2E Environment: Run Acceptance Stage - Smoke Tests, External System Contract Tests, E2E Tests
- UAT Environment: Run UAT Stage - Smoke Tests... (then QA Engineers do exploratory testing)
- Production Environment: Run Production Stage - Smoke Tests... (the Users can use the app)
----------------------------
"Similarly, does each component in “Production” run in the same environment as the system-level Production deployment?"
VJ: As part of the Production Stage, it means we deploy all components onto the Production Environment, and run Smoke Tests immediately after deployment.
(Part 1/4) Thanks Guy! As a high level overview:
1. In the COMMIT STAGE, we'll run the fast-running & isolated tests, including: Unit Level (Unit Tests & Narrow Integration Tests) and Component Level (Component Tests & Contract Tests). The Commit Stage is executed on *every* commit. It's at component-level, for each component we have an independent Commit Stage.
2. In the ACCEPTANCE STATE, we run system level tests: Acceptance Tests, External System Contract Tests, E2E Tests. These system tests are run on the deployed version of the system that contains the *latest* artifact versions of each component, i.e. the latest "release candidate" components.
The COMMIT STAGE is run in-memory.
The ACCEPTANCE STAGE is run as follows:
- Acceptance Tests are executed on Acceptance Environment
- External System Contract Tests & E2E Tests are executed on E2E Environment
Then, for the combination of the RC components that passed the ACCEPTANCE STAGE, they can be selected by QA Engineer to be released to UAT Environment, i.e. the UAT Stage. Then, they can select for those RC components to be deployed to Production, i.e. the Production Stage.