Subscribe now

The Unit Test Passed. The Customer Was Overcharged.

Your unit tests can’t catch this category of bug. Here’s proof.

Before, I showed you how Marco’s team shipped a broken tax calculation despite having green tests across the board.

I received this question:

“Can you show me the actual code? I want to prove this to my team.”

So here it is. A concrete example you can drop into any conversation about test strategy.

The Setup

We have a simple e-commerce system. When a customer places an order, the system needs to look up the tax rate from an external Tax API and calculate the total.

public class TaxGateway {
    private final HttpClient httpClient;

    public double getTaxRate(String countryCode) {
        HttpResponse response = httpClient.get(
            "https://tax-api.example.com/rates/" + countryCode
        );
        JsonObject json = JsonParser.parse(response.getBody());
        return json.getDouble("rate");
    }
}

public class OrderService {
    private final TaxGateway taxGateway;

    public Order placeOrder(String countryCode, double unitPrice, int quantity) {
        double basePrice = unitPrice * quantity;
        double taxRate = taxGateway.getTaxRate(countryCode);
        double taxAmount = basePrice * taxRate;
        return new Order(basePrice, taxAmount, basePrice + taxAmount);
    }
}

The code looks correct. OrderService calls TaxGateway, which makes an HTTP call to the external Tax API. Everything is wired up properly.

But there’s a hidden bug. The Tax API returns rates as whole numbers — { "rate": 8 } for 8%. The TaxGateway reads this value and returns it directly. It should divide by 100 first, but it doesn’t.

The Unit Test: GREEN ✅

@Test
void shouldApplyTaxRateFromExternalService() {
    when(taxGatewayMock.getTaxRate("US"))
        .thenReturn(0.08);

    Order order = orderService.placeOrder("US", 20.00, 5);

    assertEquals(100.00, order.getBasePrice());
    assertEquals(8.00, order.getTaxAmount());
    assertEquals(108.00, order.getTotalPrice());
}

This test passes. And it should pass — given the mock’s return value, the math is perfect.

The developer sees green. CI sees green. The PR gets approved.

But the mock is a lie. It returns 0.08 because that’s what the developer assumed TaxGateway would return. The mock replaces the entire TaxGateway — the HTTP call never happens, the JSON parsing never runs, and the missing / 100.0 is never exposed.

In production, the customer orders $100 worth of products and gets charged $900 — because 100 * 8 = 800 in tax.

And here’s the thing: you can’t fix this with more unit tests. You could unit test every single class — OrderService, TaxGateway — and they would all pass. The bug lives at the boundary between your code and the real world, and unit tests must mock that boundary. If the mock is wrong, every test built on top of it is wrong too.

The Acceptance Test: RED ❌

Now here’s the same scenario written as an acceptance test:

@Test
void shouldCalculateCorrectTotalWithCountryTaxRate() {
    scenario
        .given().product()
            .withUnitPrice("20.00")
        .and().country()
            .withCode("US")
            .withTaxRate("8%")
        .when().placeOrder()
            .withQuantity("5")
            .withCountry("US")
        .then().shouldSucceed()
        .and().order()
            .hasBasePrice("100.00")
            .hasTaxAmount("8.00")
            .hasTotalPrice("108.00");
}

This test runs against the real system — real backend, real database, real TaxGateway code — but with a stub for the external Tax API. The given().country().withTaxRate("8%") line configures the Tax API Stub to return { "rate": 8 } — exactly what the real Tax API would return.

The real TaxGateway code executes. The HTTP call fires against the Tax API Stub. The JSON is parsed. And the missing / 100.0 is exposed.

Result: FAILS.

Expected taxAmount: 8.00
Actual taxAmount:   800.00

Expected totalPrice: 108.00
Actual totalPrice:   900.00

The Tax API Stub returned { "rate": 8 }, just like the real API would. TaxGateway read the value and returned 8 directly to OrderService, which calculated 100 * 8 = 800 in tax. In the unit test, the mock hid this by returning 0.08 directly. In the acceptance test, the real code path runs and the bug is caught.

Why Unit Tests Can Never Catch This

The bug isn’t in OrderService. The math is correct. The wiring is correct. The bug is inside TaxGateway — in how it parses the external API response.

But the unit test for OrderService mocks out TaxGateway entirely. The HTTP call never fires. The JSON parsing never runs. The bug is invisible.

The unit test:

• Tests OrderService in isolation

• Replaces TaxGateway with a mock

• The mock returns what the developer assumes the gateway returns

• The real HTTP call and JSON parsing never execute

• Can’t catch a bug inside the mocked component

The acceptance test:

• Tests the full feature against the real system with stubs for external dependencies

• The real TaxGateway code runs — HTTP call, JSON parsing, everything

• The Tax API Stub behaves like the real Tax API — returning { "rate": 8 }

• Catches bugs anywhere in the chain, not just in the class under test

Unit tests verify your logic given your assumptions. Acceptance tests verify the feature works regardless of your assumptions. No amount of unit testing can catch a bug that’s hidden behind a mock.

But Wait — How Do We Know the Stub Is Correct?

Fair question. If the acceptance test uses a Tax API Stub instead of the real Tax API, couldn’t the stub itself be wrong?

That’s where contract tests come in. A contract test runs the same scenarios against both the real Tax API and the Tax API Stub, and verifies they return the same results. If the stub drifts from reality, the contract test fails.

So the full safety net looks like this:

• Unit tests verify each component’s logic in isolation

• Acceptance tests verify the feature works end-to-end, using stubs for external systems

• Contract tests verify the stubs behave like the real external systems

Each layer catches a different category of bug. Skip any one of them, and you have a blind spot.

This Isn’t an Edge Case

The bug in our example was a wrong assumption about the API response format. But this is just one of many ways external system integration breaks:

• Wrong format assumption — The Tax API returns { "rate": 8 } meaning 8%, but your code treats it as 0.08. Your mock returns whatever you assumed, so the unit test passes.

• Hardcoded value — The developer hardcoded return 0.10 inside TaxGateway while building the feature, planning to wire up the HTTP call “later.” Later never came. The unit test mocks out TaxGateway entirely, so the hardcoded value is never executed.

• Wrong field — The API returns { "taxRate": 8, "importRate": 12 } and your code reads importRate instead of taxRate. Your mock only returns the field you expected, so the unit test passes.

• Stale mapping — The API used to return { "rate": 8 } but v2 changed it to { "tax": { "rate": 8 } }. Your code still reads the top-level field. Your mock still returns the old format, so the unit test passes.

This pattern shows up constantly across all types of external integrations:

• A payment gateway that returns amounts in cents, but your code treats them as dollars.

• A shipping API that changed its response format in v2, but your mocks still return v1.

• An inventory service that returns "OUT_OF_STOCK" as a string, but your mock returns a boolean false.

• A currency API that returns rates with 6 decimal places, but your mock rounds to 2.

Every one of these passes unit tests — because the mock is the assumption. If the assumption is wrong, the mock is wrong, and the test is worthless.

Every one of these ships to production. Every one of these gets caught by acceptance tests.

The Takeaway

Unit tests answer: “Does this component do its job correctly, given my assumptions?”

Acceptance tests answer: “Does this feature work the way the customer expects?”

Contract tests answer: “Are my assumptions about external systems correct?”

These are fundamentally different questions. If you’re only asking the first one, you’re leaving the other two for your customers to answer.

Want to learn how to write acceptance tests like this?

I’m running a live workshop: Stop Shipping Bugs: Acceptance Tests Workshop.

4 hours. Two evenings. Live on Zoom, with me.

⚡ I’ll walk you through the full architecture — DSL, Drivers, how it all fits together — and by the end, you’ll have written a real acceptance test from scenario to assertion.

📅 May 25-26, 2:00-4:00 PM CET

Join the workshop →

Limited spots. Register now with the early bird discount - 100 EUR off with code EARLYBIRD100

— Valentina

Subscribe now

Hitting 100% code coverage doesn’t mean your code works. Here’s why ATDD matters.

I remember when my client’s team first got obsessed with code coverage.

“Let’s hit 90% coverage this sprint!” the manager said.

Great, right? Except… two months later, they had 95% coverage and yet the system kept breaking in production.

I learned something hard that day: coverage is just a number.

It doesn’t tell you if your code works, if your business rules are correct, or if anything is maintainable.

You can execute every line of code and still ship disasters.

Knight Capital Group – $440 million loss in 45 minutes

Here’s one incident that still keeps me up at night.

In 2012, Knight Capital Group deployed an update to their trading system.

An old, previously disabled piece of logic got re‑activated by accident.

The system started sending orders it shouldn’t have.

Within 45 minutes, they lost $440 million. Just like that.

Here’s the kicker:

No amount of code coverage—90%, 95%, or 100%—would have prevented the disaster.

Why? Because this wasn’t a line‑not‑executed problem. Only Acceptance Tests could have caught this before it hit production.

Metrics can make you feel safe. Tests that focus on behavior actually keep you safe.

Code Coverage is a Trap

Code coverage only tells you code ran.

Not that it worked.

Running code ≠ verifying behavior.

I learned: chasing metrics doesn’t make you a better developer—thinking in terms of behavior does.

Metrics ≠ Safety

Next time someone asks about coverage, ask: “Does it actually tell me my system works?”

If the answer is no, stop worrying about the percentage and start thinking about behavior.

This is exactly why I rely on Acceptance Tests. Unit Tests with 100% code coverage are not enough.

They protect you from the stuff metrics can’t see—accidental behavior changes, re-activated logic, “how did this even happen?” bugs.

If you want to learn how to write acceptance tests that make production failures boring instead of catastrophic…

🚀 Join me for Acceptance Testing (Live Training) on Wed 25th Feb (17:00 - 19:00 CET).

🏆 Register now

P.S. Regular price: $97 (free access for Optivem Journal paid subscribers - see event description for details)

Subscribe now

Subscribe now

Previously, we covered the Modern Test Pyramid - System Level, where we showed the importance of System Level tests, which included Smoke Tests, E2E Tests, and most importantly, Acceptance Tests & External System Contract Tests.

Acceptance Tests are executable Acceptance Criteria. We learnt that Acceptance Tests & External System Contract Tests, together, are a replacement for Manual System Regression Testing.

Does it matter whether we write Acceptance Tests last… or write Acceptance Tests first?

📢 On Wed 9th July 17:00 CET, I’m hosting TDD & Unit Tests - Part 4 (Live Q&A):

Register now

P.S. If you’re a Paid Member, you’ll receive a 100% discount (see Event Description for instructions on how to redeem your 100% discount).

Subscribe now

Approach 1. Writing & Executing Manual Tests Last

Most companies are NOT agile, but rather they work in a Waterfall way. They might be practicing Scrum in a non-agile way, whereby each Sprint is a mini-waterfall. (Or perhaps the whole release is a waterfall, whereby a one phase is executed over multiple sprints.)

1. PO specifies User Stories

2. PO hands over the User Stories to Developers

3. Developers try to implement the User Stories (based on their interpretation)

4. Developers hand over the Software to QA Engineers

5. QA Engineers test the User Stories (based on their interpretation)

Here’s a more detailed breakdown of the mini-waterfall:

Requirement Phase

• PO specifies User Stories

Development Phase

1. Developers try to implement the User Stories

2. Developers manually test the User Stories on their local machine

QA Phase

1. Deploy System to QA Environment

2. User Story Testing

   1. QA Engineers manually test the User Stories (from the Sprint Backlog)

   2. Developers fix any bugs reported

   3. QA Engineers manually test the fixes

   4. If QA Engineers find further bugs, then repeat the process (of manual testing & developer rework)

3. Regression Testing

   1. QA Engineers perform manual regression testing

   2. Developers fix any regression bugs reported

   3. QA Engineers manually test the fixes

   4. If QA Engineers find further bugs, then repeat the process (of manual testing & developer rework)

Problems of Manual Tests Last:

The following are the problems faced (at the end of the Sprint):

• Regression Bugs - QA Engineers discover lots of Regression Bugs. Time Wasted: Developers fixing & QA Engineers re-testing regression bugs.

• Misunderstood Requirements - Developers discover that QA Engineers had a different interpretation of requirements. Time Wasted: Late Developer rework due to misinterpretations & QA Engineers re-testing.

• Missing Scenarios - QA Engineers discover new scenarios that Developers didn’t think of. Time Wasted: Late Developer rework due to missed scenarios & QA Engineers re-testing.

Approach 2. Writing Acceptance Tests last

When companies adopt Acceptance Tests, to replace Manual Regression Tests, they tend to keep the same process of working, i.e. at the end of the Sprint, QA Engineers now write (automated) Acceptance Tests rather than Manual Tests.

Problems solved

Writing Acceptance Tests solves the problem:

• Regression Bugs are not detected by QA Engineer, but are rather visible to the Developer hence solved by the Developer (hence QA Engineer has time for reporting new bugs rather than regression bugs).

Note: Acceptance Tests help reduce regression bugs associated with User Story Acceptance Criteria. For example, if we implement some Acceptance Criteria in Sprint N, then we’ll be protected in all future Sprints (Sprint N+1, Sprint N + 2, etc.)

Problems NOT solved

However, since Acceptance Tests are written at the end of the Sprint, it does not solve requirement-related problems, where there’s a gap in requirement understanding between Developers & QA Engineers (and also PO):

• Misunderstood Requirements - This problem is still not solved!

• Missing Scenarios - This problem is still not solved!

There’s also a new problem:

• Invalid Tests - When writing the test at the end, since the functionality has already been implemented, we tend to write just enough of the test to see it GREEN. It could happen by accident, that we write an Acceptance Test that executes the System but does NOT verify anything (ZERO assertions!); we see the test is passing, and we move on… but this test is NOT testing anything at all! Alternatively, the test could have an assertion, but it might be the wrong assertion; it is not verifying what should be verified. It’s an always-passing test. This is a useless test, because it gives us a false sense of assurance; it does not protect us against regression bugs!

Approach 3. Writing Acceptance Tests first (ATDD)

How can we address the issue of requirement misunderstandings? How do we bring everyone (PO, Developers, QA Engineers) on the same page?

Requirement misunderstandings occur because requirements are often generalized, leaving room for misinterpretation and gaps in understanding.

What if we could write requirements in a very specific way, such that there is no room for misinterpretation? What if we could write requirements clearly, specifying behavioral expectations through specific inputs and expected outputs, i.e. example-based requirements?

If we wrote example-based requirements before coding, then it puts everyone on the same page. Acceptance Tests themselves are example-based requirements. Writing them before coding is known as ATDD.

Problems solved:

Writing example-based requirements (Acceptance Tests) before coding helps align the whole team regarding requirements:

• Misunderstood Requirements - Writing requirements as examples clarifies them for everyone and eliminates gaps in understanding.

• Missing Scenarios - QA Engineers can bring their creative thinking (“what if“ scenarios) to be brainstormed at the start of the Sprint, which is converted to requirements as examples, so that the whole team has a more comprehensive understanding of scenarios that need to be covered

• Invalid Tests - By writing the Acceptance Test first, we can verify that it fails and that the failure occurs for the expected reason. This assures us that the test itself is valid, that it’s testing what it should be testing.

Read more

Subscribe now

Manual System Regression Testing = Waste!

The default approach that most companies still rely on is Manual System Testing, which is done by QA Engineers at the end of the Sprint & also in preparation for the Release.

Problem #1: Slow Delivery

• Developer Time Wasted - Manual Regression Testing: During Sprints, as Developers implement User Stories, they also need to do some basic Manual Regression Testing to make sure their User Stories didn’t break anything. Generally, they do very basic Manual Regression Testing, it is not extensive, maybe just a few basic scenarios.

• QA Engineer Time Wasted - Manual Regression Testing: At the end of the Sprint, QA Engineers need to test User Stories and do some basic Manual Regression Testing. Furthermore, prior to releasing the System, they need to execute the Manual Regression Test suite to make sure there’s no regression bugs.

• Developer Time Wasted - Expensive Late Rework: Since Developers have to wait for the QA Engineers to perform the time-consuming Manual Regression Testing

The impact of the time wasted is that we have a high internal cost; we’re throwing money down the drain; we’re paying our developers & QA Engineers to waste time; the team spends more time on bugs rather than features.

Problem #2: Buggy Releases

• Customer-facing Regression Bugs: Prior to the Release, the QA Engineers should do “full“ Regression Testing (and keep on repeating that until Developers have resolved all Regression Bugs). However, in practice, that is infeasible - because Manual Regression Testing is too slow; the company can’t hire an infinite number of QA Engineers; the company can’t indefinitely postpone a release due to repeated Manual Regression Test cycles. So what’s the pragmatic solution? QA Engineers are forced to cut corners - execute only “partial“ Regression Testing, rather than “full“ Regression Testing. This causes Regression Bugs to escape to Customers, to be reported to the Customer Support Team.

The impact of customer-facing bugs is that customers become dissatisfied, leading to potential customer loss (hence lost revenue). Furthermore, it can cause poor reputation in the market, thus causing loss of potential future clients (hence losing potential future revenue).

📢 On Wed 9th July 17:00 CET, I’m hosting TDD & Unit Tests - Part 4 (Live Q&A):

Register now

P.S. If you’re a Paid Member, you’ll receive a 100% discount (see Event Description for instructions on how to redeem your 100% discount).

Subscribe now

Automated System Regression Testing = Solution

Impact #1: Faster Delivery

• Developers do NOT have to do Manual Testing of their User Stories and don’t have to do Manual Regression Testing either. This reduces Developer time spent per User Story.

• QA Engineers do NOT execute Manual Regression Testing. Instead, they just do Exploratory Testing (i.e. creative testing).

• Developers incur LOW cost of rework. Even if Developers introduce a Bug, they will discover it quickly through Automated System Tests (within <= 1hr)

The impact of this is that we have lower internal cost of change, hence we can spend more time on features rather than bug fixes, and we can deliver faster.

Impact #2: Working Releases

• No more Customer Regression Bugs! With Automated System Tests, we can be sure that the full suite is always executed.

The impact of this is that customers can trust our software, leading to increased customer retention, hence we can maintain our revenue consistently. Furthermore, it aids customer referrals and positive brand image, hence increasing the number of new customers we get, hence increased revenue.

Modern Test Pyramid - System Level

Here’s the Modern System Test Pyramid:

Let’s see how we can write effective System Level Tests:

• Smoke Tests

• E2E Tests

• Acceptance Tests

• External System Contract Tests

Read more

Register now

Upgrade to join the Live Q&A for free:

Subscribe now

Acceptance Tests help us verify System Behavior from the User Perspective. Thus, they are business-facing tests.

Acceptance Tests are black-box tests that are agnostic to the underlying System Architecture. Thus, the Acceptance Tests should remain the same irrespective of whether we have an MVP app, or Frontend & Monolithic Backend, or Frontend & Microservice Backend. It also doesn’t matter whether our system is well-designed or whether it’s a Big Ball of Mud. Acceptance Tests don’t care about this.

But how do we handle the problem of UX/UI changes when we’d have to rewrite all our Acceptance Tests? What about when we have multiple channels - e.g. one Web UI, one Mobile UI, and perhaps a public-facing REST API - do we then have to write three tests for each functionality?

Fortunately, Dave Farley came up with a solution to this problem - if we write tests that are expressed in terms of the domain language (the DSL), rather than being expressed in terms of the UI, we can then write a test that is insensitive to UX/UI changes and such that the single test can be executed against multiple channels (Web UI, Mobile UI, Desktop UI, public REST API).

Acceptance Tests - Four Layer Model

In the Four Layer Model, the Acceptance Test calls the DSL, which calls the Drivers, and finally the Drivers call the Systems.

We can see below that the Acceptance Test calls the System DSL to execute operations on the System, and it calls the External System Stub DSL to execute operations on the External System Stub. The DSL is written in the domain language.

The DSL needs to communicate with the underlying Systems. The System DSL needs to communicate with the System via System Driver(s). The External System Stub DSL needs to communicate with the External System Stub via External System Stub Driver.

Note: Why communicate with the External System Stub, rather than the Real External System? It’s because the External System Stub enables us to have full control over the External System, we can execute any scenario on the System. This is contrasted with E2E Tests, which do not use Stubs, and thus have limited scenarios that can be executed.

Note: In the image above, there’s only one External System. If there’s multiple external systems, then for each one you’d have a corresponding Stub DSL and Stub Driver (e.g. External System A Stub DSL, External System A Stub Driver).

You can read more about Farley’s Four Layer Model.

External System Contract Tests - Three Layer Model

Above, we saw that Acceptance Tests rely on External System Stubs, rather than connecting to the (real) External System Test Instance.

But what if the External System Stub doesn’t match the Real External System? Then, our Acceptance Tests could pass, but yet it wouldn’t work in production.

In order to have assurance that our External System Stub matches the Real External System, we need to write External System Contract Tests.

The External System Contract Test calls the External System Driver. This is done both for the “Real“ and the “Stub“ to verify that the “Stub“ matches the “Real“. This provides us assurance that our assumptions about External Systems are correct, thus we can trust our Acceptance Tests.

Sample User Story

Let’s start with this sample User Story:

User Story: Place an Order

As a customer, I want to place an order for a product, so that I can purchase items I need.

Acceptance Criteria #1: Successfully place an order for "Apple"
    Given the ERP is set up with the product "APPLE1001" and price $1.99
    When the product "APPLE1001" is added to the cart with quantity 5
    And the customer details are provided with name "John", address "My Street", and card number "123451234512"
    And the order is placed
    Then the order confirmation message should be "You have successfully placed your order"

Acceptance Criteria #2: Fail to place an order due to negative quantity
    Given the ERP is set up with the product "APPLE1001" and price $1.99
    When the product "Apple" is added to the cart with quantity -2
    And the customer details are provided with name "John", address "My Street", and card number "123451234512"
    And the order is placed
    Then an error message should be displayed indicating invalid quantity

Acceptance Criteria #3: ....

...

Now select the first Acceptance Criteria:

Acceptance Criteria #1: Successfully place an order for "Apple"
    Given the ERP is set up with the product "APPLE1001" and price $1.99
    When the product "APPLE1001" is added to the cart with quantity 5
    And the customer details are provided with name "John", address "My Street", and card number "123451234512"
    And the order is placed
    Then the order confirmation message should be "You have successfully placed order <Order number>"

How do we write the Acceptance Test & External System Test(s)? How do we implement the Test Architecture such that System Behavior is separated from System Structure?

Let’s see some examples…

Read more

Subscribe now

Initially, Manual Regression Testing seems OK!

When a company is in startup mode, they do everything quickly. The goal is to get customers, get revenue, and get the market position. At this point, all the testing is done manually - maybe just a few hours is enough to do all the manual regression testing. It doesn’t take up too much time. One QA Engineer is enough. Business is happy.

But later, as the product grows, it’s a nightmare!

Then, the company grows. After several years, the software product is bigger and more complex.

The problems faced are:

• Manual Regression Testing takes too long, and even longer

• Developers spend the majority of their sprint bug fixing, not much on features

• You have to keep hiring more QA Engineers, but it’s never enough

The obvious solution is Automated Testing

The “obvious“ solution to the above is to introduce automated testing. Everyone believes they’ll get these benefits:

• Zero regression bugs going into production

• No more manual regression testing by QA Engineers

• Developers will spend most of their time on new features, rather than on bugs

Let’s introduce the E2E Test Initiative

One common approach used by Engineering Managers is to switch manual QA Engineers to Test Automation Engineers, and to transfer manual QA procedures to E2E Tests.

This is common in companies who believe that:

• It’s the QA Engineer’s job to test

• It’s the Developer’s job to code

So then the obvious fix is to optimize the QA Department, which works in silo mode, separated from Developers.

Engineering Managers might try one of the these things:

• Writing E2E Tests that directly use Selenium / Cypress / etc.

• Using some record-and-playback tools to generate E2E Tests

E2E Tests are attractive because they are highly realistic tests, can detect regression bugs in the System and are in communication with External Systems.

Unfortunately, the E2E initiative fails!

Initially, everything looks great, the E2E Tests have eliminated Manual QA Regression Testing.

But then, 3 growing problems occur:

• Unreliable regression bug detection: Initially, the E2E Tests were detecting regression bugs from the end-user perspective, so when an E2E Test failed, developers could fix the regression bug. But later, E2E Tests start failing even though there’s no regression bugs! Then the developers have to waste time troubleshooting E2E Tests, only to discover they’ve wasted time, because there’s no regression bugs!

• High maintenance cost - wasteful: Initially, it didn’t take much time to write the E2E Tests. But later, as the software is changes due to changes in UX/UI design, the E2E Tests start failing even though there’s no regression bug. The QA Engineer has to waste time updating hundreds of tests due to a change in one UI element.

• We can’t test acceptance criteria: We can test many positive flows, but we’re quite limited in terms of negative flows. So, for negative flows, we may still need manual QA testing or we wait for user complaints.

Let’s dive deeper into these problems, and find out how we can solve them: ⬇️⬇️⬇️

Read more

Subscribe now

Coupling Contract Tests to HTTP Clients

When introducing External System Contract into Legacy Code, the most straightforward way is to directly call the HTTP client from the test.

What if the External System changes? Nightmare

The problem with that approach is that we have External System URLs scattered around everywhere across the tests! If for whatever reason, some endpoint changes on the External System, we may need to update several tests! Or even worse, if we switch from one External System Provider to another, we’d probably need to completely rewrite all the contract tests.

How to write maintainable Contract Tests?

That’s why we need Contract Tests for External Systems which can be executed both against the External System Stubs and against the External System Test Instances.

I’ll show you how to add Contract Tests to provide assurance that the External System Stub Drivers are actually valid.

Here are the steps to refactor External System Contract Tests in Legacy Code to be more maintainable. I’ll provide you with step-by-step guidance on how to refactor them in your GitHub Sandbox Project. ⬇️⬇️⬇️

Read more

Subscribe now

What are External System Contract Tests?

External System Contract Tests verify that our System can communicate with External Systems; that we have a valid expectation regarding their contracts.

• If the External System Contract Test passes when executed for both the External System Test Instance and External System Stub, the Stub has a matching contract

• If the External System Contract Test passes when executed against the External System Test instance, but fails when executed against the External System Stub, w need to make corrections to our External System Stub

• If the External System Contract Test fails when executed against the External System Test Instance, we’ve got wrong expectations or the External System has changed - we need to update the External System Contract Test and the External System Stub

External System Contract Tests target (and span) the External System Test Instances.

• If we have the possibility to deploy the External System Test Instances (e.g. self-hosted ERPNext), then we deploy them to the E2E Environment and run External System Contract Tests on those instances

• However, if the External System Test Instances already exist outside of our environment (e.g. PayPal Developer Portal), then we run the External System Contract Tests against those instances

For each External System, we would have some External System Contract Tests. As part of test execution, we should be able to “switch“ the connection between some External System Test Instance and the External System Stub. In that way, the External System Contract Test is written once, but can be executed against both.

Why External System Contract Tests?

Within Acceptance Tests, we’d seen how E2E Tests are limited because they can only test some Acceptance Criteria, whereas Acceptance Tests are able to test all the Acceptance Criteria.

So, can we just replace the E2E Tests with Acceptance Tests?

This would be problematic for the following reason: ⬇️⬇️⬇️

Read more

Subscribe now

Acceptance Tests rely on External System Stubs

In Acceptance Tests, we stub out External Systems, rather than connecting with the real External Systems, in order to test our System in isolation. By using Stubs, we have full control - we can translate any functional Acceptance Criteria into Acceptance Tests, which is a big benefit compared to E2E Tests.

But what if the External System Stubs are wrong?!

Suppose that we incorrectly stub out External Systems - perhaps we made an accidental spelling mistake in an endpoint. If that same “mistake“ is also in our System source code, the Acceptance Test will pass, even though the Stub is wrong!

The passing Acceptance Test gives us a false sense of security - this is very dangerous!

How can we ensure that the Stubs are valid?

We need Contract Tests for External Systems which will be executed both against the External System Stubs and against the External System Test Instances.

I’ll show you how to add Contract Tests to provide assurance that the External System Stub Drivers are actually valid.

Here are the steps to introduce External System Contract Tests in Legacy Code. I’ll provide you with step-by-step guidance on how to introduce them in your GitHub Sandbox Project. ⬇️⬇️⬇️

Read more

Subscribe now

What are E2E Tests?

We use E2E Tests to verify System behavior, however the scenarios are limited.

E2E Tests target the System - they span the System and External System Test Instances. They’re executed on the E2E Environment.

E2E Tests are slow. They’re limited in terms of the scenarios we can execute because we can’t fully control the External System Test Instances. Furthermore, we can’t control the System Clock either. Consequently, we’re very limited regarding scenarios that we can cover with E2E Tests, hence there’s a lot of Acceptance Criteria that we can’t even test with E2E Tests - but we can test it with Acceptance Tests.

The difference between E2E Tests and Smoke Tests is that when connecting to the External Systems, E2E Tests connect to External System Test Instances, whereas Smoke Tests can connect to either External System Test Instances or External System Stubs (depending on which environment they’re run on). Also, E2E Tests cover the critical Use Cases, to the extent that certain scenarios are possible.

If the System exposes a UI (for End Users), the E2E Test can use a Web Driver to run commands on the UI, and to do verifications via the UI:

If the System exposes a REST API (so that other systems can connect to it), the E2E Test can use a HTTP Client to connect to the System’s REST API, and do verifications via the REST API:

If the System exposes both a UI and a REST API, then we combine the above:

Why E2E Tests?

Dave Farley said: “Don’t Do E2E Testing!“

The idea is to replace E2E Tests by Acceptance Tests & External System Contract Tests.

That makes E2E Tests redundant, right?

Should we completely eliminate E2E Tests.

This is problematic for two reasons: ⬇️⬇️⬇️

Read more

Subscribe now

What are Smoke Tests?

We use Smoke Tests to verify that the System is up-and-running.

Smoke Tests target the System - they span the System and External System Stubs or External System Test Instances.

• Smoke Tests executed on the Acceptance Environment span the External System Stubs.

• Smoke Tests executed on the E2E Environment span External System Test Instances.

Smoke Tests are executed before Acceptance Tests & E2E Tests because they provide feedback. It makes sense to execute Acceptance Tests & E2E Tests only after Smoke Tests pass.

Smoke Tests are similar to E2E Tests in terms of span. The difference is that there are very few Smoke Tests, and they don’t test Use Cases, but rather basics such as whether the main page has loaded, whether login works, etc. They quickly verify that the System is up-and-running after deployment. After the Smoke Tests pass, we’re ready to run other System Level Tests (E2E Tests, Acceptance Tests, External System Contract Tests).

If the System exposes a UI (for End Users), the Smoke Test can use a Web Driver to run commands on the UI, and to do verifications via the UI:

If the System exposes a REST API (so that other systems can connect to it), the Smoke Test can use a HTTP Client to connect to the System’s REST API, and do verifications via the REST API:

If the System exposes both a UI and a REST API, then we combine the above:

Why Smoke Tests?

Suppose we just had E2E Tests (or Acceptance Tests), but didn’t have any Smoke Tests.

We run E2E Tests (or Acceptance Tests). The tests start failing. Then we start debugging, and find out it’s because deployment failed, or there’s some web server configuration error preventing us from accessing the system, or the External System Test Instances (or their Stubs) are not running, etc.

This is problematic for two reasons: ⬇️⬇️⬇️

Read more

Subscribe now

Legacy Code often has E2E Tests coupled to the UI

When we introduced E2E Tests into Legacy Code, we wrote the tests directly using a WebDriver, and then redesigned them to use the Page Object Model.

These E2E Tests are hard to maintain

In both cases, the tests are hard to maintain - in the POM Model, the tests are directly coupled to the UI structure.

We need to adopt a maintainable E2E Design

Let’s see how to introduce maintainability and make the E2E Tests more readable and decoupled from the System implementation. Even as the System API changes (due to UI changes and public REST API changes), if the behavior doesn’t change, our tests remain stable and only a small percentage of plumbing is affected:

Previously, we introduced:

• E2E Tests - WebDriver Model

• E2E Tests - POM Model

Now let’s refactor towards:

• E2E Tests - DSL-POM Model

• E2E Tests - Four Layer Model

Here are the steps for refactoring E2E Tests in Legacy Code. You’ll get tasks to implement in your GitHub Sandbox Project. ⬇️⬇️⬇️

Read more

Subscribe now

Legacy Code might not be E2E Testable

Legacy projects might have the following problems:

• Hard-coded connection strings to Production Instances of External Systems → this is dangerous because we can change the state of Production Instances during testing

• External Systems might still be in development → we might need to wait months for months for the External Team to release a Test Instance so that we can test our integration with them

Then we can’t introduce automated E2E Tests

In those cases, we can’t introduce E2E Tests at all, because we want to be able to run E2E Tests safely, at any time, no constraints, and independently of the development status of other teams. (This problem applies not only to E2E tests, but also to Smoke Tests)

We need to ensure E2E Testable Architecture

Often developers try to introduce just Unit Testability (a lot of techniques in Working Effectively with Legacy Code help you untangle messy code to make it Unit Testable).

The problem is, based on my experience, it’s too much to try to jump to Unit Testability. That’s why I start with E2E Testable Architecture, then Acceptance Testable Architecture, then Component Testable Architecture, and finally, last but not least, to Unit Testable Architecture.

Is your architecture E2E Testable? If not, you need to re-architect it to support testability.

Here are the steps for achieving E2E Testable Architecture in Legacy Code. You’ll get tasks to implement in your GitHub Sandbox Project. ⬇️⬇️⬇️

Read more

Subscribe now

Legacy Code has Manual E2E Testing

In Legacy Projects, QA Engineers often perform manual E2E tests based on written test procedures.

Hard to jump to ATDD with Manual E2E Tests

Like E2E Tests, Acceptance Tests test the functionality from the End User perspective (except that Acceptance Tests use External System Stubs rather than External System Test Instances).

It’s very hard to jump straight into writing Acceptance Tests retroactively when there aren’t even any automated E2E tests in place.

Convenient to automate E2E Tests before ATDD

It’s convenient to start with automated E2E Tests, and then transition to Acceptance Tests, by swapping out the External System Test Instances with External System Stubs).

Here are the steps for writing E2E Tests in Legacy Code. You’ll get tasks to implement in your GitHub Sandbox Project. ⬇️⬇️⬇️

Read more

Subscribe now

Legacy Code has Manual Smoke Tests

In Legacy Code, it is typical that when the system is deployed to the UAT Environment, the QA Engineer might carry out a basic smoke test to ensure that the system is up-and-running, before they proceed to perform manual E2E tests.

An example of a basic end-user facing smoke test might be to open up the main page in the browser, and check that the page has loaded. The QA Engineer might also perform manual smoke testing for the backend, such as running some health check endpoint, checking if the database is running, checking whether the message broker is up and running - by manually trying to connect to the database server, manually connecting to administrative page of the message broker.

Then, when they see that the Manual Smoke Tests are passing, they can perform Manual E2E Testing.

We can’t do ATDD with Manual Smoke Tests

Since we want to adopt ATDD, we’d need to ensure that our system is up and running after it’s released before running the Acceptance Tests. The reason is that if the Smoke Tests fail, then it doesn’t make sense to run the Acceptance Tests. Since we’d want to be able to run Acceptance Tests regularly throughout the day, it’s infeasible to run manual Smoke Tests before running automated Acceptance Tests.

We need automated Smoke Tests for ATDD

That’s why we need Smoke Tests to be automated, so that we can immediately run them after deploying to any environment. Thus, after releasing the system to the Acceptance Environment, we could run Smoke Tests, and then Acceptance Tests.

Previously, we’d set up the Pipeline - we set up the Smoke Testing step within the Pipeline, but at that point, the tests were just placeholders. In this article, we’ll be writing the actual tests.

Here are the steps for writing Smoke Tests for Legacy Code. You’ll get tasks to implement in your GitHub Sandbox Project. ⬇️⬇️⬇️

Read more