The Failed Agile Transformation: A Recipe
It's much easier to "buy" Scrum than to introduce Continuous Delivery. I can say 100% that without Continuous Delivery, delivery speed is NOT sustainable. Scrum alone is NOT enough.
This is a story of what happens when companies introduce “agile” but do NOT introduce Continuous Delivery practices…
What happens when an “Agile“ consultancy sells just Scrum + JIRA to the client, with ZERO technical practices?
In the beginning, it’s all sunshine. You think everything going great. Then the thunder strikes, and your house is in shambles.
Where did we go wrong?
Was it because we didn’t implement Scrum well? Or did we not use JIRA “efficiently“? Or could it be because… (gasp!)… because we didn’t have adequate technical practices required to be agile?
Last week, I wrote an article regarding the necessity of Continuous Delivery (see You can't be agile without CI/CD and TDD (Revisited)). In that article, we derived the need for CI/CD based on the first principle of the Agile Manifesto - to achieve Continuous Delivery, we need Continuous Integration. We also concluded that TDD is the most effective way of achieving a Self Test Build (required to satisfy Continuous Integration).
We want to be Agile.
John, the CEO, heard that Agile is THE way to deliver faster AND be responsive to changes in business needs. Everyone’s Agile these days, so he wanted to join the trend.
It must be a utopia, right? John knew his business people had plenty of ideas they wanted to push to the market. John knew the huge competitive advantage of releasing more frequently to the market. So John told everyone, “We want to be fast. So we’re going to become Agile”.
And then, the Agile Transformation began.
Certified Scrum Masters.
Scrum Teams formed.
Brand new JIRA boards.… with Story Points!
But ZERO changes in Technical Practices!
Agile transformations are “just“ Scrum transformations, right?
After all, in all the brochures that John read, all he saw was that Scrum makes developers much faster. That’s why hiring Scrum Coaches & Scrum Masters was enough.
Engineering practices? No. There were forgotten, left to gather dust. The team would continue doing manual deployment and manual testing.
We won the first 100m.
The team did deployments at the end of each sprint so the QA Engineer could do manual testing.
During the first sprint, deployment + testing + bug fixing + re-deployment + re-testing took 1hr. The next several sprints were great too.
But we lost the race.
However, several sprints afterward, as more and more features piled up, the total time increased – going up to 4 hours, then it took one day, then two days, eventually three days… Regression bugs started piling up. As features piled up, there were more and more bugs.
So to make the system ready before the sprint demo, the whole team had to do a “feature freeze” 3 days before the sprint demo.
Management hears the bad news.
The news came to management that the team was “unproductive” for 30% of the sprint.
So then management told the team since deployment & testing take up so much time, it’s a waste to do it every sprint - let’s do it in “batches“ after several sprints.
Retrospectives were also dropped because they were wasteful - the team repeated the same problems each time.
We’re now in Waterfall-Scrum mode.
The team executed several “Feature” sprints, followed by “QA” sprints, followed by “Bug Fixing” sprints, followed by “UAT” sprints, and finally, the “Production Release” sprint and the “Hotfix“ sprint.
Delivery cycles became longer, time-to-market slower. Endless bugs. Unhappy customers. Couldn’t keep up with business demands, couldn’t deliver quality. So, ultimately, their customers left.
The illusion: We weren’t “Scrum enough“
The Agile Transformation failed us.
We were back to square zero.
John hired some Scrum Consultants. Their diagnosis of the root cause was? The company didn’t have the agile “mindset“, and they weren’t doing Scrum “correctly“.
So he hired yet another army of Scrum Coaches & Certified Scrum Masters, to be able to do Scrum “correctly“. Again, without any changes in engineering practices - no Continuous Delivery.
So then John concluded: Agile is just snake oil. (For John, the words “Agile“ and “Scrum“ were synonyms. So if Scrum doesn’t work, Agile is just snake oil.)
The hard truth: We needed Continuous Delivery
Continuous Delivery is the ONLY way to ensure a sustainable pace of delivery.
You can buy a “tonne” of Scrum, and a “tonne” of Kanban… you can do any “mindset“ training you want, but if the team is not technically equipped in CI/CD practices (and if no support is provided), then sorry but the agile transformation will fail.
And just for the record, CI/CD practices necessitate TBD & Test Automation. Just because a team uses Jenkins doesn’t mean they’re practicing CI/CD! If QA Engineers are performing manual regression tests and there’s no test automation suite, then the team is not practicing CI/CD.
But if we had Continuous Delivery, is that enough? Continuous Delivery is necessary but not sufficient (see https://journal.optivem.com/i/71155171/necessary-versus-sufficient).
Discussion & Comments
Thanks to our readers who contributed their thoughts and perspectives!
This story is too common
The story written above is unfortunately too common in our industry today. As Alan Mellor commented:
This is so depressingly common. Agile isn’t a management thing. It’s a set of technical enablers
Can you drive a car without wheels?
Marco Consolaro commented with a great analogy:
Agile without technical practices is like a car without wheels: it doesn't go anywhere!
Why technical coaching is needed?
Benjamin Feireisen provided a good summary of the full extent of transformation that we need (see comment):
Completely agree. I've seen it way too much.
The root cause is indeed "buying an Agile transformation", with this intention you will only have agile coaches and scrum masters who could be great at "behaviour transformation", and never as a technical support.
My personal thought : the best way to reach business goals, be it internal or external help, is to have a mix of organisational coaching (pro team coaching, team topologies, continuous alignment...), craftsmanship (TDD, DDD, continuous delivery or even deployment, and obviously many more things I forgot) and design thinking (continuous discovery, lean thinking...), and real product management (vision, goals, "outcome" instead of "output"...).
It's hard, right ?
Indeed that’s the essence! The Agile transformation needs to cover both organizational and technical aspects to be successful. You can’t just transform one aspect and hope to succeed!
Engineering practices are the foundation
Proper engineering practices are the foundation. If this foundation is missing, we’re bound to fail.
The need for automated regression tests
In the above article, we saw what happens when a team doesn’t have automated tests - it’s a disaster. Thanks to Richard Smith for his comments explaining why we need automated tests:
If you have a long running project (like in a product company) then you need automated regression tests, otherwise your regression tests either end up eating up an unreasonable amount of your development cycle, or you don't do them properly and end up with bugs making it to production (which then obviously impacts the next release).
This is worse if you have a shorter release cycle, because the amount of time and effort needed to properly regression test is basically the same no matter how big the release is.
An automated regression suite, with the coverage you need to replicate conditions in production (including usage patterns, data volumes, multiple concurrent interactions etc, not just behavioural system tests), is essential to enable short release cycles.
Managing Technical Debt
In this article, I covered Continuous Delivery (CD) practices because it’s the foundation for any other technical practices. In this story, the company’s technical level failed the bare minimum - failed CD.
Now, let’s say a company does achieve CD. Is that enough? No, we need to manage Technical Debt too. CD provides us with the safety to improve the quality of our code base. Well explained by Ger Cloudt (ref):
Nice article, however teams becoming slow is not only about lack of CI/CD. It is also about increasing Technical Debt as well. Yes, you need short feedback loops, typically realized by means of CI/CD, however you also need to manage your technical debt to achieve a sustainable pace of development.
We need to address technical debt...... (https://cloudtsoftwareconsulting.com/uncategorized/we-need-to-address-technical-debt/ )
So to summarize, in order to ensure sustainable development, we need technical practices. We set the foundations by adopting CI/CD practices - with this, we get the safety of automated tests and automated deployment. But that’s just the beginning, it’s not the end. The automated tests provide us with the safety to be able to improve the structure of our codebase through refactoring to minimize over time.
Failure at Scale - When the whole organization goes down
In this story, I described what happened to ONE team - adopting Scrum (without technical practices) for one team only - and you’ve seen the disaster that happens.
But can it get even worse? Yes, it can! Indeed, this is what Urs Enzler commented:
You forgot: "we start at maximum scale, of course"
Exactly! Many companies do start Scrum adoption at “maximum scale”, (without technical practices) - and then they get failure at “maximum scale“.
Scrum is NOT a synonym for Agile
Brad Appleton wrote an excellent comment; I couldn’t have said it better:
QUOTE: "This is a story of what happens when companies introduce 'agile' but do NOT introduce Continuous Delivery practices."
CORRECTION: This is a story of what happens when people (WRONGLY) presume that "agile" (or "Agile") is a synonym for "Scrum", OR that "Agile = Scrum")
The idea that 'Agile' doesn't include 'Continuous Delivery practices' is just as WRONG, and serves only to reinforce this widely propageted MISCONCEPTION.
It is 100% CORRECT to say that 'Scrum ALONE (without technical/engineering practices) is NOT ENOUGH'
The technical/engineering practices & principles (for agility) were always part of Agile Development AND right there in the Agile Manifesto principles, starting with the very first principles (which explicitly calls out 'Continuous Delivery', and is in fact what cioned the term)
When the adjective 'Agile' is (ab)used as a noun, it is short for 'Agile DEVELOPMENT' OR 'Agile Manifesto' or 'Agile Delivery' (or 'Agile Software Development' OR ' The Agile Manifesto for Software Development')
Unfortunately, the popularity of Scrum has overshadowed 'Agile' to the point of severe dilution, exclusuon, pollution (and even delusion) as to what 'Agile' (and even Scrum) truly is (and is NOT).
Let’s change how the IT industry works; let’s take software quality to the next level.