A guide to Zen Agile

… or “Sh*t happens, get over it”

“Agile” is one of the most discussed #topics in tech. You might say that we’re in a ‘post-agile’ era, with it being de rigueur for software development.

However, just as much is written about the tendency of senior management to think it’s a magic trick; enabling teams to do more with less (faster) rather than something they themselves need to adopt.

Is agility a higher state attainable only by elite Silicon Valley outfits like Spotify, or is it something that any company can achieve?

Eager to set themselves apart from the crowd, certain members of product management royalty have started to pour scorn on Agile. They point out the prevalence of a dogmatic, cargo-cultish zeal for Scrum.

If being agile (note the lower-case “a”) really were a bad thing, then what would be the alternative? How about the Stolid framework, or the Denial method?

So, what is this Agile thing anyway?

It’s important to remember that Scrum and Agile are different things – it is perfectly possible to use Scrum without being Agile, and vice versa. So, when Drift CEO David Cancel shouts “I hate Agile!” Perhaps what he means is; ‘I hate when people do a Scrum thing and think that means they’re Agile!’

It’s understandable that some are calling out the enormous quantity of noise generated by the word “Agile”, especially when it’s capitalised.

In 2016, a Deloitte analyst inadvertently portrayed this maelstrom of jargon in his infamous 2016 ‘tube map’ diagram. This diagram showed (some of) the various tools and practices advocated in the Agile world:

‘tube map’ diagram

Christopher Webb’s infamous Agile tube diagram highlights the complexity of Agile – it’s more than just scrum!

Ugh – easy to see why someone might think Agile has lost its way.

Agile is something you should be, not something you should do, and no tool will make you agile. This comes more easily to some than others, particularly those blessed with a short attention span.

Let me give you some personal examples…

In my school days, I was frequently the recipient of an airborne board rubber, or a palm-stinging tawse for staring out of the window, forgetting to do my homework, turning up wearing the wrong clothes, etc. Changing my mind and accepting new evidence isn’t a problem, as I sometimes can’t remember what my point of view was yesterday!

A former CEO I worked for, couldn’t get his head around why I’d sometimes reposition half-way through a meeting. His preference was to relentlessly persevere with “the plan”, an always-out-of-reach goal to “arrive in style”. Needless to say, he was a former project manager from whom the word “agile” literally elicited a wince.

Living with such constraints is an ordeal for agile thinkers.

As a university student, I was once rushing to catch a train home to Durham (13 minutes) at the end of a busy week. I ran into the station and onto, what I thought was, the 17:25 waiting at platform 2. Imagine my dismay when we departed in the wrong direction, and then when the guard confirmed that the first stop was Edinburgh (2 hours). My commute had just turned into a round trip to Scotland.

Most of it was spent sitting in an empty carriage and with the darkness outside I couldn’t even admire the scenery. Not planning for this impromptu trip, I had nothing to read other than the train company’s brochure; exhorting me to take a day trip to historic Durham. It’s not unlike working in a project-driven company and reading blogs about A/agile.

On another occasion (stay with me); when working as a consultant, I rushed out of the office, jumped in the car and was bombing southward from Newcastle on the A1. As I passed Scotch Corner services, a sudden bolt of awareness struck cold fear in my spine: Where was I going? Stopping at Leeming Bar, I got my laptop out to check my diary (no smartphones yet) and confirmed that I should have been heading North to Glenrothes. Fortunately, the unintended detour added only 90 minutes to my journey.

Did you spot the analogy?

The train is Waterfall and the car, Agile. In both cases, I took the wrong initial course. But, on the train journey, I had to continue until the bitter end.

I did ask the guard if he would stop the train to let me off, but he just laughed at me with the incredulity of a PRINCE2 programme manager when you ask them to pivot.

For me, being anything other than agile is effectively to pretend that (a) you can predict the future and (b) sticking to a bad plan is better than changing it. So why does Waterfall still exist, given its dire reputation in product development?

Letting go of the future is counter-intuitive

We, humans, are instinctively able to discern patterns in things. We use this to foretell the future by extrapolating from our past experiences. Side effects of this include an emotional need to know what’s going to happen, and a predisposition to confirmation bias. Doing a familiar task makes you feel like you’re on safe ground and you can go a long way along the road to damnation before reality intervenes.

So, when planning a Waterfall project, everyone involved conspires to delude themselves that they’re going to release ‘on time’ and ‘on budget’. Execs push a date at which the release must happen, and delivery teams make promises based on estimates. Devs complain about this, pointing out that it’s impossible to accurately forecast software development, so someone adds ‘contingency’ to the made-up numbers. Everyone starts out feeling comfortable and positive because there is a plan.

Before long, however, stakeholders start to wonder what’s happening. As there is, as yet, no software to be seen, their nerves start to jangle about the pace of progress. They enquire into how things are going, only to be told that “There’s nothing to see yet, we’re still architecting the platform.”

The lack of visible progress on a waterfall project can lead to some nervy stakeholders

The anxiety builds to the point where they demand to know whether the deadline is going to be met and that something tangible is produced. The delivery team grumbles about ‘scope creep’, ‘changing requirements’ and having to make ‘shiny’ stuff. This negativity is driven by a growing feeling that control is slipping from their grasp.

At the same time, suspicion is growing from people outside the team who ‘just don’t understand how hard this is’. The deadline is extended, tempers fray, additional pressure is heaped on the team and people quit. We’ve all seen it before. When running a project this way, you should start out by writing the post-mortem!

In truth, it’s not feasible to accurately predict how a software project will pan out, as the control was never really there in the first place. This is the main reason for the ‘inspect and adapt’ concept in Agile.

This encourages the delivery team to review their progress early and often, thus reducing the risk of doing the wrong thing for a long time. You may have seen the following model for comparing risk between Agile and Waterfall methods :

Models comparing risk across Agile & Waterfall – But, in reality, how accurate is it?

The idea is that in a Waterfall project risk remains high until delivery. Whereas, in Agile the risk is reduced from an early stage by iterative inspection and adaptation.

However, this analysis of Waterfall is rather optimistic. In practice, it often creates a graceful, slow-motion failure, founded on stated prejudices (AKA “requirements”). Like a computer programme, the project will deliver what it has been set up to do, however unhelpful.

A bulletproof change control procedure will cover the Project Manager, and even make them look good. The aim is not to create value, it’s to ensure that a plan is followed. Boxes ticked, meetings minuted, excuses made. All too often the outcome is an unrecoverable failure and, having spent so long building the wrong thing, there’s no budget left to rectify it. First-mover advantage has been lost.

The deadline delusion

As Benjamin Franklin would have written, were he a project manager: “In this world, only two things are certain; deadlines and liquidated damages.”

It’s Agile blasphemy, but also an inconvenient truth, that deadlines are inevitable in business. Building a product costs money and unless you’re Google or in the Silicon Valley VC bubble, there is generally a finite amount of it available. Therefore, stakeholders are understandably keen to spend as little time as possible spending it. Unfortunately, in software projects, deadlines are nearly always missed. This can be because:

  • Customers always want stuff yesterday, and your competitor is threatening to give it to them, leading to pressure to say “Yes, soon”;
  • Technical people often overestimate their abilities and neglect to account for mistakes, lost time and unforeseen circumstances;
  • Once the team is making something, they feel good and fall into the trap of thinking there is plenty of time left. Only when things are obviously off track does anyone intervene, and by then it’s too late.

In fact, if you meet a deadline, it’s probably because the stakeholders were prepared to overpay to guarantee it. This is an uncommon situation in commercial product development, as money is generally most definitely an object.

Transcending the tyranny

How to escape this?

Permit me one more confessional anecdote: As a heavy smoker in my twenties, I resisted giving up for many years. I was unable to countenance a future as a clean-living kill-joy. I felt that my life’s course was already set and that I was most of the way through it. When I finally decided to quit it was genuinely terrifying for me because it felt like dying. My personality, based on the ‘smoker/rebel’ template, was ending and I didn’t know whether there was life after the death of this self-image. It was a traumatic experience, but I survived (obv), and this major version upgrade taught me some life-lessons.

Lesson 1: The only thing that really matters is what happens next

Breaking my self-fulfilling prophecy that I’d always be a smoker was hard. I’d constructed a mental life script for myself, with a sense of inevitability based on my sense of self. However, you will remember from studying the Copenhagen interpretation of quantum mechanics (yeah?), that cause and effect at the macro scale is an illusion. Although it’s cosmically unlikely, anything and everything can (and some say will) happen next. Gross effects are averages of tiny random fluctuations.

This flies in the face of human intuition, which although generally serves us well, also suffers from systemic faults such as the sunk cost fallacy. We tend to persevere with a fruitless course of action because of the cost already incurred. We think “I’ve had the bad luck, some good is due soon!” Nope, any money you’ve wasted on a product is now in the past, and the only factors governing your future success are happening right now.

Great products, like Airbnb, came about as a result of many sprints; with many mistakes and brave decisions made along the way. A stretched analogy perhaps, but life and business are anything but deterministic. Or, to put it another way, take one step at a time.

Learning 2: If you decide to keep going for something, you’ll probably get there, eventually.

One day, around a year after quitting smoking, I realised that I was no longer thinking about it all the time and had no inclination to go back. This only happened because I had decided on 17th March 2001 at 13:35 GMT to become a non-smoker. A definite goal, but with unspecific outcomes and an uncertain time scale.

Similarly, my car trip had a destination (despite the detour to Scotch Corner). If there had been no ‘north star’ goal, I may have found myself endlessly negotiating the one-way system in Middlesbrough – keeping very busy but not actually going anywhere. To paraphrase Lewis Carroll, “If you don’t know where you’re going, any road will take you there.”

Agile is not random. Rather, it’s mindful pragmatism that keeps you moving in the right direction. You continually apply course corrections, but always with a destination in mind. Rather, like driving on the A1, it requires your unwavering attention. You constantly adjust your direction, mostly minor changes, but in the case of a mission-critical blockage/error, it can be a pivot (e.g. realising it’s too late to get to Glenrothes and deciding to visit a client in Redcar instead).

Fans of classical project management will argue that this can be handled via change control. However, such frameworks are predicated on the assumption that change is an exception.

Conversely, you can think of Agile as continuous change control. Although, we shouldn’t talk about “change” and instead recognise, and indeed embrace, uncertainty. Throughout my career, I’ve heard delivery teams complain about changing requirements and “scope creep”. In most cases, the client hadn’t really changed their requirements. It was just the implications of their vague ambitions had become progressively clearer on close inspection.

Learning 3: Fear causes all the trouble

Channelling that great business philosophy guru, Yoda: “Fear leads to negativity; negativity leads to inertia; inertia leads to poor quarterly results.”

The reality is that it doesn’t matter how much you plan, things aren’t really under your control. There’s no point being afraid. The only thing you should fear is unfounded and bearish market sentiment (another delusion caused by projecting the past into the future), which is most definitely a #firstworldproblem.

What’s the worst that can happen? Well, you could lose your job? Faced with this worst-case scenario, It’s worth asking yourself; “If I’m miserable here because of constrained thinking, lack of trust and systemic fear, shouldn’t I go somewhere else?”

However, it is disruptive to have to keep switching jobs. So, how can you overcome the deadline delusions and control-freakery?

The trick is to understand that stakeholders are also prone to fear. They usually don’t grasp Agile and expect to see a linear progression, such as you might see with a house taking shape on a building site. However, making a new product is far less predictable than making a building and it’s important to understand how stakeholder confidence changes over time:

These diagrams show how stakeholder confidence may develop over the lifecycle of a project.

In the Waterfall case, confidence will be either restored or shattered completely at release time, depending on whether the product meets expectations that have built up over the project.

In the Agile case, early confidence can be dented as incomplete stages of the product are produced by the delivery team. But, so long as the team keeps taking feedback, confidence builds as subsequent iterations move visibly closer to a market-ready product. Jam might be tomorrow, but at least today it’s sugar. Then the day after tomorrow it’s scones and jam, and one day it will be the full afternoon tea.

A plan is not a goal – it’s a tool, and an adjustable one at that.

If you’re building a business and need some software made, you will probably have some idea of what you want your product to be. You will also need assurances from your product team that they can build it in a reasonable timescale, without busting your budget.

So, how to fix the three corners of the scope/resource/time triangle?

Traditionally; this would be achieved by spending a lot of time analysing the requirements in detail, planning exactly what’s going to be built, estimating it, then adding contingency and a change management process. If a high degree of confidence in the time and cost budget is achieved, the price is often accepting a longer timescale and a higher risk that the eventual solution will fail to match requirements that emerge or change during the project.

Agile is often seen as a silver bullet. Don’t plan beyond one sprint! Pivot every now and again! Prioritise everything, brutally! Que sera sera! Zealots will shout down any talk of roadmaps, planning or strategy, and take agile to its logical extreme, one-day sprints (!?). Why stop there? Why not have one hour or one-minute sprints? In fact, why don’t you just let everyone be their own singular Scrum team and do whatever they like? ULTIMATE AGILITY!

Of course, I’m being a bit silly and nobody would take this seriously as a business approach (you would think). Businesses happen because someone had a vision and pursued it with vigour and persistence, in the face of adversity. It is the founder who takes a punt on and is accountable for, an idea. Agile is great for handling a change of direction in the detail or even means of execution. But, if the big idea fails, the business is gone.

Building a business needs planning, but only for the “how”, not the “why”. To achieve the necessary blend of direction and agility, various tools can be used. I’m personally a fan of the ‘Three Horizons of Innovation” investment model (The Alchemy of Growth, Baghai, Coley, White, 1999):

The ‘Three Horizons’ of Innovation model can help us plan our current and future resources effectively

This framework suggests that you should invest 70% of your resources into this year’s problem (current product), 20% in the next generation product (say 1-2 years out), and 10% on horizon-scanning and new category products (3-5 years out).

Many of those third horizon products will be pushed out or never be realised while some next-gen product projects will fail.

This type of strategic view helps executives to organise their future planning, and at the same time illustrates the futility of trying to plan too far out. It also highlights the need for constraint on investment in the current product, as it will inevitably be superseded by the next- gen and new category replacements. Execs will be reassured that the Product Manager is capable of thinking strategically and won’t waste money on a product they’ve fallen in love with.

Relax, nothing is under control

To sum up with a Zen-like platitude…

Beware the duality trap, in which one believes they can plan the future of a product.

Learn to accept the reality of the current sprint, appreciate its innate business value; be comforted by your career prospects, which if you keep adding value, will transcend the noise of ephemeral shouty bosses.

Author’s bio

Aidan Dunphy is Chief Commercial Officer at global digital product consultancy hedgehog lab. His career in software spans more than two decades working in enterprise and consumer markets. Having held roles in all aspects and at all levels of the software industry, in recent years he has focussed on building Product Management and Strategy capability. Aidan’s passions are creating and discovering new value and evangelising the agile product strategy mindset.

 

Tags: , ,