Based in London, united kingdom. working as a software engineer for palantir technologies. thoughts and ideas expressed are my own.

Agile, you have been a bad friend

Agile, you have been a bad friend

The biggest myth of agile development :  faster, cheaper and better outcome. To rephrase, one of my biggest bugbears is the myth of faster, cheaper and deliver better outcome with agile. And here is why.

The good news is everyone is onto agile these days. The bad news is people are onto agile without knowing what it entails. Let’s start with a bit of history.

Agile — a lunatic utterance with the magical power to deliver anything under whatsoever condition

Agile — a lunatic utterance with the magical power to deliver anything under whatsoever condition

Origin

The story of the agile manifesto is well-known in the agile community. Unfortunately, not many have heard of lean manufacturing which predates agile development and has a strong influence on agile development.

Many practices in agile development are heavily influenced by lean manufacturing and both methods have a similar philosophy, which emphasizes waste reduction. These two approaches differ in execution primarily because lean manufactures physical goods while agile develops digital products.

Comparison of seven wastes in lean manufacturing and agile development

Comparison of seven wastes in lean manufacturing and agile development

By reducing waste, lean manufacturing produces higher-quality goods at a lower cost.

Which begs the question, “Isn’t agile development faster, cheaper and deliver better outcome through waste reduction?”

The myth of faster, cheaper and better outcome

Yes. It does. Unfortunately, it is even easier to do agile without waste reduction.

Never underestimate the power of wishful thinking. It’s like sticking a Porsche decal on a bicycle. QED.

Sometimes, it happens when the delivery team or scrum master is inexperienced so you will want them to read this and send them for proper training.

In most situations, stakeholders or product owners don’t really understand the implications of agile development. They often believe that agile is something for the delivery team to do, while they can remain status quo. Unfortunately, nothing can be further from the truth.

For agile to be effective, stakeholders and product owners have to change their behaviors and provide a conducive environment for the delivery team. Else, they should just stick with traditional waterfall because doing agile without reaping the benefit is sheer stupidity.

Why bother when it’s all pain and no joy. Not even a masochist.

Seven wastes to eliminate

Waste 1: Partially done work (aka work-in-progress, WIP)

Such as detailed documentation and wire-frames and I don’t mean partially done documentation and wire-frames. I am referring detailed documentation, wire-frames, and plans that are yet to be implemented as code. Until they have been implemented as working code, these are WIP.

WIP worsens task switching, waiting time and the possibility of rework.

Unfortunately, human craves certainty and planning gives comfort. Some stakeholders and product owners require more comfort than others. 🤷🏻‍♂️

Plans, documentation, and wire-frames are proxies, with varied fidelity waiting to be implemented.

The amount of waste you can reduce depends on the risk appetite of your stakeholders. WIP is an unavoidable evil so strive to keep it low.

Waste 2: Extra processing

Obviously, don’t spend too much time to polish proxies such as plans, documentation and wire-frames.

Unfortunately when organizations are structured as functional groups, functional specialists tend to over-process because they are appraised to do so — planners are evaluated by their detailed and exhaustive plan, business analysts are measured by the thickest of their requirement documentation, and designers are… you get it.

No single raindrop believes it is to be blamed for the flood

This systemic waste cannot be eliminated by the agile delivery team. It requires organizational restructuring by the senior management, to align incentives throughout the organisation. This is why our people report within the squad in GDS. Mission-focused over functional-centric.

Waste 3: Task switching

Another side effect of functional groups is the tendency to assign each specialist to multiple projects, in order to fully utilize them.

Digital delivery activities are highly abstract and require creativity. Switching across tasks incur waste due to context-switching. Additional waste is incurred to persist the current state with reasonably high-fidelity, in order to be resumed correctly at a later stage. It takes about 15 minutes to interrupt and to resume work for every task switching.

Another common cause of task switching is meetings. If you must have meetings, always move meetings to the beginning or end of the day in order to minimize waste to the entire team. Understand the difference between maker’s time and manager’s time.

To be responsive to change requires slack. A fully utilized workforce is the antithesis of agility because it increases waiting time and delay.

Waste 4: Waiting / delays

Excessive waiting often arises when the product owner is not collocated with the delivery team. The team needs the product owner to clarify user stories and validate their work when it’s completed. Waiting time can be disproportionately high when teams are distributed across different time-zones.

This impedes the team’s ability to get feedback from product owner during retrospective and from other stakeholders during sprint review. No or slow feedback means no or slow improvement, which causes persistent defects and repeated rework.

Meeting is the most common cause of delays — finding a common time slot, synchronizing room’s and people’s availability. If you must have meetings, here’s how to make the most of it.

Waste 5: Defects (rework)

The longer it takes to detect defects, more defects will be created and more waste incurred on rework.

The only solution is to shorten the feedback loop and rectify early. Daily stand-up, sprint review and retrospective session are the best avenues to obtain feedback, rectify defects and drive improvement.

Unfortunately, these sessions are often underutilized. The team requires a safe environment to share critical and constructive feedback, in order to level-up their game. The product owner must participate in and co-create this safe space.

Waste 6: Hand-offs

Like similar reasons for partially done work, extra processing and task switching, hand-off often arises due to functional organizational structure.

Knowledge is lost every time a deliverable or an artifact is handed-off across roles. This results in longer delay and higher defects, on order of magnitude more than the hand-off of physical goods. It is foolish to incur hand-off cost in favor of higher utilization of people, for digital work.

Waste 7: Overproduction

Overproduction is to develop features that are not needed by the users. The best way to prevent overproduction is to build the smallest possible feature or product (MVP) and validate the hypothesis before building a bigger and better version of it.

Unfortunately, overproduction often happens because of product arroganceby product owners and other stakeholders. They are often overly certain of their idea that they choose to skip the MVP and invest in a full-blown product.

Of all the wastes, overproduction is the worst because it exacerbates all other wastes.

Agile rituals and engineering practices are useful and will help your team improve their game. Unfortunately, the team’s performance improvement will soon hit-a-wall due to obstacles beyond their control. These obstacles are also known as wastes — a term borrowed from lean manufacturing. They arise from excessive planning, functional organization structure, unavailable product owner, excessive meetings, lack of a safe environment for critical and constructive feedback, and product arrogance by HiPPO.

Surprise! There’s nothing magical about agile.

Agile makes these wastes visible, so that product owners and stakeholders can eliminate them. Otherwise, it’s just another fancy idea that produces more noise than substance.

Screen Shot 2019-04-27 at 5.56.02 PM.png

Epics are dead. Here’s what we should do instead.

What has not been declared dead already? Test Driven Development was buried years ago. Still, it continues to spread. Of course, Agile is dead as well. But even traditional companies come into contact with Scrum. The dead continue to live, but declaring something dead is always good for a snappy headline. In that sense, witness how I destroy epics as an agile practice.

What are epics?

The term is vague. This has advantages. Epics are more for communication than specification. The vagueness makes them versatile. But there is a risk of misunderstandings. I stick to Mike Cohn’s definition:

A Scrum epic is a large user story. (Source)

I use the term like this: An epic is a story that is too big to be implemented in a Scrum sprint. The items at the top of the Product Backlog are thus not epics, but little stories. Down in the backlog you will typically find epics. Over time, the epics are “sliced” into stories that can be pulled into a sprint.

That is what I have taught for years in my training courses. It seems to be the general consensus. Intuitive at first glance. I’m here to explain why it’s not practical.

3 impractical ways to deal with epics

So far, I’ve come across three ways companies deal with epics. None of them is practical. Let’s call them:

  1. Dissolution

  2. Links

  3. Trees

1. Dissolution

The principle of dissolution is simple. An epic is completely broken down into its components, the individual little stories.

For example, an epic “Book flight” of an online flight portal can be broken down into the individual process steps. So “Log in”, “Search flight”, and so on. Every process step becomes a story. The team estimates the story. As long as it is too big, the team continues to slice it. Once all stories are small enough to fit into sprints, the team deletes the epic and starts development for the stories.

It’s the underlying idea of completeness that bothers me. The dissolution suggests that a topic can be completed with a predetermined scope. 
But if changes to the stories are possible during development, you can’t define all the stories upfront.

The Scrum Guide says:

A product backlog is never complete. […] Requirements never stop changing.

If you have to deliver a fixed scope, stop pretending. Forget about epics, and describe the detailed requirements upfront. Just don’t claim to be agile then.

2. Links

If you don’t dissolve your epics completely, it makes sense to use links. The epics remain, down in the backlog. You link new little stories to the epics from which they derive.

The risk is that over time, the amount of epics increases. The backlog becomes bloated. It contains epics that you don’t need any more. The stakeholder is no longer in the company. Or the topic is no longer relevant.

Of course, you can clean up your backlog from time to time. I regard this as non-value-added work. And you can avoid it, as I will describe later.

3. Trees

Another way is the depiction of epics and stories as a tree:

Depiction of epics as a tree

Depiction of epics as a tree

You group the little stories by epic. Not a bad idea. But what you lose is the ordered list of the backlog. How do you determine the implementation order then?

Of course you can use a digital tool that supports both views. The risk: you invest too much time and effort in the tools. What are the views? What are the attributes? What is the underlying data model? Interesting questions. But in an agile approach they should not have high priority.

In summary, the idea of grouping is good. But doing it is time-consuming.

The alternative to epics

There has long been an alternative. It is even mentioned in the same blog post by Mike Cohn, which I linked above. I am talking about themes.

A theme can be thought of as an additional attribute of the stories. Normally, several stories share the same theme. The story “Search flight” could have the theme “Book flight”. A snippet from the backlog could look like this:

User Stories with theme

User Stories with theme

Themes are not managed as separate backlog elements. This eliminates the cleanup work discussed in the Links chapter. That’s good.

But what you lose is the process of gradual refinement from the big epics to the stories that can be implemented in a sprint. That’s bad.

Luckily, there are practices that make it possible to do this refinement outside of the backlog. One way to identify themes is a use case diagram:

Screen Shot 2019-04-27 at 6.01.44 PM.png

The nice thing about such diagrams is that they show the “Big Picture” due to the high level of abstraction and the graphical representation. For that, a backlog is unsuitable.

The use case names later become themes in the backlog. But how do you get from the use cases to the stories? For this, Story Mapping is a good fit:

Screen Shot 2019-04-27 at 6.02.16 PM.png

The top two lines of the example map show the use cases “Book flight” and “Manage profile” and their basic flow. Under the individual steps, the team hangs the alternatives: other processes, errors and so on. These yellow notes are called user tasks.

In Backlog Refinement, the team derives the stories from the user tasks. A task can serve as the title of the story. The team adds details like acceptance criteria to the stories.

The consequences

Applying this alternative approach has consequences. For example, the Product Backlog will only contain stories for the next 1–2 sprints. So maybe 10–20 stories.

All activities like further prioritization, estimation and elaboration of the acceptance criteria only take place with these stories. As the 10th agile principle says:

Simplicity — the art of maximizing the amount of work not done — is essential.

If management wants to have insights into the progress of development, this is possible on three levels:

  • Use case diagrams or themes provide the long-term perspective for management. For 1 year, or even beyond. But: they are not suitable for specifying details.

  • Story maps form the basis for release planning. Stakeholders interested in the release create the story map with team members. (Due to new findings, the scope may change during development.)

  • Those who want to have a deep insight and influence the details during development participate in Sprint Review and Backlog Refinement.

Only at low altitude, we see the details. And the Product Backlog is basically like a shopping list. Would you write down what you want to buy in a year?

Last, but not least, the death of epics heralds the dying of consumerism. If you want something, you have to agree with the team and work closely together.

Post mortem

In the discussion with colleagues, they pointed out that even after a dissolution of an epic, small stories can be added. That’s right, and for me it’s an acceptable solution. What is lost in this case, however, is the “Big Picture” that I have shown in the use case diagram.

Ultimately, the suitability of a product for users determines its success. Not how it was made. This applies to all development practices, including epics.
Maybe you have come up with a sensible way to deal with epics? Then let me know.

Cheers 🍺

Kubernetes: Essentials

Kubernetes: Essentials

Zero Downtime DB Migration

Zero Downtime DB Migration