Some insights from working with Flow-based product development

Software product development organizations operate in an environment of ever-increasing volatility, uncertainty, complexity, and ambiguity. The pace of change is accelerating, business and technology complexity is growing, and organizations are struggling to keep pace. The software development industry has a $300 billion productivity problem, according to one study. Value is not flowing as it should. Flow-based software development is part of the continued evolution of contemporary software development approaches contributing to addressing this problem. It builds on agile and lean software development approaches and incorporates lessons from Deming’s management method, the Toyota Production System, Lean Product Development, Theory of Constraints, Operations Management, and other influences. Flow-based development is foundational to modern systems approaches, including DevOps, Continuous Delivery, Site Reliability Engineering, and more. Creating and sustaining flow in organizations is a challenging problem. Drawing on insights developed over many years working with multiple global product development organizations, this session presents a framework for establishing, understanding, and sustaining flow in organizations.

I presented this session at the XP2020 conference. The slides are available here.

This session addressed the following questions:

  • What is flow?
  • What is the relationship between flow, lean, and agile?
  • Why do organizations adopt flow?
  • Where to get started?
  • What contributes to flow happening?
  • How do we measure flow?
  • What makes flow difficult to achieve?
  • How can organizations overcome these impediments to flow?
  • What is the role of culture in determining how effective flow can be?

Understanding and applying different decision models for increasing organization agility

Agility implies responding to change appropriately. Every response is a decision. Sometimes we choose among several competing options, sometimes not. Some decisions demand immediate action, others require us to step back and think and weigh our options. There are many decision models that can support how we respond. Often, our decisions are part of a series of interrelated decisions that influence each other. From this perspective, the agility of an organization can be viewed as an outcome of decisions made over time.

Rebecca Wirfs-Brock and I held a workshop session at XP2020 on this topic. In this workshop session, we discuss how to apply three specific decision styles and models that are useful in different circumstances.

Some of the models we use in this workshop come from the field of Naturalistic Decision Making (NDM), which has the goal of studying how people actually make decisions in a variety of real-world settings. Settings in which NDM is appropriate are characterized by time pressure, high stakes, experienced decision makers, inadequate information, ill-defined goals, poorly defined procedures, context, dynamic conditions, and team coordination. In this session we explore NDM in the context of organization agility.

The slides are available to download here.

Learning outcomes from the workshop include:

  • Learning about Decision Models and how they apply to Agile Software Development and Organization Agility.
  • Understanding of how to apply and combine different decision models in different contexts. Specifically, we explore analytical decision making, naturalistic decision making, and recognition-primed decision models.
  • Understand the role of heuristics and experiments in making decisions and taking action.
  • Understand the role of expertise and how it influences decision and action in organizations.
  • Understand the circumstances under which decisions are best taken by individuals, and those where decisions are best taken in groups.
  • Understand how decisions influence each other and compound over time.
  • Approaches for when and how to record decisions.

Improving Flow in Large Software Product Development Organizations: A sensemaking and complex adaptive systems perspective

For my PhD research I studied several large software product development organizations to get a better understanding flow and impediments. Flow-based development is a rich and growing field with many concepts; the specific focus for this study is impediments to flow. This study takes the perspective that organizations are complex adaptive systems. This research uses sensemaking to get a richer, more-informed understanding of flow, impediments, and the context and culture of the organizations that are experiencing impediments to flow. The organizations that are part of this study are all large software product development organizations. The focus of this study, then, narrows to managing impediments to flow in large software product development organizations, using a sensemaking and complexity perspective.

More details, including a link to my thesis, can be found here.

Organization and Business Agility: Managing the Portfolio Backlog in Large Organizations

This is the abstract and summary of lessons learned from an experience report I wrote and presented at the Agile 2015 conference in Washington DC. The full paper is available here. Among other things, the paper talks about using A3 problem solving, Cynefin, and the Containers, Differences, Exchanges model from Human Systems Dynamics in the context of portfolio management in large organizations.

Abstract

Working in a multi-team, multi-program, multi-product environment brings several challenges. One of those is providing a smooth flow of work to teams, and incorporating their feedback, while staying responsive to the needs of the business in a changing environment. Managing the portfolio backlog is a critical piece of the solution. This Experience Report documents several years’ experience working in such environments. The focus of this Experience Report is specifically on managing the portfolio backlog, not the full scope of what could be considered under a portfolio management strategy and implementation. We have found that getting the portfolio backlog management strategy right is a key element in the success of the overall portfolio management approach.

Summary of Lessons Learned

This section summarizes some of the key lessons learned in managing portfolio backlogs. Some general lessons related to solving problems in organizations include:

  • Understanding the nature of the problem helps us to take appropriate action to solve the problem. The Cynefin framework helps with this.
  • Make sure you are solving actual problems and causes, not just symptoms. A3 problem solving helps with this.
  • Understand how to create a balance between agility, self-organization and coherence. HSD and the CDE model helps with this.
  • Focus on the end-to-end flow of value through your organization, and on actively removing anything that impedes the flow of work. Lean thinking helps with this.
  • Understand what success and failure could look like before running your experiments. This will help you pay beselective about the patterns you pay attention to.

Some specific lessons related to managing portfolio backlogs in large organizations include:

  • Define the focus of your portfolio. In general, it is good practice to base the portfolio structure on your product line rather than organization structure. The former is what your customers care about; the latter more temporal.
  • Understand what content goes on the portfolio backlog. Define different types of items, e.g., features, initiatives, architecture items, etc.
  • Focus on the flow of work from portfolio to teams. The portfolio backlog management approach is an enabler of flow. Define policies for centralized portfolio-level decisions and localized program- and team-level decisions.
  • Set up a portfolio backlog management meeting at a regular cadence with the right participants. Create a Definition of Ready for portfolio items. Focus the meeting on feedback from the development teams, and on moving portfolio backlog items to a ‘ready’ state. Do not let it become a status or strategy planning meeting.
  • Create conditions that encourage a strong relationship between product managers, engineering leaders and architects. Together they bring multiple important perspectives to creating the portfolio backlog items. Consider also adding user experience design leaders to this mix, depending on the nature of your products.

Finally, this is a process of continuous experimentation and improvement. While some things can ultimately be moved to the obvious domain of best practices, or the complicated domain of good practices, we still operate within an ever-changing and complex environment that requires continuous awareness, experimentation, learning and adaptation. We continue to experiment and make improvements.

A Metric-Based Approach to Managing Architecture-Related Impediments in Product Development Flow

This abstract is from a paper I co-wrote with Kieran Conboy for the 37th International Conference on Software Engineering (ICSE 2015) in Firenze, Italy. The final paper is available in the conference proceedings. A pre-print version is available here.

Abstract

Contemporary lean thinking, especially in knowledge work areas like software engineering, begins with understanding flow. Architecture plays a vital role in enabling the flow of value in software engineering teams and organizations. To date there has been little research in understanding impediments to flow in software engineering organizations. A focus on enabling flow through removing impediments is a useful perspective in creating a more agile, lean thinking software engineering organization. Particularly so when supported by appropriate metrics. This paper presents a case study of how architecture-related impediments impact the flow of work in software engineering teams and organizations. The key contributions of this paper are centered on the concept of flow and impediments in modern software engineering, and its relationship with architecture. We develop an understanding of how a focus on flow and removing impediments, supported by appropriate metrics, is helpful in identifying architecture-related challenges . Drawing on research of one company’s practices the paper presents an example of a scenario where flow analysis using specific metrics reveals architecture-related impediments and shows how addressing these impediments improves effectiveness and productivity in ways that would not otherwise have been revealed.

Metrics for Understanding Flow

This is the abstract and conclusions from a paper I presented at Agile 2014. The full text is here.

Abstract

When adopting agile and lean approaches in our company, one goal for teams and organizations is to achieve a smooth end-to-end flow of work through the system. This paper presents a useful set of metrics that reveal how work is flowing. It describes four metrics we find useful: Cumulative Flow, Throughput Analysis combined with Demand Analysis, Cycle Time and Lead Time.

Conclusions

These metrics help you understand Flow in your teams and organizations. In particular:

  • CFDs give deeper insight into what’s happening in queues or workflow states, and help diagnose problems.
  • Throughput Analysis shows how work is flowing through our system over time. It is even more useful when combined with a Demand Analysis that shows the proportion of work flowing through the systemthat is Value Demand versus Failure Demand.
  • Cycle Time analysis shows how long it takes for work items to pass through one or a subset of workflowstates. This enables teams to make predictions about how long it takes to process planned work items.
  • Lead Time analysis shows how long it takes for work items to pass through the entire organization. This enables the organization to make predictions about how long it will take to process requests. We generally use Lead Time to understand the time it takes work to pass through all states, from the moment there is arequest or idea, to the moment the work is complete and in the hands of customers.
  • All these metrics can be used to indicate the presence of impediments to Flow in your system. The combination of these metrics offers good insight into what’s happening in an organization. They provide insight and visibility on status, and inform forecasting around when specific content might be delivered.