Improving Flow in Large Software Product Development Organizations: A sensemaking and complex adaptive systems perspective

For my PhD research I studied several large software product development organizations to get a better understanding flow and impediments. Flow-based development is a rich and growing field with many concepts; the specific focus for this study is impediments to flow. This study takes the perspective that organizations are complex adaptive systems. This research uses sensemaking to get a richer, more-informed understanding of flow, impediments, and the context and culture of the organizations that are experiencing impediments to flow. The organizations that are part of this study are all large software product development organizations. The focus of this study, then, narrows to managing impediments to flow in large software product development organizations, using a sensemaking and complexity perspective.

More details, including a link to my thesis, can be found here.

A Metric-Based Approach to Managing Architecture-Related Impediments in Product Development Flow

This abstract is from a paper I co-wrote with Kieran Conboy for the 37th International Conference on Software Engineering (ICSE 2015) in Firenze, Italy. The final paper is available in the conference proceedings. A pre-print version is available here.

Abstract

Contemporary lean thinking, especially in knowledge work areas like software engineering, begins with understanding flow. Architecture plays a vital role in enabling the flow of value in software engineering teams and organizations. To date there has been little research in understanding impediments to flow in software engineering organizations. A focus on enabling flow through removing impediments is a useful perspective in creating a more agile, lean thinking software engineering organization. Particularly so when supported by appropriate metrics. This paper presents a case study of how architecture-related impediments impact the flow of work in software engineering teams and organizations. The key contributions of this paper are centered on the concept of flow and impediments in modern software engineering, and its relationship with architecture. We develop an understanding of how a focus on flow and removing impediments, supported by appropriate metrics, is helpful in identifying architecture-related challenges . Drawing on research of one company’s practices the paper presents an example of a scenario where flow analysis using specific metrics reveals architecture-related impediments and shows how addressing these impediments improves effectiveness and productivity in ways that would not otherwise have been revealed.

Metrics for Understanding Flow

This is the abstract and conclusions from a paper I presented at Agile 2014. The full text is here.

Abstract

When adopting agile and lean approaches in our company, one goal for teams and organizations is to achieve a smooth end-to-end flow of work through the system. This paper presents a useful set of metrics that reveal how work is flowing. It describes four metrics we find useful: Cumulative Flow, Throughput Analysis combined with Demand Analysis, Cycle Time and Lead Time.

Conclusions

These metrics help you understand Flow in your teams and organizations. In particular:

  • CFDs give deeper insight into what’s happening in queues or workflow states, and help diagnose problems.
  • Throughput Analysis shows how work is flowing through our system over time. It is even more useful when combined with a Demand Analysis that shows the proportion of work flowing through the systemthat is Value Demand versus Failure Demand.
  • Cycle Time analysis shows how long it takes for work items to pass through one or a subset of workflowstates. This enables teams to make predictions about how long it takes to process planned work items.
  • Lead Time analysis shows how long it takes for work items to pass through the entire organization. This enables the organization to make predictions about how long it will take to process requests. We generally use Lead Time to understand the time it takes work to pass through all states, from the moment there is arequest or idea, to the moment the work is complete and in the hands of customers.
  • All these metrics can be used to indicate the presence of impediments to Flow in your system. The combination of these metrics offers good insight into what’s happening in an organization. They provide insight and visibility on status, and inform forecasting around when specific content might be delivered.

Software Engineering Research in Product Development Flow

Research

Good research is critical to building the body of knowledge of software engineering, and understanding the principles on which our industry is built. Even more, without the solid foundation of theory that comes from research, it is difficult to scale the practices we use in industry beyond a limited context. Lero, the Irish Software Engineering Research Centre, hosted an Industry Research Day today, to highlight and share some of the great software engineering research work that goes on at the Centre.

There are some great talks from partner companies including IBM, Intel, and St. James’s Hospital, Dublin. There are keynotes by Máire Geoghegan-Quinn, the current European European Commissioner for Research, Innovation and Science, and Seán Sherlock, the current Minister of State for Research and Innovation. It is great to see the richness and variety of software engineering research going on in Ireland, and the support available at both an EU and national government level.

My talk is a short summary of my ongoing PhD research work. I talk about learning and feedback cycles, learning to see impediments to flow, and some examples of how to see impediments in your team and organisation. I also talked about some preliminary research results, including how to tell who is really influencing flow and impediments in your organisation, what reaction time can tell us about threats and opportunities, and how to empower teams and engage management through an impediment removal process.

Slides from my talk are available here:

Portfolio Management and Organization Flow

I’ll be speaking at RallyON Europe this September on the topic of Portfolio Management and Organization Flow.

Complexity of product development increases as we move from a single team or product focus to a cross-organisation portfolio focus. Organisation Flow is about achieving the Lean concept of flow at an organisation level, not just at the level of a single product or product line.

This session will look at an approach to portfolio backlog management, and describe how to manage the flow of work through a portfolio of multiple products delivered by 50 teams in 6 locations across the US, Europe, India and China.

We will include examples of some core metrics that help understand throughput and flow in the organisation. These metrics tell a story about what is happening in the organisation. Lessons from these stories include understanding impediments to flow, and understanding who and what in the organisation is influencing flow.

I did an interview last week with Hannah Shain from Rally. The recording from the interview is here:

There’s a great lineup of content for the conference. If you’re in London I hope to see you there!

Update: Here is the slide deck I used at the conference:

Lean Startup in Large Organizations

I gave a talk today at a Lean Startup event organized by Enterprise Ireland and ITAG.

The topic of my talk was Lean Startup in large organizations, and specifically talking about lessons learned from applying Lean Product Development and Lean Startup principles at Cisco.

The three specific lessons I talked about are:

  1. Learning to manage batch sizes and limit Work in Process (WIP)
  2. Customer Development
  3. Learning to see and manage waste

The slide deck is available via Slideshare: