This page contains the abstract and summary from my PhD research. A PDF of the full thesis document is available for download from the open access research collection at the National University of Ireland, Galway.
Improving Flow in Large Software Product Development Organizations: A sensemaking and complex adaptive systems perspective
Software development organizations operate in an environment of ever- increasing volatility, uncertainty, complexity, and ambiguity. The pace of change is accelerating, business and technology complexity is growing, and organizations are struggling to keep pace. The software development industry has a $3 trillion productivity problem, according to one study. Value is not flowing as it should. Flow-based software development is part of the continued evolution of contemporary software development approaches contributing to addressing this problem. It builds on agile and lean software development approaches and incorporates lessons from Deming’s management method, the Toyota Production System, Lean Product Development, Theory of Constraints, Operations Management, and other influences. Flow-based development is foundational to modern systems approaches, including DevOps, Continuous Delivery, Site Reliability Engineering, and more. Creating and sustaining flow in organizations is a challenging problem. Despite this, there is a lack of rigorous research on the topic of impediments to flow.
Stephen Hawking famously predicted that the 21st century would be the century of complexity science. Software development organizations are complex adaptive systems (CAS). Sensemaking provides a means of exploring and understanding complex systems through making sense of people’s lived experiences in organizations. This study provides a sensemaking and CAS theory-informed understanding of product development flow in large software development organizations. To achieve its objective, this research studies ten large, globally distributed, software development organizations. Using the SenseMaker® software suite this study collects and analyzes micronarratives from people in these organizations, thereby providing insights into the patterns of organization culture that influence flow and impediments. The research methods for this qualitative study include a combination of focus group research, narrative research, and case study research. The unit of analysis is the software development organization.
The main contribution of this study is the development of an impediment management framework, related to which this study presents four sets of findings. First, this study provides an analysis of the system patterns that contribute to impediments to flow in organizations. In particular, this study analyzes system patterns identified from value streams, flow metrics, and organization culture. Second, this study explores factors related to how organizations identify impediments to flow. This study identifies eight common types of impediments that affect flow. The study further identifies nine common contributing factors to impediments and six common effects of impediments. Third, this study analyzes how people in organizations make sense of impediments. This research explores the use of the Cynefin sensemaking framework to make sense of the context of impediments in order to inform how to resolve them. Fourth, this study explores four approaches used by organizations to resolve impediments. In particular, this study explores the use of continuous improvement, A3 problem solving, experiment design, and system improvement goals in the context of resolving impediments and improving flow in organizations. This study employs the CDE (containers, differences, exchanges) framework to understand better the interventions that organizations make in order to resolve impediments and improve flow. The analysis shows how these interventions relate to the sensemaking domains of the Cynefin framework.
Ultimately, this study shows impediments continually emerge to affect flow in organizations. Some impediments are problems to be solved; others are indicators of patterns that need to shift in the organization. The quality of flow and the nature of the organization’s response to impediments reflects the organization’s culture. Improving flow requires attending to the patterns, identifying impediments, making sense of the impediments in the context of the system patterns, and taking context-appropriate action in the form of interventions to resolve impediments. This study adds to the body of knowledge of contemporary software development approaches, flow-based development, management, leadership, and sensemaking in organizations. The framework can guide practical approaches to improving flow in organizations and serves as a foundation for future research.
This study begins with a broad overview of lean product development and contemporary software development, highlighting how the former continues to influence the latter. Both lean product development and contemporary software development form the foundation for flow-based software development. Flow-based development is a rich and growing field with many concepts; the specific focus for this study is impediments to flow. This study takes the perspective that organizations are complex adaptive systems. This research uses sensemaking to get a richer, more-informed understanding of flow, impediments, and the context and culture of the organizations that are experiencing impediments to flow. The organizations that are part of this study are all large software product development organizations. The focus of this study, then, narrows to managing impediments to flow in large software product development organizations, using a sensemaking and complexity perspective.
This study, therefore, focuses first on understanding impediments, the contributing factors that lead to impediments, and the resulting effects of impediments. Next, this study employs approaches from lean management and complexity-informed management to help contemporary software organizations identify the patterns in their systems that are allowing the contributing factors to emerge. From there, the study explores how organizations can operate in environments of volatility, uncertainty, complexity, and ambiguity to identify impediments, make sense of impediments, and resolve them in a way that is appropriate to their context.