For my PhD research I studied several large software product development organizations to get a better understanding flow and impediments. Flow-based development is a rich and growing field with many concepts; the specific focus for this study is impediments to flow. This study takes the perspective that organizations are complex adaptive systems. This research uses sensemaking to get a richer, more-informed understanding of flow, impediments, and the context and culture of the organizations that are experiencing impediments to flow. The organizations that are part of this study are all large software product development organizations. The focus of this study, then, narrows to managing impediments to flow in large software product development organizations, using a sensemaking and complexity perspective.
More details, including a link to my thesis, can be found here.
This is the abstract from a paper I wrote about my experiences using sensemaking in large-scale transformation efforts. I presented this at the 49th Hawaii International Conference on System Sciences (HICSS 2016). The final paper is available as part of the HICSS proceedings. A pre-print is available here.
For organizations undergoing agile and lean transformation, it can be difficult to get meaningful, actionable insights into progress and impediments. Teams and organizations are best understood as complex adaptive human systems. Understanding what is happening in such systems requires approaches grounded in the complexity sciences and social sciences. This paper describes an approach using complexity science and sensemaking that helps an organization understand its culture, how it is progressing with its strategic initiatives, and the types of impediments that are holding it back. It provides a means of qualitative and quantitative analysis that helps teams and organizations improve. This paper also correlates the experiences of the people in the organization to its goals of being a more agile organization.
This is the abstract and summary of lessons learned from an experience report I wrote and presented at the Agile 2015 conference in Washington DC. The full paper is available here. Among other things, the paper talks about using A3 problem solving, Cynefin, and the Containers, Differences, Exchanges model from Human Systems Dynamics in the context of portfolio management in large organizations.
Working in a multi-team, multi-program, multi-product environment brings several challenges. One of those is providing a smooth flow of work to teams, and incorporating their feedback, while staying responsive to the needs of the business in a changing environment. Managing the portfolio backlog is a critical piece of the solution. This Experience Report documents several years’ experience working in such environments. The focus of this Experience Report is specifically on managing the portfolio backlog, not the full scope of what could be considered under a portfolio management strategy and implementation. We have found that getting the portfolio backlog management strategy right is a key element in the success of the overall portfolio management approach.
Summary of Lessons Learned
This section summarizes some of the key lessons learned in managing portfolio backlogs. Some general lessons related to solving problems in organizations include:
- Understanding the nature of the problem helps us to take appropriate action to solve the problem. The Cynefin framework helps with this.
- Make sure you are solving actual problems and causes, not just symptoms. A3 problem solving helps with this.
- Understand how to create a balance between agility, self-organization and coherence. HSD and the CDE model helps with this.
- Focus on the end-to-end flow of value through your organization, and on actively removing anything that impedes the flow of work. Lean thinking helps with this.
- Understand what success and failure could look like before running your experiments. This will help you pay beselective about the patterns you pay attention to.
Some specific lessons related to managing portfolio backlogs in large organizations include:
- Define the focus of your portfolio. In general, it is good practice to base the portfolio structure on your product line rather than organization structure. The former is what your customers care about; the latter more temporal.
- Understand what content goes on the portfolio backlog. Define different types of items, e.g., features, initiatives, architecture items, etc.
- Focus on the flow of work from portfolio to teams. The portfolio backlog management approach is an enabler of flow. Define policies for centralized portfolio-level decisions and localized program- and team-level decisions.
- Set up a portfolio backlog management meeting at a regular cadence with the right participants. Create a Definition of Ready for portfolio items. Focus the meeting on feedback from the development teams, and on moving portfolio backlog items to a ‘ready’ state. Do not let it become a status or strategy planning meeting.
- Create conditions that encourage a strong relationship between product managers, engineering leaders and architects. Together they bring multiple important perspectives to creating the portfolio backlog items. Consider also adding user experience design leaders to this mix, depending on the nature of your products.
Finally, this is a process of continuous experimentation and improvement. While some things can ultimately be moved to the obvious domain of best practices, or the complicated domain of good practices, we still operate within an ever-changing and complex environment that requires continuous awareness, experimentation, learning and adaptation. We continue to experiment and make improvements.
This abstract is from a paper I co-wrote with Kieran Conboy for the 37th International Conference on Software Engineering (ICSE 2015) in Firenze, Italy. The final paper is available in the conference proceedings. A pre-print version is available here.
Contemporary lean thinking, especially in knowledge work areas like software engineering, begins with understanding flow. Architecture plays a vital role in enabling the flow of value in software engineering teams and organizations. To date there has been little research in understanding impediments to flow in software engineering organizations. A focus on enabling flow through removing impediments is a useful perspective in creating a more agile, lean thinking software engineering organization. Particularly so when supported by appropriate metrics. This paper presents a case study of how architecture-related impediments impact the flow of work in software engineering teams and organizations. The key contributions of this paper are centered on the concept of flow and impediments in modern software engineering, and its relationship with architecture. We develop an understanding of how a focus on flow and removing impediments, supported by appropriate metrics, is helpful in identifying architecture-related challenges . Drawing on research of one company’s practices the paper presents an example of a scenario where flow analysis using specific metrics reveals architecture-related impediments and shows how addressing these impediments improves effectiveness and productivity in ways that would not otherwise have been revealed.
This is the abstract and conclusions from a paper I presented at Agile 2014. The full text is here.
When adopting agile and lean approaches in our company, one goal for teams and organizations is to achieve a smooth end-to-end flow of work through the system. This paper presents a useful set of metrics that reveal how work is flowing. It describes four metrics we find useful: Cumulative Flow, Throughput Analysis combined with Demand Analysis, Cycle Time and Lead Time.
These metrics help you understand Flow in your teams and organizations. In particular:
- CFDs give deeper insight into what’s happening in queues or workflow states, and help diagnose problems.
- Throughput Analysis shows how work is flowing through our system over time. It is even more useful when combined with a Demand Analysis that shows the proportion of work flowing through the systemthat is Value Demand versus Failure Demand.
- Cycle Time analysis shows how long it takes for work items to pass through one or a subset of workflowstates. This enables teams to make predictions about how long it takes to process planned work items.
- Lead Time analysis shows how long it takes for work items to pass through the entire organization. This enables the organization to make predictions about how long it will take to process requests. We generally use Lead Time to understand the time it takes work to pass through all states, from the moment there is arequest or idea, to the moment the work is complete and in the hands of customers.
- All these metrics can be used to indicate the presence of impediments to Flow in your system. The combination of these metrics offers good insight into what’s happening in an organization. They provide insight and visibility on status, and inform forecasting around when specific content might be delivered.
This is the abstract and conclusions from a short paper I wrote and presented at the 15th International Conference on Agile Software Development (XP 2014) in Rome, Italy. A preprint version of the full paper is available here.
Teams and organizations are complex adaptive systems. Self- organization in complex adaptive systems evolves through a set of Simple Rules. Self-organization is a core tenet of agile teams. Self-organization does not mean everyone gets to do whatever they want to do. Team members create contracts with each other. These contracts create boundaries, or containers, within which self-organization can occur. Teams also create contracts with other teams, the wider organization and other stakeholders. The contracts are both implicit and explicit. Social contracts in complex adaptive systems are more effective if they are based on Simple Rules. Social Contract Theory acts as a lens through which we can better understand these social contracts in agile teams. This paper represents ongoing research that examines the role of Simple Rules and Social Contract Theory in fostering self-organization in agile development teams. The paper discusses four examples of social contracts in agile teams: definition of done, definition of ready, working agreements, and retrospectives.
This paper described the connection between Social Contract Theory and agile teams, viewing agile teams as complex adaptive systems. The field of Human Systems Dynamics provides a suitable lens through which to view teams and organizations as complex adaptive social systems, and defines necessary conditions for self- organization using Containers, Differences and Exchanges. The social contracts in agile teams and organizations are based on the Simple Rules that govern emergence and self-organization.
Simple Rules support coherent behaviors in a system. Definition of done, definition of ready, and working agreements are all examples of social contracts, created using Simple Rules, in agile teams and organizations. In addition, there are examples of social contracts to be found in retrospectives, including the prime directive, second directive and ground rules. These Simple Rules and Social Contracts support emergent behaviors and self-organization.
Teams own their own Simple Rules. As teams adapt their Simple Rules, new patterns are formed in the system. These patterns are governed by the social contracts created by the Simple Rules. Violating the Simple Rules creates a tension in the system that can be resolved by the team enforcing the rules or altering the rules (an Exchange intervention), or by the team membership changing (a Container intervention).
Social Contracts exist within agile teams, between agile teams, between agile teams and management, and within management teams.
Good research is critical to building the body of knowledge of software engineering, and understanding the principles on which our industry is built. Even more, without the solid foundation of theory that comes from research, it is difficult to scale the practices we use in industry beyond a limited context. Lero, the Irish Software Engineering Research Centre, hosted an Industry Research Day today, to highlight and share some of the great software engineering research work that goes on at the Centre.
There are some great talks from partner companies including IBM, Intel, and St. James’s Hospital, Dublin. There are keynotes by Máire Geoghegan-Quinn, the current European European Commissioner for Research, Innovation and Science, and Seán Sherlock, the current Minister of State for Research and Innovation. It is great to see the richness and variety of software engineering research going on in Ireland, and the support available at both an EU and national government level.
My talk is a short summary of my ongoing PhD research work. I talk about learning and feedback cycles, learning to see impediments to flow, and some examples of how to see impediments in your team and organisation. I also talked about some preliminary research results, including how to tell who is really influencing flow and impediments in your organisation, what reaction time can tell us about threats and opportunities, and how to empower teams and engage management through an impediment removal process.
Slides from my talk are available here: