Organization and Business Agility: Managing the Portfolio Backlog in Large Organizations

This is the abstract and summary of lessons learned from an experience report I wrote and presented at the Agile 2015 conference in Washington DC. The full paper is available here. Among other things, the paper talks about using A3 problem solving, Cynefin, and the Containers, Differences, Exchanges model from Human Systems Dynamics in the context of portfolio management in large organizations.

Abstract

Working in a multi-team, multi-program, multi-product environment brings several challenges. One of those is providing a smooth flow of work to teams, and incorporating their feedback, while staying responsive to the needs of the business in a changing environment. Managing the portfolio backlog is a critical piece of the solution. This Experience Report documents several years’ experience working in such environments. The focus of this Experience Report is specifically on managing the portfolio backlog, not the full scope of what could be considered under a portfolio management strategy and implementation. We have found that getting the portfolio backlog management strategy right is a key element in the success of the overall portfolio management approach.

Summary of Lessons Learned

This section summarizes some of the key lessons learned in managing portfolio backlogs. Some general lessons related to solving problems in organizations include:

  • Understanding the nature of the problem helps us to take appropriate action to solve the problem. The Cynefin framework helps with this.
  • Make sure you are solving actual problems and causes, not just symptoms. A3 problem solving helps with this.
  • Understand how to create a balance between agility, self-organization and coherence. HSD and the CDE model helps with this.
  • Focus on the end-to-end flow of value through your organization, and on actively removing anything that impedes the flow of work. Lean thinking helps with this.
  • Understand what success and failure could look like before running your experiments. This will help you pay beselective about the patterns you pay attention to.

Some specific lessons related to managing portfolio backlogs in large organizations include:

  • Define the focus of your portfolio. In general, it is good practice to base the portfolio structure on your product line rather than organization structure. The former is what your customers care about; the latter more temporal.
  • Understand what content goes on the portfolio backlog. Define different types of items, e.g., features, initiatives, architecture items, etc.
  • Focus on the flow of work from portfolio to teams. The portfolio backlog management approach is an enabler of flow. Define policies for centralized portfolio-level decisions and localized program- and team-level decisions.
  • Set up a portfolio backlog management meeting at a regular cadence with the right participants. Create a Definition of Ready for portfolio items. Focus the meeting on feedback from the development teams, and on moving portfolio backlog items to a ‘ready’ state. Do not let it become a status or strategy planning meeting.
  • Create conditions that encourage a strong relationship between product managers, engineering leaders and architects. Together they bring multiple important perspectives to creating the portfolio backlog items. Consider also adding user experience design leaders to this mix, depending on the nature of your products.

Finally, this is a process of continuous experimentation and improvement. While some things can ultimately be moved to the obvious domain of best practices, or the complicated domain of good practices, we still operate within an ever-changing and complex environment that requires continuous awareness, experimentation, learning and adaptation. We continue to experiment and make improvements.

Social Contracts, Simple Rules, and Self-Organization

This is the abstract and conclusions from a short paper I wrote and presented at the 15th International Conference on Agile Software Development (XP 2014) in Rome, Italy. A preprint version of the full paper is available here.

Abstract

Teams and organizations are complex adaptive systems. Self- organization in complex adaptive systems evolves through a set of Simple Rules. Self-organization is a core tenet of agile teams. Self-organization does not mean everyone gets to do whatever they want to do. Team members create contracts with each other. These contracts create boundaries, or containers, within which self-organization can occur. Teams also create contracts with other teams, the wider organization and other stakeholders. The contracts are both implicit and explicit. Social contracts in complex adaptive systems are more effective if they are based on Simple Rules. Social Contract Theory acts as a lens through which we can better understand these social contracts in agile teams. This paper represents ongoing research that examines the role of Simple Rules and Social Contract Theory in fostering self-organization in agile development teams. The paper discusses four examples of social contracts in agile teams: definition of done, definition of ready, working agreements, and retrospectives.

Conclusions

This paper described the connection between Social Contract Theory and agile teams, viewing agile teams as complex adaptive systems. The field of Human Systems Dynamics provides a suitable lens through which to view teams and organizations as complex adaptive social systems, and defines necessary conditions for self- organization using Containers, Differences and Exchanges. The social contracts in agile teams and organizations are based on the Simple Rules that govern emergence and self-organization.

Simple Rules support coherent behaviors in a system. Definition of done, definition of ready, and working agreements are all examples of social contracts, created using Simple Rules, in agile teams and organizations. In addition, there are examples of social contracts to be found in retrospectives, including the prime directive, second directive and ground rules. These Simple Rules and Social Contracts support emergent behaviors and self-organization.

Teams own their own Simple Rules. As teams adapt their Simple Rules, new patterns are formed in the system. These patterns are governed by the social contracts created by the Simple Rules. Violating the Simple Rules creates a tension in the system that can be resolved by the team enforcing the rules or altering the rules (an Exchange intervention), or by the team membership changing (a Container intervention).
Social Contracts exist within agile teams, between agile teams, between agile teams and management, and within management teams.