Software product development organizations operate in an environment of ever-increasing volatility, uncertainty, complexity, and ambiguity. The pace of change is accelerating, business and technology complexity is growing, and organizations are struggling to keep pace. The software development industry has a $300 billion productivity problem, according to one study. Value is not flowing as it should. Flow-based software development is part of the continued evolution of contemporary software development approaches contributing to addressing this problem. It builds on agile and lean software development approaches and incorporates lessons from Deming’s management method, the Toyota Production System, Lean Product Development, Theory of Constraints, Operations Management, and other influences. Flow-based development is foundational to modern systems approaches, including DevOps, Continuous Delivery, Site Reliability Engineering, and more. Creating and sustaining flow in organizations is a challenging problem. Drawing on insights developed over many years working with multiple global product development organizations, this session presents a framework for establishing, understanding, and sustaining flow in organizations.
For my PhD research I studied several large software product development organizations to get a better understanding flow and impediments. Flow-based development is a rich and growing field with many concepts; the specific focus for this study is impediments to flow. This study takes the perspective that organizations are complex adaptive systems. This research uses sensemaking to get a richer, more-informed understanding of flow, impediments, and the context and culture of the organizations that are experiencing impediments to flow. The organizations that are part of this study are all large software product development organizations. The focus of this study, then, narrows to managing impediments to flow in large software product development organizations, using a sensemaking and complexity perspective.
This is the abstract from a paper I wrote about my experiences using sensemaking in large-scale transformation efforts. I presented this at the 49th Hawaii International Conference on System Sciences (HICSS 2016). The final paper is available as part of the HICSS proceedings. A pre-print is available here.
For organizations undergoing agile and lean transformation, it can be difficult to get meaningful, actionable insights into progress and impediments. Teams and organizations are best understood as complex adaptive human systems. Understanding what is happening in such systems requires approaches grounded in the complexity sciences and social sciences. This paper describes an approach using complexity science and sensemaking that helps an organization understand its culture, how it is progressing with its strategic initiatives, and the types of impediments that are holding it back. It provides a means of qualitative and quantitative analysis that helps teams and organizations improve. This paper also correlates the experiences of the people in the organization to its goals of being a more agile organization.
This is the abstract and summary of lessons learned from an experience report I wrote and presented at the Agile 2015 conference in Washington DC. The full paper is available here. Among other things, the paper talks about using A3 problem solving, Cynefin, and the Containers, Differences, Exchanges model from Human Systems Dynamics in the context of portfolio management in large organizations.
Working in a multi-team, multi-program, multi-product environment brings several challenges. One of those is providing a smooth flow of work to teams, and incorporating their feedback, while staying responsive to the needs of the business in a changing environment. Managing the portfolio backlog is a critical piece of the solution. This Experience Report documents several years’ experience working in such environments. The focus of this Experience Report is specifically on managing the portfolio backlog, not the full scope of what could be considered under a portfolio management strategy and implementation. We have found that getting the portfolio backlog management strategy right is a key element in the success of the overall portfolio management approach.
Summary of Lessons Learned
This section summarizes some of the key lessons learned in managing portfolio backlogs. Some general lessons related to solving problems in organizations include:
Understanding the nature of the problem helps us to take appropriate action to solve the problem. The Cynefin framework helps with this.
Make sure you are solving actual problems and causes, not just symptoms. A3 problem solving helps with this.
Understand how to create a balance between agility, self-organization and coherence. HSD and the CDE model helps with this.
Focus on the end-to-end flow of value through your organization, and on actively removing anything that impedes the flow of work. Lean thinking helps with this.
Understand what success and failure could look like before running your experiments. This will help you pay beselective about the patterns you pay attention to.
Some specific lessons related to managing portfolio backlogs in large organizations include:
Define the focus of your portfolio. In general, it is good practice to base the portfolio structure on your product line rather than organization structure. The former is what your customers care about; the latter more temporal.
Understand what content goes on the portfolio backlog. Define different types of items, e.g., features, initiatives, architecture items, etc.
Focus on the flow of work from portfolio to teams. The portfolio backlog management approach is an enabler of flow. Define policies for centralized portfolio-level decisions and localized program- and team-level decisions.
Set up a portfolio backlog management meeting at a regular cadence with the right participants. Create a Definition of Ready for portfolio items. Focus the meeting on feedback from the development teams, and on moving portfolio backlog items to a ‘ready’ state. Do not let it become a status or strategy planning meeting.
Create conditions that encourage a strong relationship between product managers, engineering leaders and architects. Together they bring multiple important perspectives to creating the portfolio backlog items. Consider also adding user experience design leaders to this mix, depending on the nature of your products.
Finally, this is a process of continuous experimentation and improvement. While some things can ultimately be moved to the obvious domain of best practices, or the complicated domain of good practices, we still operate within an ever-changing and complex environment that requires continuous awareness, experimentation, learning and adaptation. We continue to experiment and make improvements.
This abstract is from a paper I co-wrote with Kieran Conboy for the 37th International Conference on Software Engineering (ICSE 2015) in Firenze, Italy. The final paper is available in the conference proceedings. A pre-print version is available here.
Contemporary lean thinking, especially in knowledge work areas like software engineering, begins with understanding flow. Architecture plays a vital role in enabling the flow of value in software engineering teams and organizations. To date there has been little research in understanding impediments to flow in software engineering organizations. A focus on enabling flow through removing impediments is a useful perspective in creating a more agile, lean thinking software engineering organization. Particularly so when supported by appropriate metrics. This paper presents a case study of how architecture-related impediments impact the flow of work in software engineering teams and organizations. The key contributions of this paper are centered on the concept of flow and impediments in modern software engineering, and its relationship with architecture. We develop an understanding of how a focus on flow and removing impediments, supported by appropriate metrics, is helpful in identifying architecture-related challenges . Drawing on research of one company’s practices the paper presents an example of a scenario where flow analysis using specific metrics reveals architecture-related impediments and shows how addressing these impediments improves effectiveness and productivity in ways that would not otherwise have been revealed.
When adopting agile and lean approaches in our company, one goal for teams and organizations is to achieve a smooth end-to-end flow of work through the system. This paper presents a useful set of metrics that reveal how work is flowing. It describes four metrics we find useful: Cumulative Flow, Throughput Analysis combined with Demand Analysis, Cycle Time and Lead Time.
These metrics help you understand Flow in your teams and organizations. In particular:
CFDs give deeper insight into what’s happening in queues or workflow states, and help diagnose problems.
Throughput Analysis shows how work is flowing through our system over time. It is even more useful when combined with a Demand Analysis that shows the proportion of work flowing through the systemthat is Value Demand versus Failure Demand.
Cycle Time analysis shows how long it takes for work items to pass through one or a subset of workflowstates. This enables teams to make predictions about how long it takes to process planned work items.
Lead Time analysis shows how long it takes for work items to pass through the entire organization. This enables the organization to make predictions about how long it will take to process requests. We generally use Lead Time to understand the time it takes work to pass through all states, from the moment there is arequest or idea, to the moment the work is complete and in the hands of customers.
All these metrics can be used to indicate the presence of impediments to Flow in your system. The combination of these metrics offers good insight into what’s happening in an organization. They provide insight and visibility on status, and inform forecasting around when specific content might be delivered.
Good research is critical to building the body of knowledge of software engineering, and understanding the principles on which our industry is built. Even more, without the solid foundation of theory that comes from research, it is difficult to scale the practices we use in industry beyond a limited context. Lero, the Irish Software Engineering Research Centre, hosted an Industry Research Day today, to highlight and share some of the great software engineering research work that goes on at the Centre.
There are some great talks from partner companies including IBM, Intel, and St. James’s Hospital, Dublin. There are keynotes by Máire Geoghegan-Quinn, the current European European Commissioner for Research, Innovation and Science, and Seán Sherlock, the current Minister of State for Research and Innovation. It is great to see the richness and variety of software engineering research going on in Ireland, and the support available at both an EU and national government level.
My talk is a short summary of my ongoing PhD research work. I talk about learning and feedback cycles, learning to see impediments to flow, and some examples of how to see impediments in your team and organisation. I also talked about some preliminary research results, including how to tell who is really influencing flow and impediments in your organisation, what reaction time can tell us about threats and opportunities, and how to empower teams and engage management through an impediment removal process.
Complexity of product development increases as we move from a single team or product focus to a cross-organisation portfolio focus. Organisation Flow is about achieving the Lean concept of flow at an organisation level, not just at the level of a single product or product line.
This session will look at an approach to portfolio backlog management, and describe how to manage the flow of work through a portfolio of multiple products delivered by 50 teams in 6 locations across the US, Europe, India and China.
We will include examples of some core metrics that help understand throughput and flow in the organisation. These metrics tell a story about what is happening in the organisation. Lessons from these stories include understanding impediments to flow, and understanding who and what in the organisation is influencing flow.
I did an interview last week with Hannah Shain from Rally. The recording from the interview is here:
There’s a great lineup of content for the conference. If you’re in London I hope to see you there!
Update: Here is the slide deck I used at the conference:
As of May 2012, the Lean Software and Systems Society has officially re-branded as the Lean Systems Society. As part of the launch, a number of founding Fellows have been named. I am honored and delighted to have been invited to be a founding Fellow of the Lean Systems Society.
“to improve the world by improving its systems. The Society organization will be modeled on the United Kingdom’s Royal Society and its “Fellowship of the world’s most eminent scientists.” The Royal Society was created to actively encourage thought leadership in the sciences by honoring original thinkers as Fellows, and encouraging their collaboration and debate. It has succeeded in this mission for over 300 years, and is an ideal model to emulate.“
“The Lean Systems Society believes that excellence in managing complexity requires accepting that complexity and uncertainty are natural to social systems and knowledge work. Effective systems must produce both better economic and sociological outcomes. Their development requires a holistic approach that incorporates the human condition. The Society is committed to exploring valuable ideas from all disciplines, and fostering a community that derives solutions from a common set of values and principles, while embracing specific context and avoiding dogma.“
The official Lean Systems Society press release can be found here. There are additional press releases on Yahoo News, InfoQ and others.
Looking to the Future
I’m looking forward to continuing to contribute in whatever way I can to the Lean community, and to helping the Lean Systems Society to grow and flourish. The Society’s purpose of “improving the world by improving its systems” is a stirring and timely call to action for all of us.
Jim Benson and Jeremy Lightsmith founded the Lean Coffee movement in Seattle in 2009. Since then, Lean Coffee events have sprung up around the world. The basic idea is for a group of people to get together to discuss topics around which they share a common interest, specifically around agile, lean, kanban, lean startup, etc. There is also an OpenCoffee movement, founded by Saul Klein in 2007. The intent behind OpenCoffee is to provide an open forum for investors, entrepreneurs and developers to come together, meet, discuss ideas, and find opportunities to work together.
The Lean Coffee format is essentially an approach to facilitating learning and collaboration through group discussions. The ‘Lean’ part of the name has its roots in Lean Thinking, and related areas of Lean Production, Lean Software, Lean Startup, etc. The ‘Coffee’ part of the name obviously comes from that nice drink that some of us are partial to. Meetups typically take place in the morning, at a local coffee house, at the same time and place each week.
Lean Coffee events are increasingly becoming a fixture at many conferences, in a similar way that Open Space has. We had a great Lean Coffee event at LESS2011 in Stockholm last year.
How it works
The Lean Coffee format has a lot in common with Open Space, particularly in the sense that you are asking people to move away from an agenda-driven gathering to a more open, collaborative, dynamic, emergent type of gathering.
Everyone offers topics they are interested in discussing by writing them on index cards (or sticky notes).
Everyone presents his or her topic as an Elevator Pitch (max 1 minute per topic).
Use dot voting to vote on each of the topics. Each person gets two votes. You can only vote once per topic.
Prepare three columns on a table or wall. Call them “Planned”, “In Progress” and “Done”. Add all topics to the “Planned” column, with those that received the highest votes at the top.
Discuss each topic in turn. Move the index card for the topic into the “In Progress” column. Initially, ask the proposer to explain the topic, then go round the table to give an opportunity for everyone to provide an initial comment, then open discussion.
When the topic is done, move on to the next one. The topic proposer decides when the topic is done, and moves the index card to the “Done” column. If someone disagrees, then s/he can raise a new topic. Expect to discuss 3-4 topics over the space of an hour or so.
Consider time boxing each topic to 15 minutes max.
At the end of the overall Lean Coffee session, run a quick retrospective. What did you like? What didn’t you like? What are ideas for improvement?
As a variation, particularly if you have a large group, consider splitting into sub-groups if people are particularly passionate about specific topics. Then get back together after the topic discussion to present highlights to, and get feedback from, the wider group.
Lean Coffee at work
The idea and format for Lean Coffee is very simple and effective. It can be used in work too. For example, consider hosting a Lean Coffee event every week in your office. It can be a great way to share experiences and challenges across teams and gather interested people who don’t normally get to discuss these topics together.
If you find your retrospectives are becoming routine, then this is a good way to break out of the rut. There is no leader or facilitator required. The team can self-organize a Lean Coffee style event without any preparation. The team decides which topics to discuss, and how long to spend on each topic.
The only prerequisites are a set of index cards or sticky notes, a set of Sharpies, and some interested and willing people.
Some sites for Lean Coffee Meetups around the world:
Drury, M., Conboy, K., and Power, K. (2012). “Obstacles to decision making in Agile software development teams.” Journal of Systems and Software, 85(6), 1239-1254.
The obstacles facing decision making in Agile development are critical yet poorly understood. This research examines decisions made across four stages of the iteration cycle: Iteration Planning, Iteration Execution, Iteration Review and Iteration Retrospective. A mixed method approach was employed, whereby a focus group was initially conducted with 43 Agile developers and managers to determine decisions made at different points of the iteration cycle. Subsequently, six illustrative mini cases were purposefully conducted as examples of the six obstacles identified in these focus groups. This included interviews with 18 individuals in Agile projects from five different organizations: a global consulting organization, a multinational communications company, two multinational software development companies, and a large museum organization. This research contributes to Agile software development literature by analyzing decisions made during the iteration cycle and identifying six key obstacles to these decisions. Results indicate the six decision obstacles are unwillingness to commit to decisions; conflicting priorities; unstable resource availability; and lack of: implementation; ownership; empowerment. These six decision obstacles are mapped to descriptive decision making principles to demonstrate where the obstacles affect the decision process. The effects of these obstacles include a lack of longer-term, strategic focus for decisions, an ever-growing backlog of delayed work from previous iterations, and a lack of team engagement.
I was having lunch with Jurgen, Olaf and others at XP2011 in Madrid earlier this year. The details for what would become the first ALE conference would start to be worked out later that week. Jurgen asked me if I had any thoughts about what we could do that would be a little different. I thought it would be good to find a way to include spouses and children in the conference program – not just bring them to wherever city the conference was going to be in, but actually integrate them into the conference.
Last week I wrote about Post-It Wars, and how we tried it out at work. I had the chance to hang out with families of other conference attendees (and my own family) at ALE2011, so I got them all to play too. We started off in the room that was reserved for families. The kids had a great time. After some negotiation, they quickly agreed on the music, picking out the Beatles and Bob Marley. Music blaring, they got to work. It was amazing to watch as a group of kids who had never met until a few hours earlier came together to create pictures using Post It Notes.
While the other conference attendees were in one of the sessions, we broke out into the coffee break area (bringing music with us) and used the windows there too (top left). The parents got in on the action too, and had fun with it. I got some pictures of the work-in-progress from outside the hotel (top right), and their work attracted the attention of some passers by (bottom center).
The hotel windows provided a great canvas to work from, and fortunately the hotel staff were very understanding 🙂 Here is what their work looked like from outside the hotel:
Afterwards, we went outside to admire their great art work. Never one to pass up on some fun, Mike came too 🙂
The Marshmallow Challenge is a fun game that I often play with teams. You can play it with friends and family too. At ALE2011 I played it with the ALE families. This time they picked Red Hot Chilli Peppers, U2, Bon Jovi and the Beatles. Rock’n’Roll!
Other Games and Activities
Oana Jancu, whose kids were also there, taught everyone ‘Where Are Your Keys?“, an amazing game for learning new skills, particularly new languages. The kids used the game to learn some Portuguese, French, and Irish. Monika played lots of games, arts and crafts activities. Christiane spent all day Wednesday and Friday with the families. She taught them origami and brought them to the Zoo and Natural History Museum.
That’s an elephant’s tooth the tour guide is holding.
Why Include Spouse and Children in a Conference?
There are at least three big reasons why I think its important to include a Spouse and Kids Program in the conference:
Some of us attend a lot of conferences. That’s a lot of time away from home and family. Being able to bring them with us and hang out during and around the conference is a good thing. Even if you go to only one conference, it’s still nice to be able to bring your spouse and kids. Of course you can bring your family on any trip; the difference with ALE2011 is they were an integrated part of the conference. I like that my wife and children could meet some of the people I have come to know and consider friends. It’s fantastic that my kids can have the opportunity to make friends with other kids from around the world. Vasco tweeted that it made his quality of life better; I concur.
It gives our families a chance to see what we do at conferences. It can be hard sometimes to explain to our families what we do for a living. This gives them some insights.
It is an opportunity to inspire kids to consider a career in our industry. Many countries are struggling to find people with the right skills to fill job vacancies, and there is a growing shortage of children (particularly girls) taking science, maths and engineering courses in school and university. This is a real crisis for our profession. Attending a fun conference can leave them with positive feelings and memories. If they can see that what we do can be a fun and rewarding path, and spread the word to their friends, that can only be a good thing for the future of our industry.
It was a great pleasure to meet and spend time with the families of Oana, Vasco, Kurt, Andrea and others. I hope this is a tradition we can continue at ALE conferences, and maybe even extend to some other conferences. Based on feedback so far, including the retrospective output, it seems to have been a positive experience overall. Let’s see how we can make it even better for next time.
The main theme of my talk was in support of applying concepts from Stakeholder Management to support agile and lean adoption. I used the concept of refactoring to talk about some of the changes that can be made to organizations, and showed some examples of how Martin Fowler’s classic Refactorings can be re-purposed to talk about organization design rather than software design.
The main topics of my talk were:
Using Dan Pink’s Motivation 3.0 Type-I toolkit for intrinsic motivation as a guide for what we should be refactoring towards
Refactoring applied to Organization Design
Agile transition journeys
Designing a process: core framework versus toolbox
“Every organization has a design. As an organization grows, that design evolves. A decision to embrace agile and lean methods can expose weaknesses in the design. The concept of refactoring as applied to software design helps to improve the overall structure of the product or system. Principles of refactoring can also be applied to organization design. As with software design, the design of our organization can benefit from deliberate improvement efforts, but those efforts must have a purpose, and must serve the broad community of stakeholders that affect, or are affected by, the organization. Refactoring to agile and lean organizations demands that we have a shared vision of what the refactoring needs to achieve, and that we optimize the organization around the people doing the work.“