Agile vs Traditional Project Management: A Game-based Experiment

---
This post is our contribution to the blog parade of the
PM Camp Berlin 2015 with a focus on complexity in projects
---

THE GREATEST FIGHT IN PROJECT HISTORY

Agile vs. TraditionalAbstract: The popular “Ballpoint Game” was applied to tackle the question, whether a superior project management approach exists. Two different approaches have been compared: quick iterations (as known in “agile management”) versus in-depth conceptualization (as known in “the waterfall”). The experiment took place over the course of almost two years and involved more than 1,000 persons. It revealed that in complex-complicated* scenarios, a combination of project approaches leads to best results. If you don’t have the time to read the entire blog post, you can download our  presentation on slideshare

*Sometimes people confuse “complexity” and “complicatedness”. In fact, complexity and complicatedness are two different concepts. While complicated problems can be mathematically solved and thus have a clear “best” solution; complex problems depend on time, room, and social interaction. Therefore, complex problems are non-linear and have no “best” solution (which could be objectively tested and verified). To explain this difference, many authors stressed the comparison between “technical systems” (may be complicated, but never complex) and “living systems” (may be complex, even without being complicated). In projects, challenges are often neither purely complex nor neither purely complicated. They typically come together as complex-complicated challenges. The “Ballpoint Game” represents this type of challenge (see the end of this article for more info incl. some recommended readings). 

SETTING UP THE EXPERIMENT

Ballpoint GameThe “Ballpoint Game” is an exercise which is frequently used by the agile community to demonstrate the impact of agile thinking and acting. The goal of the game is simple: a group of persons needs to move as many balls as possible from a full basket to an empty one. In this process, the following rules apply:

  • Start point is end point (the baskets stand side-by-side).
  • Each person must touch the ball.
  • Balls must have air time.
  • Balls are not allowed to touch the ground (otherwise, a ball is out).
  • Balls are not allowed to be moved to the next neighbor.

The video gives an impression of the game.

For our experiment, we chose the “Ballpoint Game”, because

  • it can be applied (and repeated) in a laboratory,
  • it can be performed independent of specific “expert skills”, and
  • the results are objectively measurable.

Originally, the game is played in iterations. After an initial planning phase (2 minutes), a team has multiple runs (of two minutes each) to move as many balls as possible. After each run, a brief retrospect (one minute) can be used to modify and thus improve the solution. After five runs and a total playing time of 16 minutes, the game is completed.

For our experiment, we kept the original “agile” game, and introduced a second one, which represents the thinking school of “traditional” project management. In this second game configuration, a group of persons had a longer period of time (14 minutes) for consideration and conceptualization. In this 14-minutes-phase, people had no access to the tennis balls and baskets. Instead, they could use all types of material like pen, paper, whiteboard, cardboard, cord, etc. for planning and designing the solution. Eventually, the team had one single run (2 minutes) to implement their concept for moving as many balls as possible. Thus, the total playing time is 16 minutes, too.

The figure illustrates the two game configurations, “iterative” (agile) versus “in-depth” (traditional).

Ballpoint Game Set-up

CONDUCTING THE EXPERIMENT

The experiment was conducted over the course of 23 months. In total, more than 1,000 persons took part. The participants came from university (business students) and companies (business professionals). The size of the single groups varied between 22 and 38 persons.

For a single experiment, a group was split into 2 reference teams of equal size and diversity (regarding gender, age, nationality, PM expertise, etc.). After building the teams, a game supervisor was assigned to each team and the teams were led to two separate rooms. There, the supervisors instructed their teams. While one team was requested to solve the ballpoint challenge “in iterations” (Team 1), the other team got instructions for “in-depth planning” (Team 2). Each supervisor called for a brilliant solution and for beating the other team. No supervisor was allowed to help and give useful hints. And of course, the teams didn’t know of the different approaches.

After the 16-minutes-procedure, the supervisors exchanged results. Independent of the actual figures, they informed their team, that the other team was “much better”. Then, they instructed their team to beat the other team in a second round (“2nd Round”). The instruction for Team 1 – the team which started in an “iterative” way – was to take the second round for in-depth planning. And the instruction for Team 2 was to tackle the problem in iterations. That means, the teams swapped approaches (unknowingly).

The figure illustrates the arrangement for “Team 1” and “Team 2”.

Ballpoint Game Full Set-up

OBSERVATIONS

We compared the maximum number of balls per run (best result) in the first completed round (1st Round) and after swapping the approaches (2nd Round). Eventually, 31 challenges were evaluated. The most exciting results are as follows

  • After the 1st Round, the “iterative” team always wins. In other words, Team 1 wins 31 of 31 first rounds.
  • After the 2nd Round, Team 1 wins only 15 times; Team 2 wins 16 times.

FINDINGS

This result was quite surprising: Even if the “iterative” approach (Team 1) always succeeds after the first round; we don’t have a clear tendency for the total game. Quite the contrary, after two rounds –we (almost) have a draw. That means, the sequence of combining iterative and in-depth planning doesn’t make a difference.

This led to the assumption that combining approaches is superior to a single approach.

To test this assumption, we designed an additional experiment. Here, we kept the original set-up, except one major change: the second reference team (“Team B”) performed the same approach in both rounds. That means, they either played purely “agile” or purely “traditional”. The big surprise: none of the pure approaches could win a single game. In every case, the combination of “agile” and “traditional” wins (see table). This verifies the assumption:

A COMBINATION OF PROJECT APPROACHES BEATS THE PURE APPROACH!

Ballpoint Extended Experiment

DISCUSSION

The “Ballpoint Game” is both, complex and complicated. It’s complicated because it has some tricky restrictions and a clear best solution which can be figured out, proven and replicated. At the same time, the scenario is complex. There are many persons involved; group dynamics plays a big role and the sequence in which a solution is suggested and experienced makes a big difference.

As a complex-complicated scenario, the ballpoint game is a reasonable representative for the majority of business challenges (see “Additional Reading”). Thus the results of the described laboratory experiment are assumed to be applicable to “real-world” scenarios: Agile vs. traditional project management is not a question of either-or; both approaches should be combined!

Finally, I want to mention that the described experiment is a piece of exploratory qualitative research. And of course, despite all careful considerations, it underlies the restrictions of this research approach.

—–by Frank Habermann —

Additional Reading (in German)

  • Habermann, Frank: Hybrides Projektmanagement. agile und klassische Vorgehensmodelle im Zusammenspiel. In: HMD : Praxis der Wirtschaftsinformatik, Vol. 50, No. 293 (2013), S. 93-102.

Recomnended readings re complexity and complicatedness

  • Holland, J.H. (2006) “Studying Complex Adaptive Systems”, Journal of Systems Science and Complexity, Volume 19, Issue 1, pp 1-8.
  • Miller, J.L., Miller, J.G. (1992) “Greater Than the Sum of its Parts: Subsystems Which Process Both Matter-Energy and Information”, Behavioral Science, Volume 37, pp. 1–38.
  • Scharmer, C.O. (2009) Theory U, Leading from the Future as it Emerges, Berrett-Koehler, San Francisco, see particularly pp. 59-75.
  • Snowden, D.J, Boone, M.E. (2007) “A Leader’s Framework for Decision Making”, in: Harvard Business Review, November, Reprint, pp. 1-9.


4 responses to “Agile vs Traditional Project Management: A Game-based Experiment”

  1. […] THE GREATEST FIGHT IN PROJECT HISTORY (english article) – Prof. Dr. Frank Habermann […]

  2. […] THE GREATEST FIGHT IN PROJECT HISTORY (english article) – Prof. Dr. Frank Habermann […]

  3. Hi,
    I have quite a bit of experience with both methods in practice (IT) and have seen agile win every round.
    Therefore I am sceptical about some of the results presented above:
    Iterative improvement always winning in the first round is exactly what I expected. Further improving these results with an in depth discussion as also what I would expect, but:
    How big was the improvement achieved by an in depth phase after iterative phases? I expect a minor improvement here, as people are very unlikely to risk any new approach in their single shot.
    Cheers,
    Timon

    • Frank says:

      Timon,
      Thank you very much for sharing your experiences. Of course, it always depends on a number of parameters what people risk or don’t risk, e.g. the corporate culture, the individual levels of experience, skills, motivation, etc. In our experiment, we have isolated/eliminated most of these aspects. And what we have observed is, that teams indeed “risk” major changes even in a following in-depth phase. AND that these changes often lead to great/improved resolts.
      Cheers,
      Frank

Leave a Reply