Harvard Business School Partners with BCG on AI Productivity Study

A recent Harvard Business School study found that artificial intelligence tools increase worker productivity and accuracy on certain tasks but have a countervailing effect on other similarly-difficult tasks outside a certain “technological frontier.”

In a recent working paper called “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality,” HBS and Boston Consulting Group conducted a joint study to investigate the practicality of AI tools for realistic knowledge-intensive applications in consulting.

The study was conducted by HBS professors Karim R. Lakhani and Edward McFowland III, HBS postdoctoral researcher Fabrizio Dell’Acqua, and researchers from the University of Pennsylvania’s Wharton School, MIT Sloan School of Management, Warwick Business School, and BCG.

The researchers found that AI capabilities currently cover an uneven set of skills they term a “jagged technological frontier”: outside of this frontier, AI output is not accurate or even might worsen human performance.

In the study, 758 BCG consultants were given 18 realistic consulting tasks within this frontier to track changes in worker productivity and accuracy. The study found that, compared to workers without AI access, those who used GPT-4 completed on average 12.2 percent more tasks, 25.1 percent quicker. Additionally, 40 percent of the trial group produced higher quality results.

At the same time, consultants using AI for tasks considered outside of the frontier were 19 percent less likely to produce the correct solutions compared to those without AI.

As a result of the study, researchers distinguished between two different patterns of AI integration into these tasks: “centaurs” and “cyborgs.”

“A centaur is someone that clearly defines what the human does and what the AI does, and uses a human for whatever humans are best at, and the AI for whatever its best at,” Dell’Acqua said.

But most consultants were observed to be a “cyborg” — where consultants have a “constant interaction with the AI.”

“These are both types of potentially successful collaborations,” Dell’Acqua said.

In one task requiring participants to analyze retail strategy from interview notes and financial data in a spreadsheet, an evaluation of correctness revealed that there was a “pretty big fall” in performance in the group with access to AI — a finding that Lakhani attributed to user error, not technological shortcomings.

“There’s no user guide and we don't really know what the frontier is,” Lakhani said, explaining that the frontier is “jagged,” or undefined, which leads to user error.

“People use it the wrong way. People use it as an information search tool like Google,” Lakhani said. “This is not Google.”

McFowland highlighted the tradeoff between accuracy and speed when AI is deployed within organizations with cases “outside the frontier.”

“We get productivity boosts. Things, they got done faster, but getting done faster to the wrong answer in many cases is not ideal — or at least not preferable — to getting the right answers more slowly,” he said.

The study concluded that understanding the “shape and position of the frontier” are crucial to optimize the impact of AI on worker productivity.

Though that frontier will move forward as AI progresses, McFowland said we are still far off from being able to “encapsulate all of human cognition and capability.”

“There’s always going to exist things outside the frontier with capabilities and often the challenging — or maybe pernicious — part is that we don’t know what those things are,” he said.

While Lakhani said “it’s going to unlock a ton more potential for humans, for all of us, if you know how to use it,” McFowland remained hesitant about calling AI a “revolution.”

“Technology has come along and changed how we do things before, over and over again,” he said. “But in hindsight, we recognize that they didn’t — they changed our world — but they weren't as life-altering as we thought they might have been.”

—Staff writer Tiffani A. Mezitis can be reached at tiffani.mezitis@thecrimson.com.

—Staff writer Camilla J. Martinez can be reached at camilla.martinez@thecrimson.com. Follow her on X @camillajinm.