Economists Show AI Bringing Positive Impact to Workplaces

Stanford-based economist Erik Brynjolfsson is surprisingly cheery about AI’s ultimate impact. “I wouldn’t be surprised 50 years from now, people looked back and say, wow, that was a really seminal set of inventions that happened in the early 2020s…” he told CBS in January. “I think we’re going to have potentially the best decade of the flourishing of creativity that we’ve ever had, because a whole bunch of people, lots more people than before, are going to be able to contribute to our collective art and science.”

But is there evidence to support this optimism? Last month Brynjolfsson teamed up with MIT-based economists Danielle Li and Lindsey R. Raymond, for a new paper to explore the question. Titled “Generative AI at Work,” it begins by noting the lack of other studies on real-world economic effects of AI, calling their work “to our knowledge, the first study of the impact of generative AI when deployed at scale in the workplace” — and over a longer period of time.

So what did they find? The three economists conclude AI “increases worker productivity, improves customer sentiment, and is associated with reductions in employee turnover.”

And the specifics were even more intriguing…

Cyborg Customer Service

Their study focused on chat-based customer service. Among businesses using AI, 22% were using it in their customer service centers, according to a McKinsey Analytics study cited by the researchers.

Though the AI tool isn’t specified, their paper specifies its functionality: it provides support agents with real-time suggestions for responses, and also prompts them with links to internal technical documentation. And of course, it tracks entire conversations for context.

Importantly, the system doesn’t make suggestions at all if it doesn’t have enough training data — which “occurs in a large minority of cases” — and human agents always have the choice to disregard all suggestions.

The conversations happened between November of 2020 and February of 2021, using a tool built on large-language models from OpenAI’s GPT family, “with additional ML algorithms specifically fine-tuned to focus on customer service interactions.”

Interestingly, its training data included conversations with both “successful” and “unsuccessful” outcomes (as well as conversations of varying length) — and indicates whether the data came from one of the firm’s top-ranked agents.

“The AI firm then uses these data to look for conversational patterns that are most predictive of call resolution and handle time,” the paper notes, adding it prioritizes “responses that express empathy, surface appropriate technical documentation, and limit unprofessional language.”

The randomly-assigned conversations “are relatively lengthy, averaging 40 minutes,” according to the report, “with much of the conversation spent trying to diagnose the underlying technical problem…”

Past automation saw a rarefied handful of engineers carefully mapping tasks onto algorithms — versus this cruder brute-force method of feeding masses of training data into an ML system until an algorithm emerges that everyone can follow.

“Equipped with this new AI system, the company’s customer support representatives are now basically part human, part intelligent machine,” quipped NPR’s Planet Money blog. “Cyborg customer reps, if you will.”

And after 3,007,501 conversations (performed by 5,179 agents) the results came in. Where before the support agents averaged 43 minutes for each call, that average dropped to just 35 minutes when AI suggestions were made available (over 1,180,446 calls).

But that’s just the beginning…

• The AI suggestions also increased the average number of resolved issues by 13.8% (per hour) — fueled partly by an increase in the number of simultaneous chats that an agent can handle. The researchers write that their evidence suggests that “AI enables agents to both speed up chats and to multitask more effectively.”
• The introduction of AI tools apparently also improved the retention of employees — in every skill group. And in a possibly-related development, the tools also improved the way customers treated their support agents, indicating, among things, in fewer requests for a manager to intervene. The researchers see changes in the very experience of work, concluding that “generative AI working alongside humans can have a significant positive impact.” As it stands, the annual turnover rates for customer service agents can reach 60%, according to one study they cite — leading to costly training for replacement workers and wider variations in experience levels and productivity.
• AI’s impact varied depending on a worker’s experience and skill level, “with the greatest impact on novice and low-skilled workers, and minimal impact on experienced and highly skilled workers.” In fact, for the highest-skilled workers, the AI help didn’t lower the average time spent on calls at all (though these agents may have been able to handle more calls simultaneously). But even more interesting, the highest-skilled workers saw “small but statistically significant decreases in resolution rates and customer satisfaction,” the researchers note, positing that AI assistance “may distract the highest-skilled workers, who are already doing their jobs effectively.” (While meanwhile, it’s the less-skilled agents who “consistently see the largest gains.”)

At the same time, there’s evidence that AI “disseminates” the “potentially tacit knowledge of more able workers.” That is, the benefits of experience get passed along to newer workers. The researchers provide specific benchmarks. Among agents given access to the AI, those agents with two months of tenure “perform just as well as untreated agents with over six months of tenure.”And the workers with AI assistance also seemed to get better faster.

Disseminating Productivity

Their paper cites a concept known as Polanyi’s paradox — that much of what we know how to do is hard to articulate into rules — and is considered a potential roadblock for full automation.

But AI appears to have the ability to acquire even those unexplained skills. Their paper argues the systems are “capturing and disseminating the patterns of behavior that characterize the most productive agents,” and the researchers saw gains in every measure of productivity — including speed, success rate, and customer satisfaction.

“[W]hat this system did was it took people with just two months of experience and had them performing at the level of people with six months of experience,” Brynjolfsson told Planet Money. “So it got them up the learning curve a lot faster — and that led to very positive benefits for the company.”

The report goes so far as to ask whether top-performing workers should be paid more — since their abilities are now propagated throughout the workforce.

But more importantly, the researchers note specifically that this is “in contrast to studies of prior waves of computerization.” Past automation saw a rarefied handful of engineers carefully mapping tasks onto algorithms — versus this cruder brute-force method of feeding masses of training data into an ML system until an algorithm emerges that everyone can follow.

Planet Money even asked Brynjolfsson if AI “could also reduce inequality by bringing the top and middle down, essentially de-skilling a whole range of occupations, making them easier for anyone to do and thus lowering their wage premium.” They report that Brynjolfsson “seemed a bit skeptical of this” — but they also see another important benefit. “It suggests that AI could benefit those who were left behind in the previous technological era.”

The paper clarifies at one point that their research wasn’t capturing the “wage effects” of AI. And it’s certainly true that AI may grow the economy, Brynjolfsson told Planet Money, using the classic metaphor of a bigger pie. But then he issued a warning. “It’s very clear that it’s not automatic that the bigger pie is evenly shared by everybody… We have to put in place policies, whether it’s in tax policy or the strategy of companies like this one, which make sure the gains are more widely shared.”

The paper’s conclusion concedes that “the effects we find may not generalize across all firms and production processes.”

Just for example, in fast-changing environments, it may be harder to train systems using historical data. But then it also acknowledges many other questions that it’s also leaving unexplored. Will customer service agents be reassigned to “more complex customer responsibilities, increasing aggregate demand”? Will customers prefer and even demand AI-enhanced support? Will AI systems uncover “patterns and insights” that change how workers are managed or how knowledge is shared?

And will wages go up or down?

There’s also the emerging issue of whether AI is passing off someone else’s skills as its own. “[O]ur findings raise questions about whether and how workers should be compensated for the data that they provide to AI systems,” the researchers write. “High-skill workers, in particular, play an important role in model development but see smaller direct benefits in terms of improving their own productivity.

“Given the early stage of generative AI, these and other questions deserve further scrutiny.”