The speed paradox
Artificial intelligence can drastically reduce the time it takes to complete a specific task. For example, if writing a report used to take eight hours and now takes two, it appears to be a 75% improvement in speed. However, if the report still requires three days for review and a week for approval before action, the overall process delay remains the same. Traditional evaluation metrics often focus on task-level speed without considering how AI affects the broader workflow. This narrow view can lead to a misunderstanding of AI's true contribution.
To fully benefit from AI, it is necessary to assess and enhance the output it produces. However, when organizations emphasize faster outcomes, they may skip this crucial step. Accepting AI results without careful review leads to a focus on quantity over quality. The speed metric, designed to measure progress, ends up undermining the value that AI could provide through thorough iteration and refinement.
Cost vs. creation
Viewing AI success solely through cost reduction sets a restrictive limit. For instance, if a content team’s annual budget is $1 million, the best-case savings scenario is also $1 million. But AI's real strength lies in unlocking new capabilities that were previously unfeasible. This includes performing complex analyses that had no time or resources allocated to them or creating personalized experiences that no team could manage before. These opportunities disappear when success is measured only in terms of savings.
When AI's effectiveness is evaluated by how many employees it can replace, knowledge workers may respond by engaging with the tool superficially. They use it just enough to avoid being replaced but keep their expertise private. This creates a harmful cycle where the metric itself discourages the deep interaction AI needs to thrive. Workers might withhold their insights and corrections, which are essential for AI to improve and adapt.
Adoption's empty promise
Having a thousand employees use AI to shorten their emails might seem like a sign of strong adoption. However, research conducted by experts at several central banks revealed that most senior executives reported little to no business improvement from AI despite its widespread use. High adoption numbers can be misleading. While leaders might highlight the high usage, they often admit it hasn’t led to real outcomes or significant changes in their organizations.
Adoption metrics often emphasize activity over impact. Just because many people use AI does not necessarily mean it is transforming business results. Many companies report high AI usage but claim it hasn’t changed anything. These numbers can be deceptive, showing engagement but not meaningful progress. The real question is not just how many people are using AI but how effectively it is changing the nature of their work and the organization's outcomes.
The fundamental challenge lies in the fact that traditional metrics for evaluating AI do not align well with its unique characteristics. Unlike earlier software, AI's value is not fixed from the start. Instead, it adapts based on how it is used. This makes traditional measures such as speed, cost savings, and adoption inadequate for capturing AI's true impact. New evaluation strategies are needed to fully understand and harness AI's potential.
AI was initially adopted as a tool for personal productivity. Users could type a question into a tab and receive a helpful response. No one defined its limits in advance because its potential is fluid and dependent on the user's input. Traditional metrics, designed for fixed capabilities, struggle to capture AI's evolving nature. This mismatch between AI and traditional metrics leads to several issues.
Speed metrics often focus on improving individual tasks, but business delays may stem from other areas of the workflow. Cost metrics limit AI to being a tool for savings, ignoring the new opportunities it can create. Adoption numbers may look impressive, but they do not necessarily translate into real business results. Each of these traditional measures fails to capture the dynamic and transformative nature of AI.
The key takeaway is that organizations must develop new metrics to evaluate AI's impact effectively. Traditional measures like speed, cost savings, and adoption were suitable for previous technologies but are not sufficient for AI. Companies need to look beyond these metrics to understand how AI is genuinely transforming their operations and delivering value.