Receiving a suspicious text at work that sounds like it might be from your bank can be tricky to assess. It seems real, but there's a chance it could be generated by an AI, making it hard to detect. This scenario played out in a recent experiment at Brigham Young University, where 25 participants were presented with a set of messages designed to resemble phishing attempts. Half were created by GPT-4, and the other half by students in a deception class. The participants struggled to distinguish between the two, performing no better than if they had flipped a coin for each guess.
AI-generated messages rival human efforts
Each person in the study completed a survey about their job, hobbies, and recent social media activity. These details were used to generate six personalized messages from GPT-4. Students also crafted messages, but they had the advantage of working under the guidance of professors who reviewed their submissions and discarded about a third for being incomplete or unhelpful. When participants sorted through all 12 messages, their ability to correctly identify AI-generated ones was nearly random. The lack of a clear pattern shows how closely AI can mimic human output.
While the AI messages slightly outperformed the human ones, the difference wasn't enough to be statistically significant. AI messages were judged as likely to prompt a click 28% of the time, compared to 21.3% for the human-written ones. The margin was small, with a confidence interval ranging from a 2.9-point advantage for the students to a 16.3-point lead for the AI. With only 25 participants, the sample size is too small to draw firm conclusions. However, the study highlights the efficiency of AI: a simple prompt template processed by GPT-4 can produce convincing messages quickly and cost-effectively.
Work-related phishing attempts pose a greater risk
The study categorized the messages into three types: those based on a participant's job, hobbies, and social media posts. Messages tied to work were the most dangerous. They reached the 'would click' threshold 38% of the time, compared to 19% for hobbies and 17% for social media. This is the most statistically robust finding from the test, underscoring the importance of work-related context in phishing messages. Attackers are likely to exploit this, as messages grounded in a person's professional life feel more credible.
Personalization played a key role, but it could also be a double-edged sword. When messages included details that were off—such as incorrect names of coworkers—participants became suspicious. For instance, one person noticed that a message mentioned a coworker named Mike, but there was no such individual in their workplace. Another found a name that didn't match anyone in their department. These errors can serve as a red flag, but they only work if the attacker attempts to include specific, relevant details in the first place.
People's theories to identify AI fall short
After sorting the messages, participants were asked to guess which ones were AI-generated. Overall, they correctly identified 52% of the messages, just slightly above chance. Many had theories about how to spot an AI message. Some believed AI would use overly formal language, while others thought it would be too generic. Perfect grammar was seen as a sign of a machine, and typos were assumed to be human. One participant even focused on punctuation, noting the excessive use of exclamation points. But these assumptions were misleading.
Only 12 out of 25 participants admitted to relying on instinct rather than logical reasoning. Emojis became one of the few reliable indicators, with 66% of AI messages containing them compared to just 2% of human-written messages. However, only five participants noticed this pattern. Of those, two thought emojis indicated AI, while the other three assumed they were used by people. This shows how even obvious clues can be overlooked if they don’t fit a person's expectations.
Machines can detect what humans miss
The researchers used machine learning to analyze all 300 messages and look for hidden patterns. While humans struggled to identify AI-generated content, the software was more successful. This suggests that while AI messages can closely resemble human communication, there are still detectable differences that most people haven’t learned to recognize. The study highlights a growing challenge in cybersecurity: humans may need new tools or training to keep up with the evolving threat landscape.
The results of this experiment reveal the growing challenge of phishing attacks in the age of AI. As these tools become more advanced, traditional signs of deception—like typos or awkward phrasing—may no longer be reliable. Work-related messages, in particular, may be the most dangerous. People must look for subtle patterns, such as the overuse of emojis or incorrect personal details, to avoid falling for AI-generated scams. For now, the best line of defense may be to approach every suspicious message with skepticism.