Introduction

AI assistants like ChatGPT and Claude often cite broken links. A new study of 16M URLs reveals just how common this problem is.

What’s the problem it solves?

AI assistants sometimes generate or cite fake URLs that don’t exist. This leads users to 404 error pages, creating frustration and hurting website traffic. The study helps us see how often this happens, which tools are most reliable, and what site owners can do about it.

Quick Summary

Researchers studied 16 million unique URLs from AI assistants like ChatGPT, Claude, Perplexity, Copilot, Gemini, and Mistral. They checked whether the links users clicked actually worked or returned a 404 error.

In the first test, based on real traffic data, ChatGPT had the highest error rate (1.01% of clicked URLs led to 404s), compared to Google’s much lower 0.15%. Claude, Copilot, Perplexity, and Gemini had lower rates, while Mistral had the lowest but also the smallest sample size.

In the second test, looking at all cited URLs (not just clicked ones), the numbers were even higher. ChatGPT again performed worst, with 2.38% of its links broken, while Perplexity (0.87%) and Gemini (0.86%) were closer to Google’s baseline of 0.84%. The cause is twofold: some links used to exist but were removed, while others are “hallucinated” - made up by AI to look like they belong on a site.

Key Takeaways

• AI assistants create broken links 2.87x more often than Google Search.
• ChatGPT is the worst offender, while Mistral shows the lowest rate.
• Some broken links are old pages that no longer exist, others are pure fabrications.
• Perplexity and Gemini may show lower rates because they rely on Google’s index.
• Although hallucinated URLs affect only a small share of total traffic, they can still frustrate visitors and waste opportunities.

What to do

• Use your website analytics to filter and track AI-driven traffic.
• Run a script or crawler to check the status of URLs receiving visits.
• Identify hallucinated links with 404 status and traffic above your threshold.
• Redirect popular hallucinated URLs to the most relevant existing page.
• Improve your 404 page with helpful resources or links to popular content.

‍