ChatGPT Recommended a Grocery Store as an Air Conditioning Installer

We asked ChatGPT for an air conditioning installer in Manchester. It recommended Acme Fresh Market. That's a grocery store. It also suggested K2 Karaoke -- a karaoke bar -- and a company based in New Jersey. For a query about Manchester, UK.

This wasn't a one-off. We tested 15 trending service categories across multiple cities. Categories like heat pump installation, EV charger fitting, smart home setup, e-bike repair. Services that are growing fast but don't have decades of web presence behind them.

13% of the recommendations were completely wrong businesses. Not slightly off. Wrong industry, wrong country, wrong everything. ChatGPT made them up or pulled names from unrelated contexts and presented them with full confidence.

Important caveat: this was gpt-4o-mini, the smaller model. We ran the same tests on full gpt-4o and the hallucination rate dropped significantly. The larger model handled emerging categories much better.

But here's why the mini model matters more than the full model for local businesses. Most free-tier ChatGPT users get gpt-4o-mini for their everyday queries. The people casually asking "who can install a heat pump near me" -- the exact customers a local business wants to reach -- are talking to the model that recommends grocery stores as HVAC installers.

The hallucination problem is worst in emerging categories because there's less data for the model to work with. Established trades like plumbing and electrical have years of directory listings, review pages, and "best of" articles. ChatGPT can find real businesses through Bing and return sensible results. For newer services, the web is thinner. Less content means more guessing, and when ChatGPT guesses, it guesses confidently.

We also found that ChatGPT without web search enabled is even worse. We tested the same 15 queries with web search turned off. 10 out of 15 returned zero real businesses. The model invented plausible-sounding names -- "Eco Energy Solutions Manchester," "GreenHeat Installations" -- that don't exist. It didn't hedge or say "I'm not sure." It presented fictional businesses as recommendations.

With web search on, gpt-4o does a reasonable job for established categories. The problem concentrates in two areas: emerging services where web data is sparse, and the smaller model that most free users interact with.

For a business in one of these growing categories, the situation is strange. Your potential customers are asking AI for help finding you. The AI is responding with confidence. And what it's saying is wrong. You're not being outranked by a competitor. You're being replaced by a karaoke bar.

The first businesses to build a real web presence in these emerging categories will have a massive advantage. Right now they'd be competing against hallucinations, not real competitors. That window won't stay open forever.