My AI writing assistant totally hallucinated a fake business partner last Tuesday and I almost sent the email

So I've been using this AI tool (won't name it but it's one of the big ones) to help draft client emails for my little consulting gig. Last Tuesday I asked it to write a follow-up to a client about a project deadline. It generated this perfect sounding email that referenced "our joint venture with Sarah Chen at InnovateCorp" and thanked her for her insights. Problem is, Sarah Chen doesn't exist and InnovateCorp is made up. I caught it right before hitting send because the client would have thought I was nuts. Now I'm paranoid and reading every single output like a hawk. Has anyone else had an AI tool just invent whole people or companies out of thin air?

3 comments

3 Comments

karen_west5924d ago

I've had this exact thing happen with a client named Tom Richardson who turned out to be a complete fabrication. The tool basically pieced together a name from one email thread and a company from another, then created a whole backstory. What works for me now is a quick copy-paste into a search engine for any names or company references before I send. Also started keeping a running list of real contacts and clients in a separate document that I check against the AI output. It's a pain but beats having to explain to a client why you're referencing imaginary people in their industry.

butler.finley24d ago

Gotta disagree a bit here. I think the real issue isn't that the AI makes stuff up, it's that we rely on it for facts instead of ideas. The Tom Richardson thing sounds like a pain, but that's what happens when you ask it to remember specifics about people. The tool is great for rough drafts, brainstorming, or rewriting sentences you already have. It's not a database of real clients. I use it to bounce ideas around or fix my own bad grammar, not to keep track of who I emailed last week. That's what my inbox is for. So instead of fighting the AI to be a Rolodex, I just use it for what it's actually good at and leave the fact checking to my own brain.

willow11424d ago

Totally feel your pain, that Tom Richardson thing sounds like a nightmare.