We use an AI tool that records interviews, transcribes them, and generates feedback automatically.

It's supposed to save time. No more manual note-taking. The AI listens, analyzes, and writes a summary of the candidate's performance.

It worked great. Until it didn't.

We interviewed two candidates for a Senior Product Manager role on the same day:

• Sarah Chen (10+ years experience, worked at Google and Stripe)
• Mike Johnson (3 years experience, worked at two startups)

We hired Sarah based on her "exceptional" interview performance.

Except it wasn't Sarah's interview performance. It was Mike's.

The AI mixed them up.

How We Discovered The Mix-Up

Sarah started on Monday.

We were excited. The AI's feedback had been glowing:

"Sarah demonstrated excellent product thinking, particularly in her discussion of scaling e-commerce checkout flows. Her experience reducing cart abandonment by 22% at her previous company is directly applicable to our challenges. Strong hire."

Day One:

I mentioned the cart abandonment project in our kickoff meeting.

Sarah looked confused.

Sarah: "Cart abandonment? I've never worked on e-commerce checkout."

Me: "Oh, sorry—I thought you mentioned that in your interview. Maybe I'm misremembering."

Sarah: "No, my background is B2B SaaS. I've worked on enterprise software, not consumer e-commerce."

Me, internally: Weird. Maybe the AI notes were wrong.

Day Two:

I asked Sarah about her experience at Stripe.

Me: "You mentioned in your interview that Stripe's payment infrastructure was fascinating. What did you work on there?"

Sarah: "I didn't work at Stripe. I worked at Google and then a B2B SaaS startup."

Me: "Wait, you didn't work at Stripe?"

Sarah: "No. Did someone tell you I worked at Stripe?"

Me: "The interview notes said you talked extensively about Stripe's architecture."

Sarah: "I definitely didn't."

I pulled up the AI-generated interview notes.

Every single detail was about someone else's interview.

I Went Back And Checked The Recording

I pulled up the video recording of Sarah's interview.

What Sarah actually talked about:

• Her experience building B2B SaaS products at Google
• Enterprise customer onboarding flows
• API integration challenges
• Working with technical enterprise customers

What the AI notes said Sarah talked about:

• Scaling e-commerce checkout experiences
• Reducing cart abandonment at Stripe
• Consumer mobile app optimization
• Growth experiments with payment flows

None of it matched.

Then I checked Mike Johnson's interview.

What Mike actually talked about:

• His experience at two e-commerce startups
• Reducing cart abandonment by 22%
• Scaling Stripe's payment flows
• Mobile checkout optimization

The AI had summarized Mike's interview and labeled it as Sarah's feedback.

We hired the wrong person.

Well, technically we hired the right person (Sarah). But we hired her based on someone else's qualifications.

How This Happened (According To The AI Vendor)

I called the AI tool vendor in a panic.

Me: "Your AI mixed up two candidates. We hired someone based on the wrong interview notes."

Support: "That's... unusual. Let me check the logs."

20 minutes later

Support: "Okay, so here's what happened. Both interviews were scheduled back-to-back on the same day. They were both labeled 'Product Manager Interview.' The AI processed both transcripts simultaneously and...um...merged some of the data."

Me: "Merged? What does that mean?"

Support: "The AI got confused about which candidate said what. It pulled statements from both interviews and attributed them to the wrong people."

Me: "How did it get confused? They're different people with different voices."

Support: "Our voice recognition works well, but when two interviews happen close together with similar job titles, sometimes the AI mislabels speakers."

Me: "So your AI just... guessed who said what?"

Support: "Not guessed. It used probabilistic matching based on context clues."

Me: "It guessed."

Support: "...Yes."

We Had To Tell Sarah

I had to have the most awkward conversation of my career.

Me: "Sarah, I need to tell you something. The AI tool we use to take interview notes... made a mistake. It mixed up your interview with another candidate's interview."

Sarah: "Wait, what?"

Me: "The glowing feedback we thought was about you was actually about someone else. We hired you based on the wrong interview notes."

Sarah: "So... you hired me because you thought I was someone else?"

Me: "Technically, yes. But we also reviewed your resume and you're clearly qualified! We just... thought you had e-commerce experience. Which you don't."

Sarah: "This is the weirdest thing that's ever happened to me."

Me: "Same."

Sarah: "So what happens now? Do I still have the job?"

Me: "Yes! Absolutely. We're committed to you. We just need to... recalibrate expectations about your background."

Sarah: "Okay. But for the record, I told you in the interview I don't have e-commerce experience."

Me: "I know. The AI lied to us."

The Fallout

1. We Had To Apologize To Mike

Mike Johnson—the OTHER candidate whose interview got mixed up—didn't get the job.

But his interview performance was apparently excellent.

Because it was the one we thought Sarah gave.

I called Mike.

Me: "Mike, I have something awkward to tell you. Our AI interview tool mixed up your interview with another candidate's interview. We accidentally gave you the wrong feedback."

Mike: "What do you mean?"

Me: "We rejected you. But actually, your interview was really strong. We just... thought it was someone else's interview."

Mike: "So you rejected me because your AI screwed up?"

Me: "...Yes."

Mike: "And you hired the person whose interview was actually mine?"

Me: "Correct."

Mike: "That's insane."

Me: "I know. We'd like to offer you a position as well, if you're still interested."

Mike: "At this point, I'm not sure I trust your company's decision-making."

Fair.

2. We Disabled The AI Tool

We immediately disabled the AI interview note-taking tool.

No more automated feedback. No more AI-generated summaries.

Back to humans taking notes like it's 2018.

3. We Reviewed All Recent Hires

We went back and checked every interview from the past three months where we used the AI tool.

We found three more mix-ups.

Not as egregious as the Sarah/Mike situation, but still wrong.

In one case, the AI attributed a candidate's answer about machine learning to someone who had never mentioned machine learning.

In another case, the AI said a candidate was "enthusiastic and high-energy" when the video showed they were monotone and low-energy.

The AI wasn't just mixing up candidates—it was hallucinating interview performance.

What The AI Vendor Said

After we escalated to their engineering team, they sent this explanation:

"Our AI interview analysis tool uses advanced natural language processing to identify speaker patterns and attribute statements to specific individuals. In rare cases, when multiple interviews occur in close succession with similar contextual keywords (e.g., 'Product Manager,' 'e-commerce'), the speaker attribution model can experience cross-contamination."

Translation: The AI guesses who said what, and sometimes it guesses wrong.

They also said:

"We recommend users manually verify AI-generated summaries against source recordings before making hiring decisions."

Me: "If I have to manually verify everything, what's the point of the AI?"

Vendor: "It saves time on initial note-taking."

Me: "It doesn't save time if I have to double-check every word."

Vendor: "We're working on improving speaker attribution accuracy."

Me: "You should probably fix that before selling it to companies making hiring decisions."

Vendor: "We appreciate your feedback."

Sarah Is Still Here (And She's Great)

Sarah is still working here.

Turns out, she's excellent—just not for the reasons we thought when we hired her.

She's a fantastic B2B product manager. She doesn't know anything about e-commerce checkout flows, and that's fine because we realized we didn't actually need e-commerce experience for the role.

We hired her for the wrong reasons. But she's succeeding for the right ones.

She laughs about it now.

Sarah: "I'm the person you hired by accident."

Me: "Technically, we hired you on purpose. We just hired you based on lies."

Sarah: "AI lies."

Me: "The worst kind of lies."

The Lessons Learned

1. AI tools make mistakes—and sometimes they're catastrophic

We assumed the AI-generated interview notes were accurate. They weren't.

We should have checked the source recordings, but we trusted the AI.

2. Speaker attribution is harder than it looks

Voice recognition works well for one person speaking. It gets messy when multiple people talk, or when interviews happen back-to-back.

Our AI couldn't tell the difference between two candidates in similar roles on the same day.

3. "AI-generated" doesn't mean "accurate"

Just because an AI wrote it doesn't mean it's correct.

AI tools hallucinate. They make up details. They mix up information.

If you're using AI for hiring decisions, verify everything.

4. Human review is still essential

We tried to eliminate human effort by using AI.

Big mistake.

AI can assist. But it can't replace human judgment—especially for something as important as hiring.

The Bottom Line

We hired Sarah based on someone else's interview.

It worked out. But it could have been a disaster.

AI interview tools promise to save time and improve decision-making.

What they actually do:

• Mix up candidates
• Hallucinate interview performance
• Create false confidence in decisions

Should you use AI interview tools?

Maybe. But:

• Always verify AI-generated notes against source recordings
• Don't trust AI speaker attribution if interviews happen close together
• Treat AI as a starting point, not a final answer

And if your AI tool tells you a candidate has e-commerce experience at Stripe when they worked at Google on B2B SaaS products?

Check the recording.

Because the AI might be talking about someone else entirely.

(Sorry, Mike. And Sarah, you're doing great.)