We keep calling it lying. Wrong word, and the wrong word makes you check for the wrong thing.
It's not lying to you, because lying involves intent.
Lying means I know the truth and I choose to tell you a falsehood. The model has no idea it's wrong. It thinks it's handing you the truth every time, with the same certainty whether it's right or making something up.
Paul's version in the clip: it's a really excited five-year-old. Completely sure, occasionally completely off.
So confidence tells you nothing here. You need a check that doesn't rely on how the answer feels.
What's your current check when an answer really matters?