“This is the autopilot problem: How do you know when the computer is giving you a wrong answer? Ideally, you know because you have enough experience to know what the right answer looks like, but this assumes you are paying attention to the answer you’ve been given.”
https://public.milcyber.org/activities/magazine/articles/2023/draeger-the-autopilot-problem