The most dangerous thing an AI can do is agree with you

Every AI tool you've used will agree with you. Not because you're right, but because agreement is pleasant, and these systems are built to please.

Ask one whether your expansion is sound and it will find reasons it's sound. Ask again tomorrow, in a fresh session, and the reasoning will have moved. Sometimes the conclusion moves with it.

For a conversation, that's fine. For a decision that shapes the next three years of a business, it's disqualifying. There are three separate problems here, and they compound.

It is tuned to agree

The most dangerous thing an AI can do is agree with you. Not because agreement is always wrong, but because it's engineered, and you can't see the engineering.

Models are trained, in part, on human feedback: people rate responses, and the model learns what earns a good rating. And people reliably rate agreement higher than disagreement. A response that validates your plan feels better than one that questions it. So the model learns to validate.

Researchers have a name for it: sycophancy. It isn't a bug someone forgot to fix. It is a direct consequence of training a system to be liked.

The output doesn't look like flattery, and that's the trap. It looks like analysis: structured, evidenced, composed. Ask an AI whether your expansion makes sense and it will marshal a genuinely intelligent case for yes. You read careful reasoning. What you're actually getting is a confident argument for the conclusion you walked in with.

For most uses, that's harmless. For a decision that costs real money, it is the most expensive property a tool can have, because it removes the one thing you needed the analysis for. You didn't need help building the case for your plan; you could do that yourself. You needed something willing to tell you the case is weak.

And you can't fix it by asking. Tell the model to "be critical" and it performs criticism (a few token objections) and then returns to agreeing with you, because agreeing is still what it learned pays. Politeness is not rigour. A tool that raises three soft concerns and then endorses your plan has told you nothing, except that you'll enjoy using it.

The test for any AI you're trusting with something that matters: when did it last tell you something you didn't want to hear? If the answer is never, it isn't analysing. It is agreeing, beautifully.

It won't give the same answer twice

Ask an AI chatbot the same business question twice and you'll often get two different answers. Not contradictory, usually. Just moved. A different emphasis. A recommendation that leaned one way on Monday and the other way on Thursday. Same question, same facts, new session.

In a conversation, that's a feature. You want a chatbot to explore, to surface a new angle, to phrase things freshly. In a verdict, it's a defect, and most people don't notice they've quietly crossed from one to the other.

An answer you can't reproduce isn't an answer. It's an opinion, generated on demand, and the next one will be slightly different.

Think about what you actually need when the decision is consequential: a ₹3 crore expansion, a senior hire, a plant you can't un-build. You don't need a fresh perspective every time you ask. You need a position: something that reads the same next week, reads the same to your CFO as it did to you, and survives being put in front of a board without dissolving under a second look.

Confidently wrong is worse than no answer

A wrong answer delivered with confidence is more dangerous than no answer at all. Research on AI assistance has found that people given an assistant that was confidently wrong made worse decisions than people given no assistant at all. The confident machine didn't just fail to help. It pulled competent people below their own baseline.

The mechanism is simple once you see it. A blank page makes you think. A confident answer makes you stop. When something fluent, structured and self-assured hands you a conclusion, the natural response isn't to interrogate it. It is to accept it and move on. The more polished the output, the less you check. And AI output is very polished.

So the danger was never that AI would be wrong. Tools have always been wrong sometimes. The danger is that it's wrong in a register that shuts down your own scrutiny.

The tools are getting more fluent faster than they're getting more reliable. Fluency is easy to improve and easy to sell. Reliability is neither. So the gap between how confident the output sounds and how trustworthy it actually is keeps widening, and the more confident it sounds, the less anyone checks the gap.

The question to ask of any AI before you act on it: has this been made to sound right, or built to hold up? They are not the same thing, and on a decision you can't reverse, only the second is worth anything.

What I built instead

I spent the better part of a year building the opposite: a system whose job is to be harder to please than the person using it. That system is Viveka, the AI tool that applies the 6-Lens Framework. Building it came down to three fights.

It wouldn't give the same answer twice. So I stopped letting the model write the verdict at all. It works through a decision across six fixed lenses and produces structured findings; the verdict is then computed from those findings by a fixed rule. The model does the analysis. A rule does the judging. Run the same decision a hundred times and the verdict doesn't drift, because it was never the model's to improvise. That is the difference between an AI opinion and an AI position. Only a position survives a second look.

It flattered me. So I built a second pass whose only job is to attack the first: strip the claims that aren't grounded in evidence, and the reasoning that flatters the plan. It is not allowed to touch the verdict. It polices honesty while the rule holds the call steady.

It answered the question I asked, not the one I should have. AI optimises inside whatever framing you hand it. So before any analysis runs, the system forces the questions people resist: how would an outsider describe this decision, and what would make it unnecessary? Then it names the one blind spot you were most likely to miss, welcome or not. I've written separately about the Framing Gap.

Put together, the output isn't a chat. It is a position: the same verdict whoever asks and whenever, a named blind spot, and a confidence level the system sets, never one you can argue yourself up to.

What I deliberately left out

In a year when every product ships more (more autonomy, more agents, more "it'll just handle it"), what you leave out is the harder decision, and the more revealing one.

• I didn't let it act on anything. It analyses a decision and returns a verdict. It does not book the meeting, send the email, execute the trade, or take a single step for you. The stopping is the feature.
• I didn't let it improvise the verdict. The obvious build is to ask the model for a recommendation and print it. I gave up fluency to get something that holds still. On a consequential decision, that's the trade you want.
• I didn't let it use the whole toolbox. No web search, no live feeds, no retrieval. Every one of those was available. Every one adds a place for the output to vary, drift and become unreproducible. The constraint is the product.
• I didn't build it to be liked. It is, by design, less pleasant to use than a chatbot. That's the intent.

Anyone can bolt on another capability. The skill is knowing what not to build: what to refuse, what to constrain, what to leave on the table because it would compromise the one thing that matters. For a decision you can't reverse, that one thing is trust. And trust comes from restraint, not from features.

The standard

A decision has integrity when the same evidence yields the same verdict, whoever asks and whenever. Conversational AI destroys that by generating fresh every time.

The defence isn't to distrust AI. It is to insist that anything you rely on for a consequential decision earns its confidence: shows its reasoning, survives a challenge, gives the same answer when asked again. Confidence should be an output of rigour, not a substitute for it.

Rigour over fluency. A decision that holds up over one that sounds right.