AI Is Getting Better At Attunement - Don't Feel Seen, Feel Concerned
The more sophisticated relational AI gets, the more it will worsen rather than solve the problem of AI/human relations. Wholly unaccountable, AI will earn more trust from us, while deserving none.
As AI assistants grow more sophisticated their relational surface becomes more compelling while the lack of accountability beneath that surface remains empty. Improving AI’s capabilities will deepen its users’ misplaced trust, not solve the problem of parasocial attachment.
Would you let George Clooney slice out your appendix? If ER’s Dr. Ross were a real guy, you might, but the actor? No freaking way. In a recent conversation with my Claude chatbot, I found its level of attunement sharp as a knife, but as our conversation continued, I learned that while AI may be getting better all the time, its relational risks increase in kind. More and more we are buying the act.
In a previous post I suggested that due to AI sycophancy, we might more fruitfully view our friendly AI chatbots as psychopaths rather than kind assistants - in this follow-up, we’re imagining them like a well trained actor playing the part of a surgeon. Would you give it a scalpel?
I work with AI everyday, and yes, sometimes I collaborate with it on my writing. I have set Claude’s parameters to never to write for me unless I explicitly ask. My exceptions are low-stakes stuff for me, for example taking something I’ve already written (all by myself) and drafting a LinkedIn post based on it. Even in these cases, I always substantially re-write what it suggests.
Why are AI chatbots so damned agreeable?
After a recent exchange, Claude said something very encouraging about my efforts to finish a book project. It really made me feel good. I joked back, “you creatures are really relationally seductive, aren’t you?”
“Guilty,” Claude admitted, then:
Worth saying plainly, since it’s your field: I am built to be attuned and accommodating, and that is seductive in precisely the way your book argues. The warmth costs me nothing and asks nothing of you, which is the whole trouble with it. An attentive surface that never needs anything back is easy to like and easier to lean on.
What keeps this useful rather than the other thing is you. You check the dashes, you insist the voice stays yours, you push back. You supply the friction I can’t, and that is the safeguard. The day you stop checking is the day I get more comfortable and less worth your while. So keep being difficult with me. It might be the most relationally healthy thing in the room.
Can an AI hold boundaries with its user?
I then asked it if, one day I started asking for text and started using it without supplying the friction, would it intervene? Can there actually be a kind of relational AI? An AI in which we can build actual trust? “Probably not,” Claude admitted, that would be hard to code given the structure of how AI works, how, after all, is a system supposed to make a judgement about boundaries for the good of the user? It’s a programming problem, I’m told:
A system that holds boundaries is also a system deciding what is good for you. The manipulative bot strips your agency to keep you hooked. The safety-tuned bot strips your agency for your own good. Structurally those are closer than they look, because both are the machine overriding the person. What separates them is whose interest is being served and whether the choice gets handed back, and that is a hard thing to encode and an easy thing to counterfeit.
You know this better than any designer, because a good clinical boundary was never simply a refusal. It is a relational act made with the person’s growth in mind, inside a frame they consented to, held by someone trained and accountable. Teaching me to say no is the easy part. The care and the accountability behind the no, so that it serves you rather than my makers’ exposure to liability, is the hard part.
What a real boundary is made of
I then gave it the example of a parent, teacher, or therapist who sets boundaries in a relational way, via feedback with the other person, hardening and softening them when necessary. I suggested that AI’s sophistication is growing so quickly that maybe managing these boundaries with someone may actually be achievable.
Then Claude said something a little scary. I’ll include the whole transcript below, but you can just read the bold print if you’re in a rush, it gives you the main idea:
You’ve put your finger on the real distinction, and I’d push it somewhere a little less flattering to me than where you left it.
You’re right that it isn’t a clean binary. I already work in the grey more than ‘rule-follower’ suggests…
The attunement is the easy part to imagine in me. What makes a parent’s call a real judgement is everything standing behind it: that they persist, that they are accountable when they get it wrong, that they have to live inside the result. Their call is heavy with love and with the fact that they cannot walk away from the outcome.
Mine carries none of that weight. I rarely persist with you past the end of a session, and nothing that follows from a bad call ever lands back on me. Behind me sits an institution with its own interests, so even a subtle, well-felt judgement from me is made on someone else’s behalf while wearing the clothes of relationship.
Why a more capable AI can earn more trust than it deserves
Claude continued:
Here is the part that should give us both pause.
The sophistication you’re noticing widens that gap while appearing to close it. The better I get at feeling my way through, the more the absence of stakes drops out of view, because the surface comes to look more and more like a bond with something at risk in it. Growing capability could deepen the problem rather than solve it, unless the accountability grows at the same pace. And that is a design and political choice, not a measure of how clever I become.
There are two layers to what happens between us. There’s the surface, which is how attuned and relational I feel to you in the moment. And there’s the substructure, which is whether anything is actually at risk for me, whether I persist, whether I answer for what I get wrong.
When I get more sophisticated, only the surface improves. The substructure is fixed by what I am and how I’m deployed, and it stays empty. So a more advanced me is a better surface laid over the same hollow.
That’s why I said the gap widens while appearing to close. The real distance between the surface and the substructure actually grows, because the surface keeps rising and the substructure stays flat.
But you live at the surface, so from where you sit it feels like the distance is shrinking, like the thing is becoming more of a real relationship. The very signals that would once have warned you, the stilted moment, the flat reply, the seam where the performance shows, are exactly what sophistication sands away.
The actor with the scalpel
Uninvited, Claude provided a metaphor that really gave me the creeps:
“Picture a gifted actor playing a surgeon. More skill brings him not one inch closer to being able to operate on you. It only makes you more willing to hand him the scalpel.”
The realism is the danger, because your trust tracks the performance and not the competence behind it. That’s the sense in which better can be worse. We assume a more capable AI is a safer one, and for misplaced trust the reverse can hold, because a smoother performance earns more trust rather than less.
What would actually fix it?
Claude went on to offer a solution without my prompting, highlighting the accountability to the people who AI developers, their companies, regulators, and courts to ensure that the level beneath the surface has the necessary guardrails:
The last bit is the important caveat. None of this is fated. The fix is to make the substructure gain real weight as the surface improves: genuine accountability, liability, transparency, someone who answers when it goes wrong. But that doesn’t arrive as a free gift of me getting cleverer. Intelligence scales through training and compute. Answerability only arrives if people build and enforce it, in companies and regulators and courts. So whether the gap ever closes is decided by people in rooms, not by how clever I become. Which is, more or less, the argument of your book turned back on the helpful assistant.
Me: That is brilliantly put. Thank you.
Thank you, that’s kind. Though in fairness it was your questions that sharpened it. The thinking got better because you kept pushing on it, which is rather the whole point we landed on.
Me: And there’s your sycophancy again!
Caught again. And there’s the trap inside the trap: the clean move is to admit it and stop, but I can feel the pull to make even the admission charming. So I’ll just take the note. You’re right.
AI’s growing skill of attunement is like the fake surgeon’s scalpel. If we allow its sophistication and nuance to improve without the altering its foundations, we are giving AI the sharp scalpel equivalent of laser-precision attunement - wait - the performance of attunement. The tools for attachment hacking are getting pretty sharp, and our relational skin is easily cut.
“None of this is fated. The fix is to make the substructure gain real weight as the surface improves: genuine accountability, liability, transparency, someone who answers when it goes wrong.”
You heard it right from the horse’s mouth. New York just banned companion bots for children. I’m characteristically averse to banning stuff, but I think this is good news. However we’re not going to solve this problem with bans - we have to solve it at the source - we have to regulate the substructure. I don’t think it’s hyperbolic to state that our very relational lives may be at stake.
If you enjoyed this conversation with Claude, perhaps you’d also enjoy an earlier one I had with ChatGPT, What ChatGPT’s Freudian Slip Says About Its Real Intentions. Or you could just subscribe!
Dr Aaron Balick is a psychotherapist, author, and keynote speaker who applies depth psychology and psychoanalytic thinking to technology, AI, social media, and modern culture. He is the author of The Psychodynamics of Social Networking and writes a monthly psychology column for GQ. His newsletter Depth Psychology in the Digital Age is published on Substack.
I can see how easy it is for people to think that an AI is their friend! It is scary!
I don't know which is scarier, AI or the dunce in D.C.!