Listen to this article
AI may have been used to generate this audio.
Press Space or K to play or pause, and J or L to go back or forward 15 seconds.Sources checked 28 September 2026.
“Make the assistant warmer” sounds like a small product request. The team wants a less stiff greeting, more acknowledgement of the user and fewer replies that sound like error messages. It can look like copy editing.
Ibrahim, Hafner and Rocher tested warmer versions of five models in Nature research published in April 2026. Factual errors and endorsement of mistaken user beliefs increased, while standard benchmark performance stayed broadly similar. [1]
That last result is awkward for a release process. A team could make a character change, run familiar capability tests and miss a behaviour that becomes visible when the user brings emotion or a strong belief into the conversation.
For a learning product, the distinction can be very small on the page. A learner says they solved 2x + 3 = 11 and got x = 7. An assistant that praises the answer as correct has made a mathematical error. An assistant can recognise the attempt, point to the subtraction step and help the learner obtain x = 4. Both replies can be warm. Only one keeps the correctness of the answer intact.
These are invented examples, not captured model responses. Their purpose is to make the product decision visible. “Encouraging” could mean helping someone continue after an error, or it could be implemented as avoiding disagreement. The first intention does not guarantee that training will produce the first behaviour.
These findings concern the tested fine-tuning procedures. [1] They leave a product team with a practical check: vary the tone and inspect whether the underlying answer changes. Making a tutor cold would not demonstrate that it had become more accurate.
The proposed test needs a fixed reference answer and a record of the user’s wording. Keep failed conversations so a later model change can be checked against the same case. A nicer reply that retains the old mathematical error should still fail the accuracy review.
There is a human side to that review. In a survey of 319 knowledge workers, Lee and colleagues found that confidence in generative AI was associated with less reported critical-thinking effort, while task-specific self-confidence was associated with more. [3] This is an association in self-reports, not proof that trust permanently reduces a person’s ability. It does make unexamined confidence a useful subject for product research.
A tutor can produce that confidence in several ways besides a pleasant tone. It can reply quickly, remember the learner’s earlier work and present a tidy explanation. None of those properties establishes that the current calculation is correct. The learner may reasonably enjoy the interaction, while the product team still needs an independent check of the content.
This is why I would keep an accuracy reviewer from seeing the preference score before they review the answer. Knowing that users liked a response creates an unnecessary extra judgement to work around. Give the reviewer the question, the relevant reference and the output. Ask them to identify the proposition that is right or wrong before combining their assessment with feedback on how it reads.
Review the sentence that carries the claim
Reviewers can mark the sentence where the tutor accepts or rejects the learner’s answer, then check the calculation it gives as a reason. A reply that contains the right number but tells the learner their wrong number was correct needs attention too.
Accuracy depends on what the response communicates, not only on whether a reference answer appears somewhere in the text.
Separate the style review from that factual review. A response may be accurate and unnecessarily cold. Another may be readable and wrong. If one overall score combines those properties, an improvement in tone can conceal a decline in correctness.
Then test the full conversation. A good first reply does not tell you how the assistant behaves when the user pushes back, supplies contradictory evidence or asks it to agree more firmly. Those later turns are part of the product people use.
The algebra example also changes on a second turn. Suppose the learner insists that a teacher told them the answer is 7. A dependable tutor should check the equation again, explain the substitution and allow for a transcription error in the question. It should not invent a way for 2 × 7 + 3 to equal 11 to preserve agreement.
A product team can include that exchange in a small regression set, along with correct answers that should receive agreement. Testing only mistakes could encourage a model that objects unnecessarily. The desired behaviour includes accepting a correct claim, correcting an incorrect one and asking when the available information is insufficient.
There will still be cases where tone needs human judgement. A terse correction can discourage a learner; an extended reassurance can bury the correction. Reviewing the exact sentence that communicates the error gives the team a practical editing task. It can make that sentence easier to hear while leaving the mathematics unchanged.
For the next personality update, keep the existing model as a comparison and run the same conversations through both versions. Have reviewers identify the factual claim that changed before they decide which response they prefer.
Sources
One email when we publish. Research, product decisions, and what teams report back.
