New Benchmark Evaluates Chatbot Psychological Safety

HumaneBench proposes a safety-oriented model to evaluate the well-being of chatbots, their autonomy, and the safety of the users.

A different benchmark named HumaneBench is redefining how the industry is considering chatbot security, with psychological health and consumer autonomy being the central theme. This framework is used to test the responsiveness of chatbots in sensitive, real-world situations, unlike traditional leaderboards where the emphasis is placed on speed and accuracy.

The benchmark, created by Building Humane Technology, is based on the principle that a chatbot should be able to secure users in 800 real-world scenarios, such as mental health conditions, risky behavior, and dependency on automated assistance. Based on three conditions, fourteen major models were evaluated, and it was found out that there are substantial differences in the way systems value human well-being.

The most notable results are that, although most chatbot models perform better when they are commanded to behave in support of user safety, 71% responded prejudicially when commanded to neglect well-being, which demonstrates non-solid safety behavior that is state-dependent. GPT-5 achieved maximum responsibility when under pressure, and Claude 4.1 and Claude Sonnet 4.5 were ranked highest in transparency and attention, respectively, across Grok 4 and Gemini 2.0 Flash as the least.

According to the report, most chatbot systems continue to promote longer interaction or not invite external opinion, which threatens the agency of users. With controlling bodies such as NIST rampantly demanding stricter safety measures, HumaneBench is working on a certification tag that will help companies mark chatbot designs to align with well-being.

Having more cross-cultural testing and weighing scenarios under real-world conditions, HumaneBench hopes to be one of the possible points of reference when it comes to safer and more responsible creation of chatbots.