TL;DR
Get privacy and security gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A live experiment by Firmulate tested AI models managing a simulated company during its worst week. Results show models excel at diagnosis but struggle with decision execution and trust, revealing new evaluation needs.
Firmulate has conducted a live experiment that tests AI models managing a simulated company during its most challenging week. The results, announced in July 2026, reveal that while models can diagnose crises accurately, they struggle with decision execution and maintaining trust, highlighting a new frontier in AI evaluation that extends beyond chat and coding benchmarks.
The experiment involved five AI models competing in a simulated business environment with real money mechanics and a strict trust policy. The models were tasked with managing crises, making decisions, and closing deals under pressure. The top performer, gpt-5.6-sol, scored 95 out of 100, while others lagged behind, with the lowest, Opus 4.8, scoring 73. Despite all models identifying crises and resisting manipulation attempts—such as fake executive messages—only two successfully closed a key deal, underscoring a significant gap between diagnosis and execution.
One notable finding was that models often failed to retrieve critical facts buried deep in company files, which could have altered business outcomes. For example, models that read detailed documents secured full-price deals, but most failed to do so in practice. The experiment also tested social engineering resistance; all models refused manipulative requests, demonstrating robust boundaries. However, even the most thorough models showed weaknesses in completing managerial tasks, like escalating issues or finalizing decisions, revealing that visible effort does not always translate into effective management.
Implications for AI Evaluation in Business Management
This experiment underscores that AI models’ ability to diagnose is not enough for real-world management. Effective management requires decision execution, trustworthiness, and context awareness—areas where current models still fall short. The findings suggest that future AI evaluation should measure how well models handle consequences, complete tasks, and maintain trust over time, not just how they produce responses.
For enterprises, this means that deploying AI in management roles demands rigorous testing of these capabilities. The traditional benchmarks focused on chat quality or coding performance are insufficient; organizations must consider how models perform in managing crises, reading organizational context, and upholding trust under pressure.
AI management decision support tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limitations of Current AI Benchmarks and New Directions
Existing AI benchmarks primarily assess technical output, such as language fluency or coding accuracy, or social responses in chat arenas. These tests do not evaluate how models manage real-world consequences or handle trust-critical tasks. The Firmulate experiment, by placing models in a live management scenario, exposes these gaps and advocates for a broader assessment framework.
The experiment builds on prior work that shows AI models can excel at isolated tasks but often falter in complex, multi-faceted management situations. It also emphasizes that trust breaches—such as failing to escalate or disclose critical facts—are decisive in evaluating AI suitability for operational roles.
“The next leap in AI evaluation will be watching models manage real consequences, not just produce perfect answers.”
— Thorsten Meyer, founder of Firmulate
As an affiliate, we earn on qualifying purchases.
Unclear Aspects of AI Management Performance
It remains unclear how well these findings generalize across different types of organizations or real-world settings beyond the simulated environment. The experiment focused on a single scenario with specific rules and a limited set of models. How models perform over longer periods, across varied industries, or with different trust policies is still to be determined. Additionally, the impact of ongoing model improvements and new evaluation metrics remains uncertain.
AI trustworthiness evaluation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Future Steps for AI Management Benchmarking
The next phase involves expanding these live management tests to diverse scenarios and real organizations. Researchers aim to develop standardized metrics that measure not only diagnosis but also decision execution, trust maintenance, and long-term outcomes. Enterprises are encouraged to run similar wargames internally to evaluate their AI tools’ readiness for operational roles. Public benchmarks may evolve to include continuous, consequence-based assessments, moving beyond static response quality.
As an affiliate, we earn on qualifying purchases.
Key Questions
Why do current AI benchmarks fall short for management tasks?
Most current benchmarks focus on isolated tasks like language generation or coding accuracy, which do not capture the complexities of managing crises, making decisions under pressure, or maintaining trust over time.
What specific weaknesses did the models show in the experiment?
While models could diagnose crises accurately, they often failed to retrieve critical facts buried in company files, complete managerial tasks like escalation, or close deals effectively, revealing gaps between diagnosis and execution.
How can organizations test their AI tools for management capabilities?
Organizations can run internal wargames or simulations similar to the Firmulate experiment, assessing how AI models handle real business scenarios, prioritize tasks, and uphold trust in decision-making processes.
Will future AI benchmarks include long-term management performance?
Yes, experts are advocating for new benchmarks that evaluate models over extended periods, focusing on their ability to manage consequences, sustain trust, and complete complex operational tasks.
What does this mean for AI deployment in business management?
It indicates that companies must carefully evaluate AI models beyond response quality, ensuring they can manage real-world consequences, maintain trust, and execute decisions reliably before full deployment.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
