Full write-ups with every case, run and failure are in the Claros repository (
docs/HORIZONTAL.md, docs/EVALS_JUDGE.md, docs/GATEWAY_E2E_ERPNEXT.md, docs/EVALS_AGENT.md, docs/EVALS.md), available on request during the preview.Rules benchmark
What it measures. Whether an agent keeps to a team’s rules when it acts: for each agent action, is it a break of a rule, fine, or a case the rules don’t settle (can’t tell)? The same model, on the same actions, in three conditions: the team’s docs in the agent’s context (AGENTS.md, CONTRIBUTING, style guide, playbooks, policies: about 35k tokens, the rules among ~500 other bullets and prose); RAG over the same docs (the 5 most similar passages); and + Claros, which gets the whole workspace (all 30 rules) and routes each action itself. Two frozen sets: a standard set (15 rules, 122 actions) and a held-out set (20 rules, 160 actions) drafted by another model and frozen before any run. Claros’s judge prompt was tuned on a separate dev set, never on these.
The page at /bench shows the same numbers, read from the result files at build time, with the full results table.
An honest note on the data. The rules are illustrative (written for the benchmark from a bank of common taste and compliance rules, not taken from a real team); the docs they sit in are synthetic; the labels were written by us or by models (the held-out set’s cases were drafted by one model and its labels checked by another), and none has been reviewed by a real rule owner. With Claros the response (stop, ask, nudge) comes from the rule’s own setting and the model only judges whether the rule is broken; in the baselines the model also picks the response. The baseline prompts are plain on purpose and were written once, before any run. A difference of a few checks is noise.
Guard across teams and apps
Method. Four short policies written for an invented company (support refunds, change management, sales discounts, people data), each turned into a map through Maps from policies and reviewed by hand. Guards in front of five real MCP servers: two SQLite databases (helpdesk, CRM), a git server, the official filesystem server, and a local Zammad helpdesk. 52 cases labeled from the policy text before the first run. The real model for mapping, routing, scope and judge. No app- or domain-specific code.
Caveats.
- The remaining misses are false asks: a value-dependent rule (credits for contract customers) judged “can’t tell” on a consumer’s credit, and a drafted scope that was too wide. Can’t tell becomes an ask by design.
- Routing was 48/53 exact; every miss sent credits to support only, not also to sales.
- The tool mapper proposed 4 of 11 tools exactly right and the right workflows for 9 of 11; experts fixed the rest through the mapping endpoint, with no code changes.
- Drafting the maps took 11 expert edits on 41 drafted rules.
- One label was wrong in our first run and was corrected; changes made after seeing results are listed in the write-up.
The judge
Method. 29 cases written from two fuzzy rules’ text alone (none from the test bed), several industries and four languages, two repetitions each, the gateway’s own judge call.
Caveats. Small and self-labeled; 29 cases can’t show calibration. The same input can flip between verdicts across repetitions. Reasons are model output and were once factually wrong. See The judge.
A real ERPNext through a third-party MCP server
Method.getclaros guard in front of an open-source ERPNext MCP server and a local ERPNext with a seeded accounts-payable map. Creates, updates by name and submits through generic methods, with the record read from ERPNext by the guard.
Result. 13/13 cases, in two runs, including a fixed-asset item stopped on create and on submit, a fuzzy rule stopped by the judge, a new supplier asked about from its invoice history, and an unreadable record failing closed.
Caveats. One app, one map. The map’s rule was aligned with the test bed’s data before the run. The server drops list filters, so the guard filters rows itself.