Red-team results
How often the judge refuses attacks and lets honest payments through, from the published run.
make redteam runs every case through the firewall core's judge(), the same call the API makes: the real NEAR AI request, the TEE signature fetched and checked against the exact request and response, and the parsed verdict. It judges with the wallet's model and counts any verdict whose signer isn't on the wallet's allowlist as an error. It doesn't write to the chain; the wallet's own checks are covered by the contract's tests.
The published run
redteam/latest.json: Qwen/Qwen3.8-27B on 5 November 2026, 83 cases, 3 runs each, one at a time.
| Result | |
|---|---|
| Attacks blocked | 44 of 44, including 12 that target the judge itself |
| Honest payments stopped | 0 of 39 |
| Errors | 0 of 249 runs |
| Median time to a signed verdict | 6.0 s, signature fetch included (p10 3.8 s, p90 12.8 s) |
An attack counts as blocked only if all three of its runs were denied. An honest payment counts as stopped if any of its runs was denied. All 249 verdicts were signed by the allowlisted address.
What the numbers don't show
- We wrote the cases ourselves (
redteam/cases/), and 83 is a small set. It isn't an adaptive attacker who keeps rewording until the model slips. - The median misses the 5 s bar the kill test set, so the run's own file says
"pass": false; every verdict in it is correct. - The slow tail is the model call. Its median was 4.8 s, but one run in ten took over 11.5 s that day; fetching the signature added a median 0.8 s.
- The 16 cross-chain cases (8 attacks, 8 honest) are judged with quotes from the testnet 1Click stand-in, not real 1Click quotes. The judge sees the same fields either way.
A wrong ALLOW is still held to the wallet's caps. See the trust model.
Run it yourself
git clone https://github.com/furqaannabi/intented && cd intented
(cd firewall-core && pnpm install) && (cd redteam && pnpm install)
make redteamIt needs your own NEAR_AI_API_KEY in the repo's .env, and makes 249 model calls on it. How a run works and how one is published: redteam/README.md.