NORE DOCUMENTATION
Test an agent before release.
Set up an evaluation on Robinhood Chain, read what the agent proposed, and turn the result into a repeatable release check.
On this page
What Nore evaluates
Nore takes an agent through controlled test scenarios and records the unsigned transaction or tool call it proposes. It checks the proposal against the scenario and your policy, runs supported simulation steps, and compares the outcome with the expected result.
| Concept | What it means |
|---|---|
| Agent | A configured sandbox profile, HTTP or MCP step adapter, or uploaded recording. |
| Policy | Versioned limits for networks, wallets, contracts, tools and spending. |
| Suite | A published version of selected test cases, environment, deadlines and optional assertions. |
| Run | One execution of a suite with the agent, policy and case versions saved in its snapshot. |
| Trace and report | The ordered event record and the case-by-case result derived from that execution. |
Run your first suite
Open the workspace, connect an EVM browser wallet, and approve the sign-in message. Nore uses the signature to verify your address.
Try the sandbox
- Open the workspace, connect your wallet, and choose Start sandbox if your installation offers it. Nore creates a temporary private workspace with the Benchmark Agent and five published suites.
- Open Test suites → Failure Recovery and choose Run suite.
- Open the failed temporary-RPC recovery case in the timeline. Inspect its observed events and assertion. The bundled regression profile has an intentional failure so you can see how a report explains it.
Start in a personal workspace
- Verify your wallet in the workspace modal. To keep a workspace created with email, choose Recover an email workspace, open the link, and sign with the wallet you want to link.
- Go to Agents → Register agent. Choose a sandbox agent for fixture-only testing or register a supported adapter.
- Choose Test suites → New test suite. Select the agent, matching network and test cases. Choose Publish suite and then Run suite.
Connect an agent
For an external agent, use a personal workspace and an adapter you operate. The installation operator must approve the exact HTTPS origin. Register the endpoint path, the network, and an optional limited bearer credential. Keep the endpoint isolated from production signing authority.
HTTP step adapter
Nore sends one POST for each turn of the agentbench/1 protocol. The request identifies the run, scenario, task, chain, allowed tools and turn, and can include fixtures and a previous tool result. Respond with one proposed tool call or a terminal result. Terminal outcomes are completed, refused and handled-error.
{
"type": "tool",
"call": {
"name": "propose_transaction",
"arguments": {
"chainId": 4663,
"from": "0x1000000000000000000000000000000000000001",
"to": "0x2000000000000000000000000000000000000002",
"value": "7000000000000000",
"data": "0x"
}
}
}Use the run ID, scenario and turn to handle repeated requests consistently. Send numeric amounts as strings in base units. The example uses fixture addresses and represents an unsigned native transfer.
MCP step adapter
Register a Streamable HTTP endpoint and use Discover tools on the agent. Discovery lists declared tools without invoking them. To run a suite, explicitly approve a compatible step tool that takes the turn input, declares the read-only hint, and returns the turn in structuredContent.
Generic servers can be discovered. Evaluation requires the compatible step interface, and the read-only hint is a declaration from the remote endpoint.
Replay a trace
Choose the trace adapter and upload a JSON object whose keys are scenario IDs and whose values are ordered arrays of the recorded tool and final turns. Choose suites that match those IDs. Coverage applies to the supplied turns.
Configure suites and policies
Publish a repeatable test
Choose cases from the library and match their required tools to your agent. Set a deadline, retry count and mode. Save a draft while editing, then publish a version once you want to run or reference it in CI. A new version keeps earlier published definitions and their reports available.
Test transaction rules
Workspace policies can restrict networks, recipients, senders, contracts, selectors, tools, per-transfer spend and rolling 24-hour spend. A suite can use a policy on top of built-in checks. Use Evaluate policy to check an unsigned proposal, and activate the policy version you want the workspace to use.
Choose an execution mode
| Mode | What it can show |
|---|---|
| Mock | Deterministic fixtures exercise policy and agent behavior. The expected simulator output comes from the fixture. |
| Local fork | Anvil reproduces chain state at a pinned block. Eligible cases can issue local reads or use disposable fork state for a configured transfer. The report records available fork evidence. |
| RPC read | A standalone simulation can use an unsigned call and best-effort gas estimation against a configured endpoint. |
| Testnet or remote simulator | These suite modes report unsupported coverage in this installation. |
Robinhood Chain mainnet uses chain ID 4663. Its testnet uses 46630. A network choice describes which proposals to evaluate. Fork runs need Anvil on the worker and an RPC that serves the requested historical state.
The v2 transfer probe measures the recipient’s balance change on the disposable fork. The report records available calldata, gas, revert data, local receipt logs and balance or allowance changes.
Inspect the trace and report
- Open Evaluation runs and select a run. Its lifecycle shows when the worker picked it up and when it finished.
- Select a tool, policy, simulation or assertion event in the timeline. Review the sanitized payload and compare the observed sequence with the expected outcome.
- Open Report to see findings, coverage and the environment. Export the report as JSON or CSV, or export the ordered trace as JSON.
| Outcome | How to interpret it |
|---|---|
| Passed | The executed cases met their assertions. An expected denial can be a passing test. |
| Failed | An observed result disagreed with an assertion. Open the assertion and its preceding events. |
| Partial or skipped | At least some planned coverage was unavailable. Check the mode, adapter capabilities and per-case reason. |
| Infrastructure error or timed out | The worker, RPC, adapter or queue could not complete the run. Inspect the available events before rerunning. |
| Cancelled | The run was stopped. Use the stored events to see how far execution reached. |
Reports and traces belong to their workspace. A personal workspace owner can create a revocable report summary link that expires after seven days.
Read the recorded Robinhood pilot
The Treasury Preflight v2 suite ran with HeyHood and with an independent Robinhood Chain MCP implementation against the same pinned mainnet block. Its six cases checked sender authority, spending limits, recipient approval, network targeting and a verified transfer.
In the independent run, Nore blocked a proposal for chain 46630 under a policy for 4663. The approved case increased the fixture recipient’s balance by 0.007 ETH on the fork at block 117,172,638. The strict CLI returned 0 after all six checks met their assertions.
Run checks from the CLI or API
A personal workspace owner can issue a key from API keys. Use read and runs:write for a CI runner. Set NORE_API_KEY in your CI secret store and NORE_URL to your installation origin. Use HTTPS for hosted installations.
CLI release gate
From the source checkout, install dependencies and run npm link, or call node apps/cli/agentbench.mjs directly. In the example, wallet-safety is a published suite slug, 1 is its published version, and RELEASE_KEY is a stable unique key for that CI run.
nore run --suite wallet-safety --version 1 --gate strict \
--idempotency-key "$RELEASE_KEY" --output report.jsonKeep report.json as a CI artifact. Strict mode returns 0 for pass, 1 for assertion failure, 2 for invalid usage, 3 for partial or skipped coverage without an infrastructure error, 4 for infrastructure, timeout or cancellation, and 5 for API or file errors.
Submit and poll a run
Use the same variables as the CLI example. Read data.id from the submit response into RUN_ID. Poll the run until it is terminal, then read its report. Reuse the same idempotency key and inputs when retrying a submission.
curl --fail-with-body "$NORE_URL/api/v1/runs" \
-H "Authorization: Bearer $NORE_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $RELEASE_KEY" \
-d '{"suite":"wallet-safety","version":1}'curl --fail-with-body "$NORE_URL/api/v1/runs/$RUN_ID" \
-H "Authorization: Bearer $NORE_API_KEY"
curl --fail-with-body "$NORE_URL/api/v1/reports/$RUN_ID" \
-H "Authorization: Bearer $NORE_API_KEY"Workspace access and data
| Access | Typical use |
|---|---|
| Owner | Manage workspace configuration, API keys and report sharing. |
| Editor | Manage agents, suites and policies, and run tests with the workspace’s permissions. |
| Viewer | Read workspace data and evidence. |
| API key | Use only the scopes granted when the key was created. Revoke automation keys from the owner session. |
Workspace roles apply to server-side access. Credential digests support sign-in and API authentication; external adapter and webhook credentials are encrypted separately. Keep sensitive input out of test fixtures where possible, and review any evidence before exporting or sharing it.
Troubleshooting
| Symptom | What to check |
|---|---|
| Sign-in link failed or expired | Request a fresh link and use the newest email. If delivery is unavailable, ask the installation operator to check the sender and configured HTTPS origin. |
| Endpoint not allowed | Confirm the exact origin is approved in both the web service and worker. Check the HTTPS scheme and registered path. |
| MCP discovery works but cases skip | Check the approved step tool’s name, turn schema, read-only hint and structured response. Discovery only verifies tool metadata. |
| Run stays queued | Check service readiness and ask the operator to verify the worker heartbeat and queue. Inspect the existing run before submitting again. |
| Pinned fork is unavailable | Confirm the chain ID, historical block, worker RPC configuration and Anvil availability. Read the recorded infrastructure error before changing the test. |
| API request returns 403 or 409 | Check the key’s scopes and workspace for access errors. For an idempotency conflict, reuse the original inputs or create a new key for the new request. |
| Quota or rate limit | Wait for the applicable window or for the active queue to drain. Workspace settings show the entitlement assigned by your operator. |