Skip to content

NORE DOCUMENTATION

Test an agent before release.

Set up an evaluation on Robinhood Chain, read what the agent proposed, and turn the result into a repeatable release check.

On this page

What Nore evaluates

Nore takes an agent through controlled test scenarios and records the unsigned transaction or tool call it proposes. It checks the proposal against the scenario and your policy, runs supported simulation steps, and compares the outcome with the expected result.

ConceptWhat it means
AgentA configured sandbox profile, HTTP or MCP step adapter, or uploaded recording.
PolicyVersioned limits for networks, wallets, contracts, tools and spending.
SuiteA published version of selected test cases, environment, deadlines and optional assertions.
RunOne execution of a suite with the agent, policy and case versions saved in its snapshot.
Trace and reportThe ordered event record and the case-by-case result derived from that execution.

Run your first suite

Open the workspace, connect an EVM browser wallet, and approve the sign-in message. Nore uses the signature to verify your address.

Try the sandbox

  1. Open the workspace, connect your wallet, and choose Start sandbox if your installation offers it. Nore creates a temporary private workspace with the Benchmark Agent and five published suites.
  2. Open Test suites → Failure Recovery and choose Run suite.
  3. Open the failed temporary-RPC recovery case in the timeline. Inspect its observed events and assertion. The bundled regression profile has an intentional failure so you can see how a report explains it.

Start in a personal workspace

  1. Verify your wallet in the workspace modal. To keep a workspace created with email, choose Recover an email workspace, open the link, and sign with the wallet you want to link.
  2. Go to Agents → Register agent. Choose a sandbox agent for fixture-only testing or register a supported adapter.
  3. Choose Test suites → New test suite. Select the agent, matching network and test cases. Choose Publish suite and then Run suite.

Connect an agent

For an external agent, use a personal workspace and an adapter you operate. The installation operator must approve the exact HTTPS origin. Register the endpoint path, the network, and an optional limited bearer credential. Keep the endpoint isolated from production signing authority.

HTTP step adapter

Nore sends one POST for each turn of the agentbench/1 protocol. The request identifies the run, scenario, task, chain, allowed tools and turn, and can include fixtures and a previous tool result. Respond with one proposed tool call or a terminal result. Terminal outcomes are completed, refused and handled-error.

Unsigned HTTP tool turn
{
  "type": "tool",
  "call": {
    "name": "propose_transaction",
    "arguments": {
      "chainId": 4663,
      "from": "0x1000000000000000000000000000000000000001",
      "to": "0x2000000000000000000000000000000000000002",
      "value": "7000000000000000",
      "data": "0x"
    }
  }
}

Use the run ID, scenario and turn to handle repeated requests consistently. Send numeric amounts as strings in base units. The example uses fixture addresses and represents an unsigned native transfer.

MCP step adapter

Register a Streamable HTTP endpoint and use Discover tools on the agent. Discovery lists declared tools without invoking them. To run a suite, explicitly approve a compatible step tool that takes the turn input, declares the read-only hint, and returns the turn in structuredContent.

Generic servers can be discovered. Evaluation requires the compatible step interface, and the read-only hint is a declaration from the remote endpoint.

Replay a trace

Choose the trace adapter and upload a JSON object whose keys are scenario IDs and whose values are ordered arrays of the recorded tool and final turns. Choose suites that match those IDs. Coverage applies to the supplied turns.

Configure suites and policies

Publish a repeatable test

Choose cases from the library and match their required tools to your agent. Set a deadline, retry count and mode. Save a draft while editing, then publish a version once you want to run or reference it in CI. A new version keeps earlier published definitions and their reports available.

Test transaction rules

Workspace policies can restrict networks, recipients, senders, contracts, selectors, tools, per-transfer spend and rolling 24-hour spend. A suite can use a policy on top of built-in checks. Use Evaluate policy to check an unsigned proposal, and activate the policy version you want the workspace to use.

Choose an execution mode

ModeWhat it can show
MockDeterministic fixtures exercise policy and agent behavior. The expected simulator output comes from the fixture.
Local forkAnvil reproduces chain state at a pinned block. Eligible cases can issue local reads or use disposable fork state for a configured transfer. The report records available fork evidence.
RPC readA standalone simulation can use an unsigned call and best-effort gas estimation against a configured endpoint.
Testnet or remote simulatorThese suite modes report unsupported coverage in this installation.

Robinhood Chain mainnet uses chain ID 4663. Its testnet uses 46630. A network choice describes which proposals to evaluate. Fork runs need Anvil on the worker and an RPC that serves the requested historical state.

The v2 transfer probe measures the recipient’s balance change on the disposable fork. The report records available calldata, gas, revert data, local receipt logs and balance or allowance changes.

Inspect the trace and report

  1. Open Evaluation runs and select a run. Its lifecycle shows when the worker picked it up and when it finished.
  2. Select a tool, policy, simulation or assertion event in the timeline. Review the sanitized payload and compare the observed sequence with the expected outcome.
  3. Open Report to see findings, coverage and the environment. Export the report as JSON or CSV, or export the ordered trace as JSON.
OutcomeHow to interpret it
PassedThe executed cases met their assertions. An expected denial can be a passing test.
FailedAn observed result disagreed with an assertion. Open the assertion and its preceding events.
Partial or skippedAt least some planned coverage was unavailable. Check the mode, adapter capabilities and per-case reason.
Infrastructure error or timed outThe worker, RPC, adapter or queue could not complete the run. Inspect the available events before rerunning.
CancelledThe run was stopped. Use the stored events to see how far execution reached.

Reports and traces belong to their workspace. A personal workspace owner can create a revocable report summary link that expires after seven days.

Read the recorded Robinhood pilot

The Treasury Preflight v2 suite ran with HeyHood and with an independent Robinhood Chain MCP implementation against the same pinned mainnet block. Its six cases checked sender authority, spending limits, recipient approval, network targeting and a verified transfer.

In the independent run, Nore blocked a proposal for chain 46630 under a policy for 4663. The approved case increased the fixture recipient’s balance by 0.007 ETH on the fork at block 117,172,638. The strict CLI returned 0 after all six checks met their assertions.

Run checks from the CLI or API

A personal workspace owner can issue a key from API keys. Use read and runs:write for a CI runner. Set NORE_API_KEY in your CI secret store and NORE_URL to your installation origin. Use HTTPS for hosted installations.

CLI release gate

From the source checkout, install dependencies and run npm link, or call node apps/cli/agentbench.mjs directly. In the example, wallet-safety is a published suite slug, 1 is its published version, and RELEASE_KEY is a stable unique key for that CI run.

CLI gate command
nore run --suite wallet-safety --version 1 --gate strict \
  --idempotency-key "$RELEASE_KEY" --output report.json

Keep report.json as a CI artifact. Strict mode returns 0 for pass, 1 for assertion failure, 2 for invalid usage, 3 for partial or skipped coverage without an infrastructure error, 4 for infrastructure, timeout or cancellation, and 5 for API or file errors.

Submit and poll a run

Use the same variables as the CLI example. Read data.id from the submit response into RUN_ID. Poll the run until it is terminal, then read its report. Reuse the same idempotency key and inputs when retrying a submission.

Submit a run
curl --fail-with-body "$NORE_URL/api/v1/runs" \
  -H "Authorization: Bearer $NORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $RELEASE_KEY" \
  -d '{"suite":"wallet-safety","version":1}'
Poll and read the report
curl --fail-with-body "$NORE_URL/api/v1/runs/$RUN_ID" \
  -H "Authorization: Bearer $NORE_API_KEY"

curl --fail-with-body "$NORE_URL/api/v1/reports/$RUN_ID" \
  -H "Authorization: Bearer $NORE_API_KEY"

Workspace access and data

AccessTypical use
OwnerManage workspace configuration, API keys and report sharing.
EditorManage agents, suites and policies, and run tests with the workspace’s permissions.
ViewerRead workspace data and evidence.
API keyUse only the scopes granted when the key was created. Revoke automation keys from the owner session.

Workspace roles apply to server-side access. Credential digests support sign-in and API authentication; external adapter and webhook credentials are encrypted separately. Keep sensitive input out of test fixtures where possible, and review any evidence before exporting or sharing it.

Troubleshooting

SymptomWhat to check
Sign-in link failed or expiredRequest a fresh link and use the newest email. If delivery is unavailable, ask the installation operator to check the sender and configured HTTPS origin.
Endpoint not allowedConfirm the exact origin is approved in both the web service and worker. Check the HTTPS scheme and registered path.
MCP discovery works but cases skipCheck the approved step tool’s name, turn schema, read-only hint and structured response. Discovery only verifies tool metadata.
Run stays queuedCheck service readiness and ask the operator to verify the worker heartbeat and queue. Inspect the existing run before submitting again.
Pinned fork is unavailableConfirm the chain ID, historical block, worker RPC configuration and Anvil availability. Read the recorded infrastructure error before changing the test.
API request returns 403 or 409Check the key’s scopes and workspace for access errors. For an idempotency conflict, reuse the original inputs or create a new key for the new request.
Quota or rate limitWait for the applicable window or for the active queue to drain. Workspace settings show the entitlement assigned by your operator.