AIARCOASC Docs
Sign inStart free

Evals

Model evals — datasets, runners, leaderboards.

What it is

Evals lets you define datasets (jsonl), runners (a function that takes a row and returns a prediction) and judges (rule-based, or LLM-as-judge). Runs are reproducible and produce a leaderboard scoped to your project.

When to use it

  • Compare two fine-tunes on the same eval set.
  • Track regressions across deploys.
  • LLM-as-judge with a structured rubric.

Quickstart

asc evals datasets create golden --file ./golden.jsonl
asc evals runs submit golden --runner my-model --judge llm-as-judge
client.evals.datasets.create('golden', file='./golden.jsonl')
client.evals.runs.submit(dataset='golden', runner='my-model', judge='llm-as-judge')
await client.evals.runs.submit({ dataset: 'golden', runner: 'my-model', judge: 'llm-as-judge' });

Limits & quotas

LimitDefaultBurstNotes
Max dataset rows1,000,000—
Concurrent runs10 / project—

Pricing

See pricing. Pay-as-you-go, billed monthly via Stripe.

API surface

  • POST /v1/evals/datasets
  • POST /v1/evals/runs

Required scope(s): evals:write. See Scopes.

Security

All access is authenticated and scoped. See Auth & scopes and Network controls.