Inference Fleet · ASC Docs

What it is

Inference Fleet is the managed serving layer that powers the Inference product. It takes a model + a runtime (open-source serving engine for LLMs, audio, image) and gives you a stable URL that scales from zero to thousands of replicas per-second. Cold start is sub-second on warm SKUs because weights are streamed from object storage via the Fast Weight Loader.

When to use it

Serve a fine-tuned model as an HTTP/JSON endpoint with no infra glue.
Run multiple replicas with traffic-split for A/B model rollouts.
Burst from 0 → 200 replicas during a launch, scale back down at night.

Quickstart

asc inference deploy llama3-70b-ft --weights s3://my-bucket/ckpt --runtime openai-compatible --gpu h100 --gpus 4 --min 0 --max 50

deployment = client.inference.deploy(
    name='llama3-70b-ft',
    weights_uri='asc://my-bucket/ckpt',
    runtime='openai-compatible',
    gpu='h100', gpu_count=4,
    min_replicas=0, max_replicas=50,
)

const dep = await client.inference.deploy({ name: 'llama3-70b-ft', weightsUri: 'asc://my-bucket/ckpt', runtime: 'openai-compatible', gpu: 'h100', gpuCount: 4, minReplicas: 0, maxReplicas: 50 });

Limits & quotas

Limit	Default	Burst	Notes
Cold start (warm SKU + warm weights)	<1 s	—
Cold start (cold weights)	<30 s	—	First request after dormancy
Max replicas per deployment	500	5,000	Lift via sales
Autoscale window	10 s	—	Decision interval

Pricing

See pricing. Pay-as-you-go, billed monthly via Stripe.

API surface

POST /v1/inference/deployments
POST /v1/inference/deployments/{id}/invoke

Required scope(s): inference:read, inference:write. See Scopes.

Security

All access is authenticated and scoped. See Auth & scopes and Network controls.