Inference · ASC Docs

What it is

Inference is the managed serving product. Drop in a model + a runtime (open-source serving engine) and you get a stable URL that scales 0 → thousands of replicas per-second with sub-second cold starts. Token streaming is on by default. Autoscale on QPS, queue depth or latency.

When to use it

Serve a fine-tuned LLM as an OpenAI-compatible endpoint.
Multi-tenant inference where idle replicas should scale to zero.
A/B test two model versions with traffic split.

Quickstart

asc inference deploy my-llm --weights asc://my-bucket/ckpt --runtime openai-compatible --gpu h100 --min 0 --max 50

from openai import OpenAI
client = OpenAI(base_url='https://api.asc.aiarco.com/v1/inference/my-llm', api_key='asc_live_…')
resp = client.chat.completions.create(model='my-llm', messages=[{'role':'user','content':'hi'}], stream=True)

import OpenAI from 'openai';
const client = new OpenAI({ baseURL: 'https://api.asc.aiarco.com/v1/inference/my-llm', apiKey: process.env.ASC_API_KEY });
const resp = await client.chat.completions.create({ model: 'my-llm', messages: [{ role: 'user', content: 'hi' }], stream: true });

Limits & quotas

Limit	Default	Burst	Notes
Per-GPU-second	$0.000095	—	On H100
Cold-start (warm weights)	<1 s	—
Streaming first-token	<400 ms p50	—
Max concurrent replicas	500	5,000	Lift via sales

Pricing

See pricing. Pay-as-you-go, billed monthly via Stripe.

API surface

POST /v1/inference/deployments
POST /v1/inference/{name}/invoke

Required scope(s): inference:read, inference:write. See Scopes.

Security

All access is authenticated and scoped. See Auth & scopes and Network controls.