Fast Weight Loader
Sub-second model load from object storage. Streams shards directly into GPU memory.
What it is
Loading a 70 B model from object storage with a naive reader takes tens of seconds — too long for serverless inference. The Fast Weight Loader serialises model weights into a shardable, mmap-friendly format and streams them straight into pinned GPU memory on cold start. Result: sub-second model warmup for models that previously took 30+ s.
When to use it
- Serverless inference patterns where cold-start latency matters.
- Multi-tenant model swap-in/out on a shared GPU pool.
- Recovery after a node-health eviction.
Quickstart
asc weights pack --in ./hf_ckpt --out ./packed.asc
asc storage upload my-bucket packed.asc
asc inference deploy my-model --weights asc://my-bucket/packed.asc --fast-loadfrom aiarco.weights import pack
pack(in_dir='./hf_ckpt', out_path='./packed.asc')
client.storage.upload('my-bucket', 'packed.asc', open('./packed.asc', 'rb'))// Use the Python tool to pack; reference the packed artifact from TS deploys.
await client.inference.deploy({ name: 'my-model', weightsUri: 'asc://my-bucket/packed.asc', fastLoad: true });Limits & quotas
| Limit | Default | Burst | Notes |
|---|---|---|---|
| Max packed file size | 2 TiB | — | |
| Supported formats | safetensors, bin, pt | — | Output is a shardable container |
Pricing
See pricing. Pay-as-you-go, billed monthly via Stripe.
API surface
POST /v1/inference/weights/pack(CLI-only)
Required scope(s): inference:write. See Scopes.
Security
All access is authenticated and scoped. See Auth & scopes and Network controls.