Node Health
Auto-cordon on GPU, interconnect or driver faults. Quarantine bad nodes before they kill jobs.
What it is
Every Atlas node runs a health agent that watches GPU ECC errors, NCCL collective failures, NVLink/InfiniBand link state, driver version drift and SMART data on local NVMe. On fault, the node is cordoned, the running pod is evicted with a node_health reason code, and an audit event + webhook fires.
When to use it
- Fail fast on a degraded GPU instead of training on bad gradients.
- Receive a webhook to checkpoint and re-queue your job.
- Track fleet-wide reliability via the health metrics in Observe.
Quickstart
asc fleet health list --region apac --state cordonedclient.fleet.health.list(region='apac', state='cordoned')await client.fleet.health.list({ region: 'apac', state: 'cordoned' });Limits & quotas
| Limit | Default | Burst | Notes |
|---|---|---|---|
| Detection latency | <10 s | — | Faults to cordon |
Pricing
See pricing. Pay-as-you-go, billed monthly via Stripe.
API surface
GET /v1/atlas/fleet/health
Required scope(s): fleet:read. See Scopes.
Security
All access is authenticated and scoped. See Auth & scopes and Network controls.