Autoscaling
Scale on concurrency and queue depth, with a floor you set. Keep one replica warm for latency, or scale to zero for cost.
NXAARA Platform service
Put a model behind an API and keep it standing up
Deploy open or custom models as serverless, dedicated or private endpoints. Autoscaling, streaming, per-token metering, and an API surface your existing client library already speaks.
Serving is where most AI projects actually break
A model that scores well in a notebook and a model that survives production traffic are different engineering problems. The second one involves cold starts, concurrency limits, tail latency, cost that scales in ways nobody modelled, and a pager that goes off at three in the morning.
Inference Cloud is the part of the platform that deals with that. You deploy a version from the registry, choose a serving mode, and get an endpoint with autoscaling, streaming responses, request logging and per-token metering already attached.
The API is OpenAI-compatible, which is a deliberate and unglamorous decision. It means you can point an existing client at a NXAARA endpoint by changing a base URL and a key, evaluate it against whatever you are using now, and switch back just as easily if it does not hold up. We would rather win that comparison than prevent you from running it.
Three serving modes exist because three cost profiles exist. Serverless when traffic is spiky and you would rather not pay for idle. Dedicated when latency has to be predictable. Private when the workload cannot leave a specific boundary.
- Serverless endpoints — scale to zero, pay per token, accept a cold start on the first request
- Dedicated replicas — reserved capacity, no cold start, predictable tail latency
- Private endpoints — single-tenant, in your network or ours, no shared infrastructure
- Streaming — token-by-token server-sent events for anything a human is waiting on
- OpenAI-compatible API — change a base URL and a key, keep your client library
- Per-token metering — input and output tokens broken down per endpoint, per project, per day
Capabilities
What sits around the endpoint
An endpoint is a small part of serving. These are the parts that decide whether it is still up in six months.
Versioned deployment
Endpoints point at a specific model version. Roll forward or back by changing the pointer, not by redeploying infrastructure.
Traffic splitting
Send a percentage of traffic to a new version, compare quality and latency on real requests, and shift the split when you are satisfied.
Rate limits and quotas
Per-key limits so one misbehaving client cannot exhaust a shared endpoint or your monthly budget.
Request logging
Optional, off by default, project-scoped. Log for debugging or evaluation, with retention you configure and can switch off entirely.
Metering and attribution
Token counts and cost attributed per key and per project, so internal chargeback is arithmetic instead of an argument.
How it works
Getting a model to production traffic
Each step gates the next — this is the deployment sequence, not a menu.
Select a version
Choose a model version from the registry. It carries its evaluation result and lineage with it.
Choose a serving mode
Serverless, dedicated or private, based on your latency requirement and traffic shape.
Configure scaling
Set minimum and maximum replicas, concurrency per replica, and the timeout your client actually honours.
Issue a scoped key
Create an API key scoped to this endpoint with its own rate limit, so you can revoke it without touching anything else.
Route and watch
Send traffic, or split it against the incumbent. Watch latency, error rate and token cost from the first request.
Latency, cost and the trade-off between them
Serverless endpoints scale to zero, which is excellent for cost and poor for the first request after an idle period. A cold start on a large model is measured in seconds, not milliseconds. If a human is waiting on that response, they will notice.
Dedicated replicas remove cold starts entirely because the model is already resident. You pay for the replica whether or not it is busy, which is straightforwardly better economics once utilisation is reasonably steady.
The practical pattern most teams land on is a small dedicated floor for the traffic they always have, with serverless absorbing spikes above it. You can configure exactly that, and the console shows utilisation against cost so the decision is made from your numbers rather than from a rule of thumb.
| Serving modes | Serverless, dedicated replicas, private single-tenant, on-premises |
|---|---|
| API | OpenAI-compatible REST, with streaming via server-sent events |
| Model sources | Open-weight catalogue models, versions tuned in Model Foundry, or your own registered weights |
| Scaling | Concurrency and queue-depth based, configurable minimum and maximum replicas |
| Deployment controls | Version pinning, traffic splitting, instant rollback |
| Security | Scoped API keys, per-key rate limits, rotation and revocation, TLS in transit |
| Logging | Optional and off by default, project-scoped, configurable retention |
| Metering | Input and output tokens per key, per endpoint, per project |
Where it is used
Typical production shapes
Customer-facing assistants
Streaming responses where a person is watching the text appear, so time-to-first-token matters more than total generation time.
Internal document processing
High-volume extraction and classification where throughput and cost per thousand documents are the numbers that matter.
Embedded product features
A model behind a feature inside an existing application, with rate limits per tenant and clean cost attribution per customer.
FAQ
Common questions
Is the API really compatible with OpenAI client libraries?
Yes, for chat completions, streaming and embeddings. Change the base URL and the API key and your existing client works. This is intentional — it makes evaluating us against your current provider a fifteen-minute job rather than a migration project.
What is the cold start on a serverless endpoint?
It depends on model size and quantisation, and it is measured in seconds rather than milliseconds. If that matters to your users, run a dedicated replica with a minimum of one so there is always a warm instance. Most production deployments do exactly that.
Do you log my prompts and completions?
Not by default. Request logging is off unless you switch it on for a project, and when it is on the logs are project-scoped with a retention period you set. You can disable it and delete existing logs at any time.
Can I roll back a bad deployment quickly?
Yes. Endpoints point at a model version, so rolling back is repointing the endpoint at the previous version. It takes effect immediately and does not require redeploying infrastructure or rebuilding anything.
How do I stop one client burning my whole budget?
Scope an API key per client with its own rate limit, and set a project spend ceiling. The console shows token consumption per key, so when something spikes you can see which key did it rather than guessing.
Deploy an endpoint and point your client at it
Change a base URL, send real traffic, and compare latency and cost against whatever you are running today.