Baseten raises $300M — inference engineering is a discipline now

The infrastructure layer just got its valuation moment.

And because a large round does not explain itself, here is what the number is actually saying.

What Baseten builds

Baseten sits between the model and the product. You train or fine-tune a model somewhere upstream. You have an application downstream that needs to call it. Between those two things there is a substantial engineering problem: batching, routing, auto-scaling, GPU memory management, quantization, latency budgets, fallback logic, and observability.

Baseten handles that layer. Their platform packages models, provisions GPU infrastructure, and serves predictions at production load. The Truss open-source framework lets teams define model environments as code and deploy them to Baseten-managed or self-hosted infrastructure.

The customers are engineering teams that need to run custom or fine-tuned models — not teams that can route everything through a single hosted API. When the model is yours, the infrastructure problem is also yours.

What “inference engineering” actually means

Training gets the attention. Inference pays the bills.

At scale, inference cost routinely exceeds training cost by an order of magnitude. A model trained once will be queried millions of times. Each query touches GPU memory, CPU cycles, and network bandwidth. Efficiency at this layer is not a nice-to-have; it is the margin.

Inference engineering is the practice of making that layer fast, cheap, and predictable. It covers: model quantization (reducing weight precision without losing accuracy), continuous batching (grouping concurrent requests to share GPU passes), speculative decoding (drafting cheap tokens to verify in parallel), KV-cache management, and autoscaling against bursty workloads.

None of this is new as a technical problem. What is new is that it has become expensive enough at scale to justify a platform company valued at $5B.

Why NVIDIA is in the round

NVIDIA sells GPUs. Baseten fills them efficiently. The two interests align tightly.

Underutilized GPU capacity is waste — for the customer and, indirectly, for the ecosystem that needs GPU adoption to continue. Inference engineering platforms that maximize utilization per GPU-hour make the hardware purchase easier to justify. NVIDIA's investment here is ecosystem hygiene, not just financial return.

It also signals something about the stack. NVIDIA is now placing bets at the inference layer, not just at the silicon layer. That choice shapes roadmap conversations: where TensorRT-LLM integrates, where NIM fits, what the next generation of H-series cards optimizes for.

The inference stack — four layers

Inference engineering spans hardware to observability. Here is where each layer sits and who owns it.

What we notice

DWS IQ runs 18+ industrial agents on NVIDIA DGX Spark hardware. Our inference stack today is Ollama on-premises, routed through the DWS agent orchestration layer. The problems Baseten solves at scale — batching, routing, observability — are problems we manage by hand at our current load.

The round tells us two things. First, inference engineering is now a named career and a named budget line inside enterprise AI teams. Second, EU-sovereign inference — models running inside the EU perimeter, under GDPR and AI Act constraints — is a segment the US-headquartered platforms have not yet prioritized. That gap is ours to build.

NVIDIA's participation also matters to us directly. The DGX Spark roadmap and the inference platform ecosystem evolve together. Watching which platforms NVIDIA co-invests in tells us where the hardware-software joint optimization will land next.