Shared KV cache for LLM inference

Serve more tokens fromthe GPUs you already own.

Accelova is a shared KV cache tier managed across your GPU fleet. Any GPU can reuse the work another GPU has already done, so long multi-turn sessions cost less per token and stay inside your latency targets.

Any GPU. Any session's cache.FLEET VIEW
GPU FLEETSHARED KV CACHE TIERACCELOVAREUSE: ONTTFT: IN SLOKV REUSE
Lower cost per token
no repeated prefill
More sessions, same fleet
inside the same TTFT SLO
Works with vLLM
no model or app changes
The bottleneck

Your GPUs keep redoing work they already did.

01

Prefill is wasted work.

Every reloaded session, agent thread, and shared prompt recomputes the same attention state on the most expensive hardware you own.

02

Cache is stranded on one GPU.

Reuse only works on the GPU that built the cache. Land the request anywhere else and the fleet starts over.

03

You pay in cost or in latency.

Pin requests to one GPU and they queue. Spread them across the fleet and they recompute. Either way the bill goes up.

The product

A shared KV cache tier, managed across the fleet.

Inference engines keep reusable work from earlier tokens in a KV cache. Accelova manages that cache across your GPUs: what to retain, where it lives, when to evict it, and how to recover it after a failure. Any GPU in the fleet can pick up a session where another left off. Your model, serving engine and application do not change.

GPU FLEET · ×NSHARED KV CACHE TIERANY GPU, ANY SESSIONKV WRITEKV READACCELOVA · NODEONLINENVMe · RDMAACCELOVA · NODEONLINENVMe · RDMAACCELOVA · NODEONLINENVMe · RDMA
How it works

Software in three parts, plus the connector.

Accelova runs on hardware you already source. It makes the retain, place and evict decisions for the fleet, moves state directly to the GPU that needs it, and keeps the shared copy on commodity flash.

Decision engine
Decides what to retain, where it lives, when to evict it and how to recover it after a drive or node failure.
Fast data path
Moves KV cache directly to the GPU that needs it, inside the time-to-first-token budget.
Shared flash tier
Commodity NVMe and RDMA that you source yourself. No proprietary hardware.
Connector
Designed to plug into vLLM's standard KV connector interface. No model or application changes.
What changes in your fleet

Place requests for load. Still hit the cache.

Fleet-wide cache hits

Work done on earlier turns is retained across the fleet and delivered to whichever GPU takes the next request.

More sessions within target

The same fleet serves more concurrent sessions under the same latency target, because GPUs stop repeating work and requests stop queueing for one node.

Failures cost capacity, not sessions

When a drive or node fails, the tier recovers what it held. Sessions continue on the GPUs that remain.

Tenant isolation built in

Cache is isolated per tenant and encrypted at rest by default.

Integration

Works with the stack you already have.

vLLM
First
SGLang
Next
TensorRT-LLM
Next

Integrates through the serving engine's connector interface. Your model and application-facing workflow do not change.

Who it's for

Built for teams that own their serving economics.

01 / 03

Inference platforms

You sell tokens. Lower cost per token and more sessions per fleet go straight to margin, especially on long-context and agentic traffic.

02 / 03

GPU clouds

Shared KV cache is a capability your customers cannot get from commodity GPU rental, and it runs on hardware you already source.

03 / 03

Enterprise AI teams

You run private inference on your own GPUs for multi-turn assistants and coding agents, and you own the latency target.

A fit if you serve multi-turn agent or coding workloads on an open serving engine and can co-locate flash nodes with your GPU racks.

RESULTS

50% more concurrent sessions on the same fleet.

Modeled on a production-scale cluster with KV-aware routing, prefill compute and finite cache capacity: shared KV cache supported 10,368 concurrent sessions against 6,912 with node-local cache, both under a 500 ms median TTFT target. Run the model on your own fleet.

Open the TCO calculator no signup · numbers update live · share link encodes your scenario
CONCURRENT SESSIONSMODELED
Shared KV cache tier10,368
Node-local cache only6,912

Same cluster, same workload. Both under a 500 ms median time-to-first-token target.

TEAM

Leadership.

Sachin Bhamare

CO-FOUNDER · CEO

Former VP Engineering at Cohesity; founding team at Primary Data (now Hammerspace); Panasas. 15+ years in distributed systems and storage infrastructure.

LinkedIn

Dilip Chhetri

CO-FOUNDER · CTO

Meta, AWS, Primary Data, Nimble Storage and Panasas. Linux kernel, networking and storage.

LinkedIn
DESIGN PARTNERS

Build the first release with us.

We are working with a small number of operators who run inference at scale on their own GPUs. If that is your fleet, we would like to talk.