The science under the hood.Latency, inference, compute, state.
How a spoken sentence becomes a typed decision, a model call, an action and a durable memory, and where every millisecond goes. Every figure on this page is cited to the published documentation of the infrastructure we build on.
A reply feels human when the stages overlap. Streaming STT, a context-aware end-of-turn model, preemptive generation and colocated GPUs each remove a serial wait. Flip them to see the shape change.
trace · single turnschematic · not to scale except labelled values
first audio
caller.speech
stt.streaminterim → final · Nova-3 / Flux on GPU 5
turn.end_of_utterancemin delay 0.3 s · model ~50–160 ms/turn 9
STT, LLM and TTS colocated with the agent runtime. No external API on the hot path.
02Inference
Classifyfirst.Thenspendcomputewhereitpays.
Most decisions in a business are bounded: route, score, qualify, escalate. Those go to a non-generative typed model. Generation is reserved for the work that needs it, on the lane built for it.
request
classify
lane 1
Typed decision
Jev · non-generative
Choice, Score and Noul questions evaluated in parallel against one state. Returns typed values, probabilities and confidence 18. Published example: 0.114 s vs 8.566 s for an LLM 19.
lane 2
Voice LLM
real-time · reasoning off
Voice-verified open-weight models on colocated GPUs. Reasoning is always disabled on voice calls 4.
lane 3
Long context
up to 1M tokens
Hosted open-weight models with 1M-token context for whole-history summaries and research 4.
lane 4
Model router
400+ models
Any model in a 400+ catalogue behind one OpenAI-compatible API, picked per task for quality, cost and latency 20.
→ lane 1 · Typed decision
A bounded question with a fixed answer set. No prose needed.
score: 0..1 · choice: "call_now" | "nurture" | "disqualify" · probabilities · confidenceIllustrative routing. Thresholds and model choices are configured per workspace.
Hosted open-weight models on colocated GPUs (listed by Telnyx, Oct 2026)4
Inference runs in the same facilities that terminate the call. Each voice region carries its own SFU, SIP, agent runtime and models, so the hot path never leaves the building.
failoverTo the next-lowest-latency region, not an error3
statelessChat-completions requests and responses are not stored3
04State and memory
Statefulbyconstruction.Oneactorpercustomer.
Every event for a customer routes to the same durable actor. Requests are serialized, so two agents never race on one record, and state recovers after a restart. Context for each turn is assembled from facts, recent history and retrieval.
01 ingestcall · sms · email · web · review
02 routeper-entity, one mailbox
03 persistdurable writes, merge-patch state
04 indexembeddings, 4,096-d
05 assemblefacts + recent + retrieved
customer_actor.tsillustrative shape
type CustomerActor = {
id: CustomerId // routes every event for this customer here
timeline: Event[] // calls, texts, email, visits, reviews
facts: Fact[] // each with source + observed_at
tasks: ScheduledTask[] // durable, survive restarts
policy: ContactPolicy // who may contact, when, on which channel
}
// One mailbox per customer: requests are serialized, writes are durable.on(event) → append(timeline) → extract(facts) → embed(chunks)
on(turn) → assemble(context: facts + recent + retrieved) → model
Building blocks: per-entity routing, serialized requests, durable writes and recovery 16; persistent history, durable scheduled tasks and merge-patch state 17; embeddings 4.
05Real-time media
WebRTCattheedge.Peoplecanstepinmid-call.
Distributed SFU mesh
Each participant connects to the nearest SFU; servers relay tracks to each other over a FlatBuffers-based protocol carrying RTP. Design target: a media server within 100 ms of anyone, 99.99+% availability 11.
Supervisor as a third leg
A manager joins a live call as a third leg without converting it to a conference, so the agent and caller are undisturbed 14.
Warm transfer with context
Hold the caller, brief the person privately, then connect. If nobody picks up, the call returns to the original flow 13.
Turn-aware STT
Deepgram Flux on colocated GPUs emits end-of-turn signals from the transcription stream itself, with tunable eager thresholds 5.
06Observability and evals
Everyturnisatrace.Everyversionisscored.
Audio, transcripts, traces and logs share one session timeline. Assistant versions are compared on instruction-following and estimated caller satisfaction before they ship wider.
session timelinetrace shape · illustrative · no measured values
session
turn[3]
vad.speech
stt.final
turn.eou
jev.decide intent
tool.crm.lookup
llm.generate
tts.stream
eval.instruction_following
memory.write facts
Session timeline 12; managed insights and version comparison 15.
07One endpoint
Naturallanguagein.Typedplansandapprovalsout.
One orchestration endpoint turns a sentence into a plan, typed decisions and actions. Anything above the workspace’s autonomy level stops for approval, and every step lands in memory.
illustrative shape · not a published API
POST /v1/ask
Content-Type: application/json
{
"workspace": "ws_demo",
"input": "Call back everyone who asked about pricing this week.",
"autonomy": "with_approval",
"stream": true
}
08Sources
Checkourwork.Everynumber,cited.
Figures are vendor-published numbers, defaults and design targets for the infrastructure IntelAgents builds on, checked on October 6, 2026. They are not IntelAgents benchmarks; real latency depends on region, models, carrier and network.
STT, TTS and LLM on GPUs colocated with the agent runtime, approximately 2 ms from the agent; on-net inbound call flow; autoscaled, isolated agent workers; each region a full stack.
GPU infrastructure across five regions on four continents; latency-based routing by ingress domain; failover to the next-lowest-latency region; storage governed by data locality.
Hosted open-weight models with parameter counts, context lengths and voice verification; reasoning disabled on voice calls; embedding models and dimensions.
Open-weights end-of-turn model: ~50–160 ms per turn, 396 MB on disk, 14 languages; endpointing minimum delay 0.5 s by default, 0.3 s with the audio detector.
Talk to us