Architecture

A model on its own is inert - it reads tokens and predicts the next ones. The harness is everything wrapped around it: the environment prompts and tools flow into, and useful behavior flows back out of. It’s the part your users actually touch; the model never does directly.

A model is a network in a box; the agent harness is the environment wrapped around it, the thing users interact with.

Orion sits between your application and your inference engine. You own the top (UI/app) and the bottom (the model runtime); Orion owns the middle - the orchestration that turns a prompt into a streamed, tool-augmented answer.

┌──────────────────────────────────────────────────┐
│  Your application (CLI, server, desktop app, …)   │
├──────────────────────────────────────────────────┤
│  Agent                                            │
│  ├── prompt("Hello")                              │
│  ├── Conversation state (Vec<Message>)            │
│  ├── System prompt                                │
│  ├── Registered tools                             │
│  └── AgentConfig (inference params, context cfg)  │
├──────────────────────────────────────────────────┤
│  Context pipeline                                 │
│  └── prepare_context() - prune + template format  │
├──────────────────────────────────────────────────┤
│  LlmBackend (trait) ← you implement this          │
│  ├── generate() - run inference, stream tokens    │
│  ├── tokenize_count() - count tokens              │
│  └── is_ready() - check model status              │
├──────────────────────────────────────────────────┤
│  Your inference engine                            │
│  (llama.cpp, MLX, ONNX, cloud API, etc.)          │
└──────────────────────────────────────────────────┘

The Orion stack from top to bottom: your app, the agent, the context pipeline, the LlmBackend trait, and your engine.

You write the top (your app) and the bottom (your engine). Orion is everything in between - and the only seam that matters is the LlmBackend trait.

The loop

A single call to Agent::prompt drives the whole cycle:

Append the user message to the conversation.
Prepare context - the pipeline prunes old messages to fit the token budget and formats the survivors into a prompt string using the active chat template (injecting tool schemas if any tools are registered).
Generate - the backend streams tokens, which surface as MessageDelta events in real time.
Tool loop - if the model emitted tool calls, the agent runs the matching tools, appends their results, and loops back to step 2. This repeats until the model returns a tool-free answer (bounded by AgentConfig::max_tool_iterations, default 8).
Finish - the final assistant message lands and the call returns.

Events flow upward

Every meaningful step emits an AgentEvent through an unbounded channel (tokio::sync::mpsc). Your UI or application layer subscribes and reacts in real time - streaming tokens to the screen, showing tool progress, or rendering the live context-budget gauge. The agent never touches your UI directly; it only emits events.

Backend-agnostic by design

Orion knows nothing about your model runtime. It calls three methods on the LlmBackend trait and orchestrates everything else. That keeps the harness identical whether you run a local GGUF model through llama.cpp, an MLX model on Apple silicon, or a remote OpenAI-compatible endpoint.