Skip to content
Orion Orion Orion

Orion

Orion sits between any backend (llama.cpp, MLX, cloud API) and your app, turning a prompt into a streamed, tool-augmented answer.

Agent harness for local LLM inference. Backend-agnostic - bring your own model runtime (llama.cpp, MLX, cloud APIs, anything).

Orion handles the conversation loop so you don’t have to: context management, token budgets, streaming events, chat formatting, and an automatic tool-execution loop. You implement one trait - LlmBackend - for your inference engine, and Orion handles everything above it.

How it works

Section titled “How it works”
User sends "Hello"
  → Agent.prompt("Hello")
    → Context pipeline (prune old messages to fit token budget)
      → Format prompt (ChatML template + tool definitions)
        → LlmBackend.generate() (streams tokens one by one)
          → AgentEvent stream (your UI subscribes here)
            → If the model called tools: run them, append results, loop back
              → Done (model returns a tool-free answer)
Getting started Install the crate, wire up a backend, and run your first prompt.
Implement a backend The one trait you write: generate, tokenize_count, is_ready.
Give the model tools Define tools and let the agent run the call → execute → reply loop.
Context & budgets Pruning strategies, pinned messages, and token accounting.

What you get

Section titled “What you get”

One trait to implement

Plug in any engine by implementing LlmBackend. It runs on a blocking thread - no async required. Orion owns the async orchestration.

Streaming events

Subscribe to an AgentEvent stream for tokens, tool execution, and context-budget updates - everything a reactive UI needs.

Automatic tool loop

The agent parses tool calls, runs your tools, feeds results back, and loops until the model returns a final answer.

Context management

Prune to a token budget with a sliding window or summarization, pin messages that must survive, and format with the right chat template.