Route between models

In this guide you send a node’s model requests to more than one model. You build a route over an ordered list of models. It moves on to the next model when one fails, refuses or gives an unusable answer. You then start a second model when the first is slow, skip models by your own rules, and skip a provider that keeps failing.

A model route is a chat client over ordered candidates: ModelRoute in both languages. It goes anywhere a chat client goes, so a model node or an agent uses it without changes. Each attempt is a model call of its own, and one run event says which candidate answered.

The code follows demo 24, model-routes (.NET, Rust). Every model in it is a scripted stand-in, so it runs without an API key.

Before you start

• Set up a project as in Installation. Routes are in Datalium.Strata.AI (which Datalium.Strata.Hosting brings) and in the datalium-strata-ai crate.
• Know how to call one model from a node. See Call a model. A route builds on its fault handling.
• Have a chat client per provider. The examples use clients named northwind, contoso, fabrikam and local. Any IChatClient (C#) or Arc<dyn ChatClient> (Rust) works, including the scripted ones in Test a route without a model.

Build a route

Give the route a name and its candidates in the order to ask them. Each candidate has a name, unique within the route, and a client. It can also name the model to ask for, which replaces the request’s model for that candidate. Accept is optional. It judges each answer and returns a reason to reject it.

C#

using Datalium.Strata.AI;
using Microsoft.Extensions.AI;

var triage = new ModelRoute(new ModelRouteOptions
{
    Name = "triage",
    Candidates =
    [
        new ModelCandidate("primary", northwind, model: "nw-large"),
        new ModelCandidate("careful", contoso, model: "ct-guarded"),
        new ModelCandidate("terse", fabrikam, model: "fb-mini"),
        new ModelCandidate("local", local, model: "llama-local"),
    ],
    // Your rule for a usable answer: here, it must give a reason.
    Accept = response => response.Text.Contains("because", StringComparison.OrdinalIgnoreCase) ? null : "it gives no reason",
});

ModelRoute takes a ModelRouteOptions. Name and Candidates are required, and the constructor refuses an empty list or two candidates with one name. Accept returns null to accept an answer.

Rust

use datalium_strata_ai::{ModelCandidate, ModelRoute};

let triage = ModelRoute::new("triage")
    .candidate(ModelCandidate::new("primary", northwind).with_model("nw-large"))
    .candidate(ModelCandidate::new("careful", contoso).with_model("ct-guarded"))
    .candidate(ModelCandidate::new("terse", fabrikam).with_model("fb-mini"))
    .candidate(ModelCandidate::new("local", local).with_model("llama-local"))
    // Your rule for a usable answer: here, it must give a reason.
    .with_accept(|response| {
        if response.text().to_lowercase().contains("because") {
            Ok(())
        } else {
            Err("it gives no reason".to_string())
        }
    });

ModelRoute::new(name) starts an empty route, and .candidate() adds the next one. It panics on a second candidate with the same name. with_accept returns Ok(()) to accept an answer. A route is cheap to clone, and clones share everything.

Set the model on each candidate rather than on the node or agent. Different providers rarely share model names.

Use it where a chat client goes

ModelRoute is an IChatClient (C#) and a ChatClient (Rust). Pass it to a model node, to StrataChat / complete, or to an agent, in place of a single client:

C#

var graph = new StateGraph<Ticket>()
    .AddModelNode<Ticket>("Ask", triage,
        (ticket, _) => [new(ChatRole.User, ticket.Question)],
        (ticket, response) => ticket with { Answer = response.Text })
    .Compile();

// An agent takes a route the same way.
var agent = new StrataAgent<RootCause>(route, tools, new StrataAgentOptions<RootCause> { Id = "rca-analyst" });

Rust

let graph = StateGraph::<Ticket>::new()
    .add_model_node(
        "Ask",
        Arc::new(triage),
        |t: &Ticket, _| vec![Message::user(t.question.clone())],
        |t: Ticket, r: ChatResponse| Ticket { answer: r.text(), ..t },
        ModelNodeOptions::new(),
    )
    .compile()
    .expect("the graph is valid");

// An agent takes a route the same way.
let agent = Arc::new(Agent::<RootCause>::new(Arc::new(route), registry, AgentOptions::new("rca-analyst")));

An agent makes one model call per turn, so any turn of its tool loop can fail over. See Build an agent.

Pass the route itself. A route wrapped inside another chat client is called as that client, as one model call, and its attempts aren’t recorded one by one.

See which candidate answered

Each call through a route records a ModelRouted run event. It names the route, the candidate that answered (Answered) and its model, how many hedges started, and one ModelRouteAttempt per candidate in the route’s order, with its outcome and a reason. Read it from the run’s observer:

C#

var events = new RunEventChannel();
var result = await graph.RunAsync(thread, ticket, store, new RunOptions { Observer = events });

await foreach (var runEvent in events.Reader.ReadAllAsync())
{
    if (runEvent is ModelRouted routed)
    {
        Console.WriteLine($"route '{routed.Route}': answered by {routed.Answered ?? "nobody"} ({routed.Model}), hedges {routed.Hedges}");
        foreach (var attempt in routed.Attempts)
        {
            Console.WriteLine($"  {attempt.Candidate,-9} {attempt.Outcome,-12} {attempt.Reason}".TrimEnd());
        }
    }
}

Rust

let (observer, mut events) = RunEventChannel::new();
let result = graph.run_with(thread, ticket, store, RunConfig::new().with_observer(observer)).await;

while let Some(event) = events.recv().await {
    if let RunEventKind::ModelRouted { route, answered, model, attempts, hedges, .. } = &event.kind {
        println!(
            "route '{route}': answered by {} ({}), hedges {hedges}",
            answered.as_deref().unwrap_or("nobody"),
            model.as_deref().unwrap_or_default()
        );
        for attempt in attempts {
            let line = format!(
                "  {:<9} {:<12} {}",
                attempt.candidate,
                attempt.outcome.as_str(),
                attempt.reason.as_deref().unwrap_or_default()
            );
            println!("{}", line.trim_end());
        }
    }
}

Run the triage graph with candidates scripted to fail with a 429, refuse, give an answer without a reason, and answer well. The C# run also prints each model-call event:

ModelCallStarted   nw-large     route=triage
ModelCallCompleted nw-large     Failed
ModelCallStarted   ct-guarded   route=triage
ModelCallCompleted ct-guarded   Succeeded (content_filter)
ModelCallStarted   fb-mini      route=triage
ModelCallCompleted fb-mini      Succeeded (stop)
ModelCallStarted   llama-local  route=triage
ModelCallCompleted llama-local  Succeeded (stop)
route 'triage': answered by local (llama-local), hedges 0
  primary   Failed       429
  careful   Refused
  terse     Rejected     it gives no reason
  local     Answered
Answer: Roll back deploy 412, because the error rate rose with it.

Rust prints the same, with the outcomes in snake case (failed, not_tried). Each attempt has its own ModelCallStarted and ModelCallCompleted, with its model, token usage and the route’s name. Each ModelRouteAttempt has a CallId that pairs it with those events. A candidate that wasn’t asked has none.

The outcomes are Answered, Failed, Refused, Rejected, Ineligible, CoolingDown, BreakerOpen, LostRace, NotTried and Cancelled. The reason holds the error type (429, connection), your Accept or eligibility reason, or the seconds left for a skipped candidate.

In telemetry, the route opens a route <name> span. Each attempt’s model-call span sits under it, tagged with strata.route and strata.route.candidate. The metrics are strata.route.calls (by strata.outcome), strata.route.failovers (by strata.route.reason), strata.route.hedges and strata.model.breaker.transitions.

Fail over on faults

The route asks one candidate at a time and moves on after:

• A transient fault, as Call a model classifies it: a 408, 409, 429 or 5xx, a timeout or a lost connection.
• A refusal: an answer with the finish reason content_filter.
• An answer Accept rejects.

A fatal fault, such as a 400, 401 or 404, stops the route at once. The request or the configuration is wrong, and another model would hide it. The other candidates are NotTried, and the node fails with the candidate’s own error, so a recovery policy doesn’t retry it:

route 'fatal': answered by nobody (), hedges 0
  primary   Failed       401
  backup    NotTried
Node 'Ask' failed after 1 attempt(s): model call failed (401): 401 Unauthorized: invalid API key

To treat one provider’s fault differently, set the route’s FaultClassifier (an IModelFaultClassifier), or with_fault_classifier in Rust, which gets the candidate and its ChatError and can return another ChatErrorKind.

A route never asks a candidate twice in one call. When no candidate answers, the route fails transiently with the error type route_exhausted. Its retry hint is the shortest wait until some candidate can be asked again. The node’s recovery policy retries the whole route and, with a backoff, waits that hint. See Retry with backoff. Demo 24, scenario 6, shows it: two candidates fail with Retry-After 8 s and 3 s, and the retry waits 3 s.

Hedge a slow candidate

With HedgeAfter (Rust: with_hedge_after), a candidate that hasn’t answered within the delay keeps running, and the next available candidate starts beside it. After another delay the next one starts, and so on, each candidate at most once. Hedging is off by default.

C#

var hedged = new ModelRoute(new ModelRouteOptions
{
    Name = "hedged",
    Candidates =
    [
        new ModelCandidate("primary", northwind, model: "nw-large"),
        new ModelCandidate("local", local, model: "llama-local"),
    ],
    HedgeAfter = TimeSpan.FromSeconds(2),
});

Rust

let hedged = ModelRoute::new("hedged")
    .candidate(ModelCandidate::new("primary", northwind).with_model("nw-large"))
    .candidate(ModelCandidate::new("local", local).with_model("llama-local"))
    .with_hedge_after(Duration::from_secs(2));

The first accepted answer wins. Every other attempt is cancelled and recorded LostRace. Here primary hangs, so local starts after 2 seconds and wins:

route 'hedged': answered by local (llama-local), hedges 1
  primary   LostRace
  local     Answered

A hedge costs the tokens of every attempt that started. A losing or cancelled attempt doesn’t count against the provider’s health. In .NET the winner doesn’t wait for the losers to stop, so a loser’s ModelCallCompleted can arrive after the run has ended and be dropped. Read the losers from ModelRouted instead. Rust drops a loser before the route returns.

Choose candidates per request

Eligible (Rust: with_eligible) is your rule for whether a candidate may take a request, such as a tenant’s plan, a budget or the prompt’s size. It is asked when the candidate’s turn comes. It gets the messages, the options and the node’s context (Rust: the ChatRequest and the context), and returns a reason to skip the candidate. The context is null (Rust: None) outside a node.

C#

var plans = new ModelRoute(new ModelRouteOptions
{
    Name = "plans",
    Candidates =
    [
        new ModelCandidate("frontier", northwind, model: "nw-large")
        {
            // A tenant rule: the run's metadata says the customer's plan.
            Eligible = (_, _, ctx) => ctx?.Metadata.GetValueOrDefault("plan") == "free"
                ? "the free plan doesn't use frontier models"
                : null,
        },
        new ModelCandidate("local", local, model: "llama-local")
        {
            // A context-size rule: the local model takes short prompts only.
            Eligible = (messages, _, _) => messages.Sum(m => m.Text.Length) > 4_000
                ? "the prompt is too long for its context"
                : null,
        },
    ],
});

Rust

let plans = ModelRoute::new("plans")
    .candidate(
        ModelCandidate::new("frontier", northwind)
            .with_model("nw-large")
            // A tenant rule: the run's metadata says the customer's plan.
            .with_eligible(|_, ctx| {
                let plan = ctx.and_then(|ctx| ctx.metadata().get("plan"));
                if plan.map(String::as_str) == Some("free") {
                    Err("the free plan doesn't use frontier models".to_string())
                } else {
                    Ok(())
                }
            }),
    )
    .candidate(
        ModelCandidate::new("local", local)
            .with_model("llama-local")
            // A context-size rule: the local model takes short prompts only.
            .with_eligible(|request, _| {
                let length: usize = request.messages.iter().map(|m| m.text().len()).sum();
                if length > 4_000 {
                    Err("the prompt is too long for its context".to_string())
                } else {
                    Ok(())
                }
            }),
    );

An ineligible candidate isn’t asked. When no candidate is eligible, the route fails fatally with route_ineligible, since a retry would get the same answer. Three runs with plan set in the run’s metadata print:

plan=paid
route 'plans': answered by frontier (nw-large), hedges 0
  frontier  Answered
  local     NotTried
plan=free
route 'plans': answered by local (llama-local), hedges 0
  frontier  Ineligible   the free plan doesn't use frontier models
  local     Answered
plan=free, long prompt
route 'plans': answered by nobody (), hedges 0
  frontier  Ineligible   the free plan doesn't use frontier models
  local     Ineligible   the prompt is too long for its context
Node 'Ask' failed after 1 attempt(s): model call failed (route_ineligible): no candidate of route 'plans' can take this request: frontier is ineligible (the free plan doesn't use frontier models), local is ineligible (the prompt is too long for its context)

Skip a provider that keeps failing

A ModelHealth keeps each provider’s health and skips its candidates while it is unhealthy. It does two things:

• Cooldown. When a transient fault carries a Retry-After, the provider is CoolingDown until it passes, at most 24 hours. This is always on.
• Breaker. With a ModelBreaker(Failures, OpenFor) (Rust: with_breaker(failures, open_for)), that many transient failures in a row open the breaker for OpenFor. Then it is half-open: one test call goes through while other calls still skip it. A success closes it, and a failure opens it again for OpenFor. There is no breaker by default.

Only transient faults count. A refusal, a rejected answer, a lost race and a cancellation leave health alone, and any success resets the count to 0.

Health is kept per health key: the client’s provider name by default (ChatClientMetadata.ProviderName, Rust provider()), else the candidate’s name. A provider’s outage then skips all its models. Set HealthKey (Rust: with_health_key) to keep models or regions apart.

Each route has a private ModelHealth with no breaker unless you give it one. Share one between routes so they learn from each other:

C#

var health = new ModelHealth(new ModelHealthOptions
{
    Breaker = new ModelBreaker(Failures: 2, OpenFor: TimeSpan.FromSeconds(60)),
});

var diagnose = new ModelRoute(new ModelRouteOptions
{
    Name = "diagnose",
    Candidates = [new ModelCandidate("large", large), new ModelCandidate("steady", steady)],
    Health = health,
});
var summarise = new ModelRoute(new ModelRouteOptions
{
    Name = "summarise",
    Candidates = [new ModelCandidate("small", small), new ModelCandidate("steady", steady)],
    Health = health,
});

ModelHealthOptions.TimeProvider sets the clock for the cooldowns, the breaker and the routes’ hedging. It defaults to the system clock.

Rust

let health = ModelHealth::new(ModelHealthOptions::new().with_breaker(2, Duration::from_secs(60)));

let diagnose = ModelRoute::new("diagnose")
    .candidate(ModelCandidate::new("large", large))
    .candidate(ModelCandidate::new("steady", steady.clone()))
    .with_health(health.clone());
let summarise = ModelRoute::new("summarise")
    .candidate(ModelCandidate::new("small", small))
    .candidate(ModelCandidate::new("steady", steady))
    .with_health(health.clone());

Clones of a ModelHealth share their state. Its clock is tokio’s, so tests can pause it.

Here large and small are two models of one provider, northwind. A graph runs diagnose, then summarise. In the first run each fails once with a 503, which is two failures in a row on one key, so the breaker opens. The second run skips northwind in both routes without calling it. After 60 seconds the key is half-open, and the third run’s test call succeeds and closes it:

Run 2
route 'diagnose': answered by steady (llama-local), hedges 0
  large     BreakerOpen  60 s
  steady    Answered
route 'summarise': answered by steady (llama-local), hedges 0
  small     BreakerOpen  60 s
  steady    Answered
Answer: Roll back deploy 412, because the error rate rose with it. / Summary: Deploy 412 broke checkout.
(+60 s)
Health: local=Available(0), northwind=HalfOpen(2)
Run 3
route 'diagnose': answered by large (nw-large), hedges 0
  large     Answered
  steady    NotTried

Snapshot() (Rust: snapshot()) lists each key’s state, failures in a row and, for a cooldown or an open breaker, when it ends. Use it for your own health check or dashboard. Strata adds no health check for it. Health lives in memory in one process.

Streams

When the run streams, the route streams too. It fails over only until a stream’s first update. After that it has committed, and a break fails the call as stream_broken, as with a single model. With hedging, the first stream to produce an update wins. Accept and the refusal check run when the stream ends, and a rejected or refused stream fails the call transiently, because its tokens have already gone out. See Stream the answer.

Test a route without a model

Use scripted clients as the candidates, as demo 24 does. Its FakeModel plays one scripted turn per call. Each turn answers, refuses, hangs until cancelled, or fails. It names its provider and model, so the default health key works as it does with a real client.

Script a fault with the types the route classifies:

C#

public static Turn Fail(int status, TimeSpan? retryAfter = null, string? message = null) => new()
{
    Failure = new ModelCallException(
        message ?? $"{status} from the provider",
        isTransient: ModelFaults.IsTransientStatus(status),
        statusCode: status,
        retryAfter: retryAfter),
};

// In FakeModel: the provider name is the default health key.
public object? GetService(Type serviceType, object? serviceKey = null) =>
    serviceType == typeof(ChatClientMetadata) ? new ChatClientMetadata(provider, null, model) : null;

A refusal is a ChatResponse with FinishReason = ChatFinishReason.ContentFilter.

Rust

fn fail(status: u16, retry_after: Option<Duration>, message: Option<&str>) -> Self {
    let message = message.map_or_else(|| format!("{status} from the provider"), str::to_string);
    Self { failure: Some(ChatError::status(status, retry_after, message)), ..Self::default() }
}

// In `impl ChatClient for FakeModel`: the provider is the default health key.
fn provider(&self) -> &str {
    self.provider
}

A refusal is a ChatResponse whose finish_reason is Some("content_filter").

To test a route on its own, call it directly and read the outcome from OnRouted (Rust: on_routed). It is called once per call, inside a run or not, with what ModelRouted carries. It must not block, and what it throws is ignored.

C#

var primary = new FakeModel("northwind", "nw-large", Turn.Fail(503));
var backup = new FakeModel("local", "llama-local", Turn.Answer(RollBack));
ModelRouteResult? routed = null;
var route = new ModelRoute(new ModelRouteOptions
{
    Name = "triage",
    Candidates = [new ModelCandidate("primary", primary), new ModelCandidate("backup", backup)],
    OnRouted = result => routed = result,
});

var response = await route.GetResponseAsync([new(ChatRole.User, "Why is checkout failing?")]);

Console.WriteLine(response.Text);
Console.WriteLine(string.Join(", ", routed!.Attempts.Select(a => $"{a.Candidate}={a.Outcome}")));

Rust

let primary = FakeModel::new("northwind", "nw-large", vec![Turn::fail(503, None, None)]);
let backup = FakeModel::new("local", "llama-local", vec![Turn::answer(ROLL_BACK)]);
let routed: Arc<Mutex<Option<ModelRouteResult>>> = Arc::default();
let seen = routed.clone();
let route = ModelRoute::new("triage")
    .candidate(ModelCandidate::new("primary", primary))
    .candidate(ModelCandidate::new("backup", backup))
    .on_routed(move |result| *seen.lock().unwrap() = Some(result.clone()));

let response = route
    .complete(ChatRequest::new(vec![Message::user("Why is checkout failing?")]))
    .await
    .expect("backup answers");

println!("{}", response.text());
let routed = routed.lock().unwrap().take().expect("on_routed was called");
let outcomes: Vec<String> =
    routed.attempts.iter().map(|a| format!("{}={}", a.candidate, a.outcome.as_str())).collect();
println!("{}", outcomes.join(", "));

Roll back deploy 412, because the error rate rose with it.
primary=Failed, backup=Answered

Outside a run there are no run events, but the attempts still open spans and record metrics. When no candidate answers, C# throws a ModelRouteException with IsTransient, RetryAfter, ErrorType and the Attempts. Rust returns a ChatError with the error type route_exhausted or route_ineligible, and the attempts come from on_routed.

For time, use a fake clock: ModelHealthOptions.TimeProvider with a FakeTimeProvider in C#, or tokio’s paused clock in Rust. The breaker’s open time and the hedge delay then pass without waiting. See Test without waiting.

Related

• Call a model: model nodes, streaming and how faults are classified.
• Build an agent: a tool loop over a chat client or a route.
• Retries and backoff: the recovery policy that retries a route.
• Graphs: fallible nodes and recovery policies.
• Demo 24, model routes: .NET, Rust.
• The design: design/46-Model-Routes.md.