Call a model

In this guide you add a node to a graph that asks a chat model a question and writes the answer into the state. You then stream the answer token by token, retry the call when the provider is busy, and point the same code at a local model.

The steps follow demo 06 (.NET, Rust), which runs the whole flow against a live model.

Before you start

• Set up a project as in Installation.
• Have a model to call: an OpenAI API key, or an OpenAI-compatible server on your machine, such as Ollama. See Use a local model.

Add the AI package

C#

Model calls are in Datalium.Strata.AI, which Datalium.Strata.Hosting already brings. It works with any IChatClient from Microsoft.Extensions.AI, so you add the provider package you want. This guide uses OpenAI’s:

dotnet add package Datalium.Strata.Hosting --version 0.9.0-preview.1
dotnet add package Microsoft.Extensions.AI.OpenAI --version 10.11.0

Rust

Model calls are in the datalium-strata-ai crate. It has its own ChatClient trait and two clients, each behind a feature that is on by default: openai for OpenAiChatClient and anthropic for AnthropicChatClient.

[dependencies]
datalium-strata = "=0.9.0-preview.1"
datalium-strata-ai = "=0.9.0-preview.1"
datalium-strata-memory = "=0.9.0-preview.1"

To build only one client, turn the defaults off: datalium-strata-ai = { version = "=0.9.0-preview.1", default-features = false, features = ["openai"] }.

Create a chat client

C#

Create the OpenAI SDK’s ChatClient and turn it into an IChatClient with AsIChatClient(). Endpoint points it at another OpenAI-compatible server. Leave it null for OpenAI itself.

using System.ClientModel;
using System.ClientModel.Primitives;
using Microsoft.Extensions.AI;
using OpenAI;

static IChatClient CreateOpenAIClient(string model, string? apiKey, string? baseUrl) =>
    new OpenAI.Chat.ChatClient(
        model,
        new ApiKeyCredential(apiKey ?? "not-needed"),
        new OpenAIClientOptions
        {
            RetryPolicy = new ClientRetryPolicy(maxRetries: 0),
            Endpoint = baseUrl is null ? null : new Uri(baseUrl),
        })
    .AsIChatClient();

ClientRetryPolicy(maxRetries: 0) turns the SDK’s own retries off, so the node’s recovery policy is the only thing that retries (see Handle provider faults). A local server that needs no key still gets a placeholder, because the SDK won’t send an empty credential.

Any other IChatClient works the same way, for example a provider’s own package or a fake one in tests.

Rust

Build a client from its options and keep it as an Arc<dyn ChatClient>:

use std::sync::Arc;

use datalium_strata_ai::{
    AnthropicChatClient, AnthropicOptions, ChatClient, OpenAiChatClient, OpenAiOptions,
};

// OpenAI
let openai: Arc<dyn ChatClient> = Arc::new(OpenAiChatClient::new(
    OpenAiOptions::new(api_key).with_model("gpt-4o-mini"),
));

// Any OpenAI-compatible server: Ollama, vLLM, LM Studio, ...
let local: Arc<dyn ChatClient> = Arc::new(OpenAiChatClient::new(
    OpenAiOptions::compatible("http://localhost:11434/v1").with_model("qwen2.5:3b"),
));

// Anthropic
let anthropic: Arc<dyn ChatClient> = Arc::new(AnthropicChatClient::new(
    AnthropicOptions::new(anthropic_key).with_model(anthropic_model),
));

OpenAiOptions::new(key) targets https://api.openai.com/v1. OpenAiOptions::compatible(base_url) targets another server, up to /v1, with no key. Add one with .with_api_key(key) if the server needs it. OpenAiOptions::ollama("http://localhost:11434") is a shortcut for Ollama. Both option types also have with_timeouts and with_header.

Neither client retries. A transient error carries the provider’s retry hint, and the node’s recovery policy retries.

Add a model node

A model node is a node that makes one model call. You give it two functions:

• prompt builds the messages from the state.
• apply writes the response into the state and returns the next state.

This graph from demo 06 asks the model a question, then counts the characters of the answer in a plain node:

C#

static CompiledGraph<LlmStreamState> BuildGraph(IChatClient client) =>
    new StateGraph<LlmStreamState>()
        .AddModelNode<LlmStreamState>(
            "AnswerQuestion",
            client,
            (state, _) =>
            [
                new(ChatRole.System, "You are a helpful assistant. Answer in two or three sentences."),
                new(ChatRole.User, state.Question),
            ],
            (state, response) => state with { Answer = response.Text },
            new ModelNodeOptions { Chat = new ChatOptions { MaxOutputTokens = 200 } })
        .SetRecoveryPolicy("AnswerQuestion", RecoveryPolicy.Retry(2).WithBackoff(TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(10)))
        .AddNode("SummarizeAnswer", state => state with { AnswerCharCount = state.Answer.Length })
        .AddEdge("AnswerQuestion", "SummarizeAnswer")
        .Compile();

sealed record LlmStreamState
{
    public required string Question { get; init; }
    public string Answer { get; init; } = string.Empty;
    public int AnswerCharCount { get; init; }
}

prompt gets the state and the NodeContext and returns ChatMessages. apply gets the state and the ChatResponse. ModelNodeOptions.Chat is copied into every call, so you set the model (ModelId), Temperature or MaxOutputTokens there.

Rust

add_model_node comes from the StateGraphAiExt trait:

use datalium_strata::graph::{CompiledGraph, Jitter, RecoveryPolicy, StateGraph};
use datalium_strata_ai::{ChatClient, Message, ModelNodeOptions, StateGraphAiExt};

fn build_graph(client: Arc<dyn ChatClient>) -> Arc<CompiledGraph<LlmStreamState>> {
    let graph = StateGraph::<LlmStreamState>::new()
        .add_model_node(
            "AnswerQuestion",
            client,
            |state: &LlmStreamState, _| {
                vec![
                    Message::system(
                        "You are a helpful assistant. Answer in two or three sentences.",
                    ),
                    Message::user(state.question.clone()),
                ]
            },
            |state: LlmStreamState, response| LlmStreamState {
                answer: response.text(),
                ..state
            },
            ModelNodeOptions::new().with_max_tokens(200),
        )
        .set_recovery_policy(
            "AnswerQuestion",
            RecoveryPolicy::retry(2).with_backoff(
                Duration::from_secs(1),
                Duration::from_secs(10),
                Jitter::Full,
            ),
        )
        .add_node("SummarizeAnswer", |state: LlmStreamState| LlmStreamState {
            answer_char_count: state.answer.chars().count(),
            ..state
        })
        .add_edge("AnswerQuestion", "SummarizeAnswer")
        .compile()
        .expect("graph is structurally valid");
    Arc::new(graph)
}

prompt gets &S and the NodeContext and returns a Vec<Message>. apply gets the state and the ChatResponse. ModelNodeOptions sets the model (with_model, else the client’s default), with_max_tokens and with_temperature.

A model node is a fallible node (see Graphs), so a recovery policy can retry it. Run the graph as usual, with RunAsync or run. The node makes one request and waits for the whole answer.

Make the call from your own node

The model node is a shortcut. When a node needs more than one call, or other work around the call, call the model yourself with StrataChat.GetResponseAsync (C#) or complete (Rust). Pass the node’s context, and the call streams, records and fails exactly as it does in a model node:

C#

.AddFallibleNode("AnswerQuestion", async (state, context, cancellationToken) =>
{
    var response = await StrataChat.GetResponseAsync(
        client,
        [new(ChatRole.User, state.Question)],
        new ChatOptions { MaxOutputTokens = 200 },
        context,
        cancellationToken: cancellationToken);
    return state with { Answer = response.Text };
})

Rust

.add_async_fallible_node("AnswerQuestion", move |state: LlmStreamState, ctx| {
    let client = client.clone();
    async move {
        let mut request = ChatRequest::new(vec![Message::user(state.question.clone())]);
        request.max_tokens = Some(200);
        let response = complete(&*client, request, &ctx).await?;
        Ok(LlmStreamState { answer: response.text(), ..state })
    }
})

Stream the answer

Run the graph with RunStreaming (C#) or run_streaming (Rust) instead. Because the run streams, the model call streams too, and each piece of text reaches you as a Token chunk while the node runs. You don’t change the node. The node still gets the whole answer back as one response, and joined in order the Token chunks are exactly that answer.

C#

var (reader, completion) = BuildGraph(client).RunStreaming(
    "stream-thread", new LlmStreamState { Question = "What is a borrow checker?" }, new InMemoryCheckpointStore());

await foreach (var chunk in reader.ReadAllAsync())
{
    switch (chunk.Type)
    {
        case StreamChunkType.Token:
            Console.Write(chunk.Payload.ToString());
            break;
        case StreamChunkType.Retry:
            Console.WriteLine($"\n[retry] {chunk.NodeName}");
            break;
        case StreamChunkType.Error:
            Console.WriteLine($"\n[error] {chunk.NodeName}: {chunk.Payload}");
            break;
    }
}

var result = await completion;
if (result is GraphExecutionResult<LlmStreamState>.Completed done)
{
    Console.WriteLine($"\n\n{done.State.AnswerCharCount} characters");
}

RunStreaming returns a ChannelReader<StreamChunk> and a task. Reading the channel to the end shows what happened during the run. Awaiting the task gives you the same result RunAsync would.

Rust

let (handle, mut chunks) = graph.run_streaming("stream-thread", initial, store);

while let Some(chunk) = chunks.recv().await {
    match chunk.chunk_type {
        StreamChunkType::Token => print!("{}", chunk.content),
        StreamChunkType::Retry => println!("\n[retry] {}", chunk.node_name),
        StreamChunkType::Error => println!("\n[error] {}: {}", chunk.node_name, chunk.content),
        _ => {}
    }
}

match handle.await.expect("the run's task doesn't panic") {
    GraphExecutionResult::Completed(state) => {
        println!("\n\n{} characters", state.answer_char_count)
    }
    // ...
}

run_streaming takes an Arc of the compiled graph and returns a task handle and a receiver of StreamChunks. Reading the receiver to the end shows what happened during the run. Awaiting the handle gives you the same result run would.

Every node also sends NodeStart and NodeEnd chunks, streaming or not. This is demo 06 against a local model, with those chunks printed too. The wording varies from run to run:

[node start] AnswerQuestion
Rust's borrow checker is unique compared to garbage collection in that it operates entirely at compile time rather than runtime, ...
[node end]   AnswerQuestion
[node start] SummarizeAnswer
[node end]   SummarizeAnswer

When an attempt fails transiently and the node is retried, the stream carries a Retry chunk, and the new attempt streams its answer again from the start. Drop the tokens you collected when you see one.

Handle provider faults

A failed model call fails the node, as transient or fatal. A recovery policy retries transient failures and doesn’t retry fatal ones.

The call	Failure
Got 408, 409, 429 or any 5xx (Anthropic’s 529 included)	Transient, with the provider’s retry hint
Timed out, or lost its connection	Transient
Broke off after the answer started streaming	Transient
Got any other 4xx: a bad request, a wrong key, an unknown model	Fatal

The retry hint is the response’s retry-after-ms header, else its Retry-After header, in seconds or as an HTTP date. A policy with a backoff, like the one in the graph above, waits the hint before the retry, capped at the backoff’s maximum, instead of its own delay. A policy without a backoff retries at once and ignores the hint.

When a fatal failure, or the last transient one, isn’t recovered, the run ends failed, or goes to the policy’s fallback node if it has one. Demo 06 asks for a model that doesn’t exist. The provider answers 404, so the node fails after one attempt and the policy’s two retries go unused. In .NET, against OpenAI, it prints:

node failed: AnswerQuestion, after 1 attempt(s)
message: model call failed (404): HTTP 404 (invalid_request_error: model_not_found)
chunks the consumer saw: NodeStart, Error

C#

ModelFaults reads the exceptions it knows: HttpRequestException, ClientResultException (what the OpenAI and Azure SDKs throw), timeouts, and ModelCallException. Anything else is fatal. For an SDK that throws its own exception type:

• Implement IModelFaultClassifier and set it as ModelNodeOptions.FaultClassifier.
• Or, for an SDK built on HttpClient, add RetryAfterHandler to its handler chain. It turns a transient status into a ModelCallException with the retry hint, which the SDK would otherwise drop.

Turn the SDK’s own retries off in both cases.

Rust

The built-in clients return a ChatError with a kind (Transient or Fatal), the status and the retry hint, and the call turns it into a NodeError. A client you write yourself does the same: build its errors with ChatError::status(status, retry_after, message), which picks the kind from the status, or with ChatError::transient and ChatError::fatal. An answer the client can’t read is fatal.

Use a local model

Any server that speaks OpenAI’s Chat Completions API works with the OpenAI client: Ollama, vLLM, LM Studio and others. Give the client the server’s address up to /v1, and a model the server has. With Ollama:

ollama pull qwen2.5:3b

Then use http://localhost:11434/v1 as the endpoint (C#) or OpenAiOptions::compatible("http://localhost:11434/v1") (Rust), and qwen2.5:3b as the model. If you go on to give the model tools, pick a model that supports tool calls.

The demos read OPENAI_BASE_URL, OPENAI_MODEL and OPENAI_API_KEY from the environment to choose between OpenAI and a local server. That is the demos’ own convention, not Strata’s: Strata reads no environment variables for model calls, so you pass these values to the client yourself.

What a call records

Every call, in a model node or through StrataChat / complete, records ModelCallStarted and ModelCallCompleted run events with the model, the token usage, the finish reason and the duration. It also opens a chat <model> span and records the gen_ai.client.token.usage and gen_ai.client.operation.duration metrics. Prompts and answers are never recorded.

Related

• Build an agent: run a model in a loop with tools and a typed answer.
• Define tools: the tools an agent calls.
• Graphs: fallible nodes, recovery policies and fallbacks.
• Demo 06, token streaming: .NET, Rust.
• Demo 03, a live model calling a tool: .NET, Rust.