Skip to content
These docs cover Strata 0.9.0-preview.1.It is a preview, and the API can still change before 1.0.

Observe runs

You can see what a run does in four ways:

  • A stream of output chunks, such as a model’s tokens, while the run is going.
  • Run events, typed records of what the engine did, sent to an observer you register.
  • Spans and metrics, in OpenTelemetry form, from the engine itself.
  • Checkpoints, which stay in the store after the run, and which the strata CLI reads.

Events, spans and metrics come from the same hooks in the engine, so they agree on what happened. This guide shows how to use each one.

Install Strata as Installation describes. Most samples use an in-memory store and the review graph from demo 27 (.NET, Rust): Classify fails transiently once and is retried (see Retries and backoff), and the run pauses at the interrupt boundary on Approve (see Pause for approval).

Start the run with RunStreaming (C#) or run_streaming (Rust) instead of RunAsync / run. You get a reader of StreamChunks and a handle to the run’s result. A streaming node writes chunks while it runs. Here, Answer sends four tokens:

var graph = new StateGraph<Answer>()
.AddStreamingNode("Answer", async (Answer state, NodeContext ctx, CancellationToken ct) =>
{
var text = "";
foreach (var token in new[] { "Checkpoints ", "make ", "runs ", "durable." })
{
await ctx.Stream!.EmitTokenAsync(token, ct);
text += token;
}
return state with { Text = text };
})
.AddNode("Count", state => state with { Text = $"{state.Text} ({state.Text.Length} chars)" })
.AddEdge("Answer", "Count")
.Compile();
var (reader, completion) = graph.RunStreaming("answer-1", new Answer(""), new InMemoryCheckpointStore());
await foreach (var chunk in reader.ReadAllAsync())
{
Console.WriteLine(chunk.Type switch
{
StreamChunkType.Token => $" {chunk.NodeName}: \"{chunk.Payload}\"",
StreamChunkType.Retry or StreamChunkType.Error => $"{chunk.Type} {chunk.NodeName}: {chunk.Payload}",
_ => $"{chunk.Type} {chunk.NodeName}",
});
}
var result = await completion;
Console.WriteLine(result is GraphExecutionResult<Answer>.Completed done ? $"Completed: {done.State.Text}" : $"{result}");

An async streaming node finds the stream on NodeContext.Stream, and passes its cancellation token to EmitTokenAsync. A sync streaming node (StreamingNodeHandler<TState>) gets an IStreamContext and calls EmitToken.

Both print:

NodeStart Answer
Answer: "Checkpoints "
Answer: "make "
Answer: "runs "
Answer: "durable."
NodeEnd Answer
NodeStart Count
NodeEnd Count
Completed: Checkpoints make runs durable. (30 chars)

Reading the stream to the end shows what happened while the run went on. Awaiting the handle gives you the same result RunAsync / run would. The stream ends when the run ends, whatever the outcome.

These are the chunk kinds (StreamChunkType):

Kind Sent by Payload
NodeStart The engine, before every node, streaming or not.
NodeEnd The engine, after a node completes. A node that fails or is cancelled gets none.
Token Your node, with EmitTokenAsync / emit_token. A model node sends its answer this way. The text.
Retry The engine, when a fallible node’s attempt failed transiently and the node runs again. The failed attempt’s message.
Error The engine, when a fallible node fails for good. A node can also send one with EmitDiagnosticAsync / emit_diagnostic. The message.

When you see Retry, drop the tokens you kept for that node: the next attempt streams again from the start. Chunks from a subgraph’s child are named Work/Send, nested to any depth, and the streaming branches of a parallel group are named Gather/Acme. To stream a model’s answer, see Stream the answer.

To pass run options, use the overloads that take RunOptions (C#) or run_streaming_with with a RunConfig (Rust). A resume streams too: ResumeStreaming / resume_streaming start the stream at the node the resume enters, and say nothing about the nodes that ran before the pause.

A streaming node needs a stream to write to. A plain run has none, so it fails when it reaches one:

var plain = await graph.RunAsync("answer-2", new Answer(""), new InMemoryCheckpointStore());
if (plain is GraphExecutionResult<Answer>.Failed failed)
{
Console.WriteLine(failed.Message);
}
node 'Answer' is a streaming node but this run has no StreamContext — use RunStreaming instead of RunAsync

The same goes for a parallel group with a streaming branch, and a subgraph that streams. An ordinary async node can also write chunks through NodeContext.Stream / ctx.stream(). Under a plain run, that is null / None, so check for it.

The stream is a channel bounded at 1,024 chunks (StreamContext.DefaultChannelCapacity / DEFAULT_CHANNEL_CAPACITY). When it is full, the node’s next emit waits for a free slot. The node pauses; no chunk is dropped, and memory doesn’t grow. Demo 11 shows it with a node that emits 20,000 tokens: while the reader reads nothing, the node stops at 1,023 tokens (the engine’s NodeStart holds the 1,024th slot). Once the reader starts, all 20,000 arrive, in order. See demo 11, .NET and Rust.

A reader that stops reading leaves the node paused for good. To free it, cancel the run: pass a CancellationToken to RunStreaming (C#), or RunConfig::with_cancellation (Rust). An async node that passes its token to EmitTokenAsync stops at once, and the run ends Cancelled with a checkpoint you can resume. In Rust, dropping the receiver also works: the waiting emit_token returns Err(StrataError::ChannelClosed), so stop emitting when it does. A sync node blocked in EmitToken / emit_token_blocking can’t see a token, so the engine stops waiting for it and lets it go when the run ends.

A run event is a typed record of one thing the engine did: a node started, an attempt failed, a checkpoint was saved. Events exist in every run mode, streaming or not. Register an observer in RunOptions.Observer (C#) or with RunConfig::with_observer (Rust). For one run, use RunEventChannel, an observer whose reader ends after the run’s last event:

var events = new RunEventChannel();
var result = await graph.RunAsync("order-2001", Order.New(40), store, new RunOptions
{
Metadata = { ["tenant"] = "acme" },
Observer = events,
});
await foreach (var e in events.Reader.ReadAllAsync())
{
Console.WriteLine($"#{e.Sequence} {e.GetType().Name} {e.NodePath}{(e.Attempt is { } a ? $" #{a}" : "")}");
}

The events are records in Datalium.Strata.Observability that derive from RunEvent. A switch over them needs a default arm, because later versions add event types.

Both print:

#1 RunStarted
#2 NodeStarted Classify #1
#3 RetryScheduled Classify #1
#4 NodeCompleted Classify #2
#5 NodeStarted Approve #1
#6 NodeCompleted Approve #1
#7 CheckpointSaved
#8 Interrupted
#9 RunCompleted

Every event carries the thread ID, the run ID, a sequence number (1, 2, 3, … with no gaps), and a timestamp. Node events also carry the node path (Work/Send inside a subgraph) and the attempt. Run-level events have neither.

A RunEventChannel serves one run. An observer that outlives runs, for example one that forwards every run’s events to a queue, implements IRunObserver (C#) or RunObserver (Rust). It has one method, called once per event:

sealed class ConsoleObserver : IRunObserver
{
public void OnEvent(RunEvent runEvent)
{
var line = runEvent switch
{
NodeCompleted n => $"{runEvent.NodePath} {n.Result} in {n.Attempt} attempt(s)",
RetryScheduled r => $"{runEvent.NodePath} attempt {r.Attempt} failed: {r.Error}",
ModelCallCompleted m => $"{m.Model}: {m.Usage.InputTokens} in, {m.Usage.OutputTokens} out",
RunCompleted c => $"run {runEvent.RunId} {c.Outcome}",
_ => null, // later versions add event types
};
if (line is not null)
{
Console.WriteLine($"[{runEvent.ThreadId}] {line}");
}
}
}
await graph.RunAsync("order-2002", Order.New(40), store, new RunOptions { Observer = new ConsoleObserver() });
[order-2002] Classify attempt 1 failed: gateway timeout
[order-2002] Classify Committed in 2 attempt(s)
[order-2002] Approve Committed in 1 attempt(s)
[order-2002] run 01a18d45-64a0-79ac-afbd-b8d730dbdd45 Interrupted

How the engine delivers events:

  • In order, one at a time. The observer gets each run’s events in sequence order, never two at once for the same run. An observer you share between runs must be thread-safe.
  • Off the run’s path. Events go into an unbounded queue per run, and a separate task delivers them. A slow observer only makes its own queue grow; it never makes the run wait. In Rust, on_event runs on a tokio task, so it must not block: hand slow work to a channel.
  • Failures are counted, not raised. An exception (C#) or panic (Rust) from OnEvent / on_event is caught and counted in the strata.observer.errors metric, and delivery goes on. An observer can’t fail a run.
  • The run doesn’t wait for delivery. RunAsync / run_with can return before the observer has seen RunCompleted. If you need every event, wait for RunCompleted on the observer’s side, as RunEventChannel does.
  • Pairs always close. Every NodeStarted gets one NodeCompleted, and every BranchStarted one BranchCompleted, even when the node fails or is cancelled. RunStarted is always first and RunCompleted always last.
Event When
RunStarted First, every run. Carries the mode (Run, Resume, End, Recover), the node it starts at, the run’s metadata, the graph identity and the time spent waiting for a scheduler.
Resumed A resume, after the old checkpoint is stamped. Who resumed, whether they approved, and what woke the thread (a person, an event or a timer).
Recovered A recovery claimed a thread whose process died.
NodeStarted A node is about to run its first attempt.
RetryScheduled An attempt failed transiently and will run again: the error, whether it timed out, and the delay.
NodeCompleted The engine is done with a node: Committed, Failed, Interrupted or Cancelled, the time over every attempt, the error, and the fallback node if there is one.
BranchStarted, BranchCompleted A parallel group’s branch started, or ended (Succeeded, Failed, Cancelled, with a reason such as LostRace).
JoinDecided A group whose join isn’t All decided it.
MapExpanded A map read its items. Each item then reports as a branch, at <map>/<key>.
CheckpointSaved A checkpoint was written to your store: its status, where it waits, and how long the save took.
Interrupted The run paused at an interrupt boundary, and the checkpoint is saved.
HopLimitExceeded The hop limit stopped a loop.
LeaseAcquired, LeaseLost The run took its thread’s lease, or lost it and stopped. See Run multiple workers.
ToolCallStarted, ToolCallCompleted A tool registry dispatched a call from a node: the tool, the result and the duration.
ModelCallStarted, ModelCallCompleted A node called a model: the model, the provider, the token usage and the finish reason.
ModelRouted A call through a model route ended: which candidate answered, and what happened to each one.
RunCompleted Last, every run: Completed, Interrupted, Failed or Cancelled, the run’s duration, and the error or the cancel reason.

Each run gets an ID, a UUID v7 unless you set one with RunOptions.RunId / RunConfig::with_run_id. A resume is a new run with a new ID; the thread ID connects them. The run ID is on events, spans and the node’s context, but not on checkpoints.

Metadata is your own key/value pairs for a run, such as a tenant or a ticket, set with RunOptions.Metadata / RunConfig::with_metadata. The engine:

  • writes it on every checkpoint the run saves, at every subgraph level, beside its own keys;
  • puts it on RunStarted and on the run’s span;
  • gives it to every node, as NodeContext.Metadata / ctx.metadata().

A resume carries the stored metadata forward, and the resume’s own metadata overrides it key by key. A resume can add or change a key, but not remove one. The engine reserves error, error_kind, attempts, cancel_reason and every key that starts with strata.. A run that uses one, or an empty key, is refused before anything is written: C# throws ArgumentException, and Rust returns Failed with run metadata key '<key>' is reserved.

No event includes your state, a diff of it, a prompt, a model’s answer, or a tool’s arguments or result. Building a diff of an arbitrary state type means serializing it twice per node, and it would send your domain data, which can include personal data, to every telemetry system by default. To follow the state, read the checkpoints, as Inspect a thread afterwards shows.

Export traces and metrics with OpenTelemetry

Section titled “Export traces and metrics with OpenTelemetry”

The engine has one ActivitySource and one Meter, both named Datalium.Strata (StrataTelemetry.Name). Nothing is recorded until something listens. In a hosted app, AddInstrumentation() adds every Strata source and meter to your OpenTelemetry providers. See Send telemetry to OpenTelemetry. Without the host, add the names yourself:

using Datalium.Strata.Observability;
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;
using var tracing = Sdk.CreateTracerProviderBuilder()
.AddSource(StrataTelemetry.Name)
.AddConsoleExporter()
.Build();
using var metrics = Sdk.CreateMeterProviderBuilder()
.AddMeter(StrataTelemetry.Name)
.AddConsoleExporter()
.Build();

StrataTelemetry.Name covers the engine. Some other Strata packages, such as storage and hosting, have sources or meters of their own, and StrataInstrumentation.ActivitySourceNames and StrataInstrumentation.MeterNames, in Datalium.Strata.Hosting, list them all.

Run the review graph, and the first attempt of Classify exports as this span (part of the output):

Activity.ParentSpanId: f4324e820a09964f
Activity.DisplayName: Classify #1
Activity.Kind: Internal
Activity.Tags:
strata.thread_id: order-4001
strata.run_id: 01a18d45-b27c-75e3-8c96-a4598037128d
strata.node.path: Classify
strata.attempt: 1
error.type: transient
StatusCode: Error
Activity.StatusDescription: gateway timeout

A run makes a tree of spans: run, then node, then attempt. Your handler runs inside its attempt’s span, so a span your own code or an HTTP client opens nests under it. To add to the attempt’s span, use NodeContext.Activity (C#, null when nothing listens) or ctx.span() (Rust).

Span name Name a trace viewer shows Parent Main attributes
strata.run strata.run, strata.resume, strata.end or strata.recover Whatever span is current when you start the run. strata.thread_id, strata.run_id, strata.run.mode, the metadata, strata.outcome, strata.cancel_reason, strata.graph.id, strata.graph.version
strata.node The node path, such as Classify or Work/Send The run span, or the subgraph node’s attempt span. strata.node.path, strata.retry_count, strata.node.result
strata.attempt Classify #2 The node span. strata.node.path, strata.attempt, error.type
strata.branch The branch path, such as Gather/Acme The group’s attempt span. strata.node.path, strata.attempt
strata.tool execute_tool <tool> The calling node’s attempt span. gen_ai.tool.name, gen_ai.tool.call.id, strata.tool.result
strata.model chat <model>, or embeddings <model> The calling node’s attempt span. The gen_ai.* attributes below
strata.route route <name> The calling node’s attempt span. strata.route, strata.route.answered

Every span carries strata.thread_id and strata.run_id. A failed span has an error status and an error.type of transient, fatal or timeout (or the exception type, for a bug). A cancelled run, node or attempt doesn’t: a cancellation isn’t an error. The node span has a retry event for each retry, and the run span a checkpoint.saved event for each save.

The metadata is one attribute per key in C# (strata.metadata.tenant), and one JSON attribute in Rust (strata.metadata). A resume’s spans aren’t linked to the trace of the run that paused: each run has its own strata.run span, and the thread ID connects them.

The main metrics. Thread IDs, run IDs and metadata are never metric attributes, because they have no bound.

Metric Type Unit Attributes
strata.graphs.started Counter {run} strata.run.mode
strata.graphs.active Up-down counter (a gauge in Rust) {run}
strata.graph.duration Histogram s strata.run.mode, strata.outcome
strata.node.duration Histogram s strata.node.path, strata.node.result
strata.node.retries Counter {retry} strata.node.path, error.type
strata.graph.transition Counter {transition} strata.node.path, strata.next_node.path
strata.checkpoint.latency Histogram s strata.checkpoint.operation (save, load), strata.checkpoint.status
strata.observer.errors Counter {error}
strata.tool.requests Counter {request} gen_ai.tool.name, strata.tool.result
strata.tool.duration Histogram s gen_ai.tool.name, strata.tool.result
gen_ai.client.token.usage Histogram {token} gen_ai.token.type (input, output), the operation, provider and model
gen_ai.client.operation.duration Histogram s The same, plus error.type
strata.model.time_to_first_chunk Histogram s A streamed call’s time to its first update.

The same meter also has metrics for the hop limit (strata.graph.hop_limit_exceeded), waits (strata.thread.wakes), recovery (strata.thread.recoveries), leases (strata.lease.acquisitions, strata.lease.losses), the run scheduler (strata.scheduler.*) and model routes (strata.route.*, strata.model.breaker.transitions). In Rust, the names are constants in datalium_strata::observability::metric_names.

Every model call made through Strata’s AI package, or through a ModelCallScope in your own client, opens a strata.model span with the OpenTelemetry GenAI attributes: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.response.finish_reasons, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. It records the two gen_ai.client.* metrics and the ModelCallStarted / ModelCallCompleted events. Token counts are what the provider reported, and a count it didn’t report is left empty, never estimated.

Prompts and answers are never recorded, in spans, metrics or events. Neither are tool arguments or tool results.

Events and spans are not stored. If nothing listened during a run, they’re gone. Checkpoints stay in the store, with your metadata on them, and the strata CLI reads them from any SQLite, Postgres or Redis store. Find a tenant’s threads with --meta, then look at one:

Terminal window
strata thread list --storage checkpoints.sqlite --meta tenant=acme
strata thread inspect order-5001 --storage checkpoints.sqlite
+------------+--------------+-------------+-------------+-------------+----------------------+
| Thread ID | Current Node | Sequence ID | State | Waiting For | Updated At |
+------------+--------------+-------------+-------------+-------------+----------------------+
| order-5001 | Approve | 0 | interrupted | - | 2026-10-30T14:48:27Z |
+------------+--------------+-------------+-------------+-------------+----------------------+
Thread: order-5001
Status: interrupted
Thread state: interrupted
Current node: Approve
Metadata: tenant=acme, ticket=T-1001
Next node: Execute
Checkpoint count: 1
Created at: 2026-10-30T14:48:27Z
Resumed at: -
State:
{
"amount": 40,
"log": [
"classify",
"approve"
]
}

The Metadata line shows your keys only. For a failed thread, --json shows the engine’s keys too, such as metadata.error. strata checkpoint log lists every checkpoint of a thread, and strata checkpoint diff shows which state fields changed between two of them. From code, query threads by metadata as Querying threads shows.