Observe runs
You can see what a run does in four ways:
- A stream of output chunks, such as a model’s tokens, while the run is going.
- Run events, typed records of what the engine did, sent to an observer you register.
- Spans and metrics, in OpenTelemetry form, from the engine itself.
- Checkpoints, which stay in the store after the run, and which the
strataCLI reads.
Events, spans and metrics come from the same hooks in the engine, so they agree on what happened. This guide shows how to use each one.
Before you start
Section titled “Before you start”Install Strata as Installation describes. Most samples use an in-memory store and the review graph from demo 27 (.NET, Rust): Classify fails transiently once and is retried (see Retries and backoff), and the run pauses at the interrupt boundary on Approve (see Pause for approval).
Stream a run
Section titled “Stream a run”Start the run with RunStreaming (C#) or run_streaming (Rust) instead of RunAsync / run. You get a reader of StreamChunks and a handle to the run’s result. A streaming node writes chunks while it runs. Here, Answer sends four tokens:
var graph = new StateGraph<Answer>() .AddStreamingNode("Answer", async (Answer state, NodeContext ctx, CancellationToken ct) => { var text = ""; foreach (var token in new[] { "Checkpoints ", "make ", "runs ", "durable." }) { await ctx.Stream!.EmitTokenAsync(token, ct); text += token; }
return state with { Text = text }; }) .AddNode("Count", state => state with { Text = $"{state.Text} ({state.Text.Length} chars)" }) .AddEdge("Answer", "Count") .Compile();
var (reader, completion) = graph.RunStreaming("answer-1", new Answer(""), new InMemoryCheckpointStore());
await foreach (var chunk in reader.ReadAllAsync()){ Console.WriteLine(chunk.Type switch { StreamChunkType.Token => $" {chunk.NodeName}: \"{chunk.Payload}\"", StreamChunkType.Retry or StreamChunkType.Error => $"{chunk.Type} {chunk.NodeName}: {chunk.Payload}", _ => $"{chunk.Type} {chunk.NodeName}", });}
var result = await completion;Console.WriteLine(result is GraphExecutionResult<Answer>.Completed done ? $"Completed: {done.State.Text}" : $"{result}");An async streaming node finds the stream on NodeContext.Stream, and passes its cancellation token to EmitTokenAsync. A sync streaming node (StreamingNodeHandler<TState>) gets an IStreamContext and calls EmitToken.
let graph = Arc::new( StateGraph::<Answer>::new() .add_async_streaming_node("Answer", |state: Answer, ctx: NodeContext| async move { let stream = ctx.stream().expect("a streaming run always has a stream"); let mut text = state.text; for token in ["Checkpoints ", "make ", "runs ", "durable."] { if stream.emit_token(token).await.is_err() { break; // the consumer dropped the receiver } text.push_str(token); } Answer { text } }) .add_node("Count", |state: Answer| Answer { text: format!("{} ({} chars)", state.text, state.text.len()), }) .add_edge("Answer", "Count") .compile() .expect("the answer graph is valid"),);
let (handle, mut chunks) = graph.clone().run_streaming( "answer-1", Answer { text: String::new() }, Arc::new(InMemoryCheckpointStore::new()),);
while let Some(chunk) = chunks.recv().await { match chunk.chunk_type { StreamChunkType::Token => println!(" {}: {:?}", chunk.node_name, chunk.content), StreamChunkType::Retry | StreamChunkType::Error => { println!("{:?} {}: {}", chunk.chunk_type, chunk.node_name, chunk.content) } _ => println!("{:?} {}", chunk.chunk_type, chunk.node_name), }}
match handle.await.expect("the run's task doesn't panic") { GraphExecutionResult::Completed(state) => println!("Completed: {}", state.text), _ => println!("not completed"),}run_streaming takes an Arc of the compiled graph and must be called inside a tokio runtime. An async streaming node finds the stream with ctx.stream(). A sync one (add_streaming_node) gets a &StreamContext and calls emit_token_blocking.
Both print:
NodeStart Answer Answer: "Checkpoints " Answer: "make " Answer: "runs " Answer: "durable."NodeEnd AnswerNodeStart CountNodeEnd CountCompleted: Checkpoints make runs durable. (30 chars)Reading the stream to the end shows what happened while the run went on. Awaiting the handle gives you the same result RunAsync / run would. The stream ends when the run ends, whatever the outcome.
These are the chunk kinds (StreamChunkType):
| Kind | Sent by | Payload |
|---|---|---|
NodeStart |
The engine, before every node, streaming or not. | |
NodeEnd |
The engine, after a node completes. A node that fails or is cancelled gets none. | |
Token |
Your node, with EmitTokenAsync / emit_token. A model node sends its answer this way. |
The text. |
Retry |
The engine, when a fallible node’s attempt failed transiently and the node runs again. | The failed attempt’s message. |
Error |
The engine, when a fallible node fails for good. A node can also send one with EmitDiagnosticAsync / emit_diagnostic. |
The message. |
When you see Retry, drop the tokens you kept for that node: the next attempt streams again from the start. Chunks from a subgraph’s child are named Work/Send, nested to any depth, and the streaming branches of a parallel group are named Gather/Acme. To stream a model’s answer, see Stream the answer.
To pass run options, use the overloads that take RunOptions (C#) or run_streaming_with with a RunConfig (Rust). A resume streams too: ResumeStreaming / resume_streaming start the stream at the node the resume enters, and say nothing about the nodes that ran before the pause.
A plain run refuses a streaming node
Section titled “A plain run refuses a streaming node”A streaming node needs a stream to write to. A plain run has none, so it fails when it reaches one:
var plain = await graph.RunAsync("answer-2", new Answer(""), new InMemoryCheckpointStore());if (plain is GraphExecutionResult<Answer>.Failed failed){ Console.WriteLine(failed.Message);}node 'Answer' is a streaming node but this run has no StreamContext — use RunStreaming instead of RunAsynclet plain = graph .run("answer-2", Answer { text: String::new() }, Arc::new(InMemoryCheckpointStore::new())) .await;if let GraphExecutionResult::Failed(StrataError::ExecutionAborted(message)) = plain { println!("{message}");}node 'Answer' is a streaming node but this run has no StreamContext — use run_streaming instead of runThe same goes for a parallel group with a streaming branch, and a subgraph that streams. An ordinary async node can also write chunks through NodeContext.Stream / ctx.stream(). Under a plain run, that is null / None, so check for it.
When the reader falls behind
Section titled “When the reader falls behind”The stream is a channel bounded at 1,024 chunks (StreamContext.DefaultChannelCapacity / DEFAULT_CHANNEL_CAPACITY). When it is full, the node’s next emit waits for a free slot. The node pauses; no chunk is dropped, and memory doesn’t grow. Demo 11 shows it with a node that emits 20,000 tokens: while the reader reads nothing, the node stops at 1,023 tokens (the engine’s NodeStart holds the 1,024th slot). Once the reader starts, all 20,000 arrive, in order. See demo 11, .NET and Rust.
A reader that stops reading leaves the node paused for good. To free it, cancel the run: pass a CancellationToken to RunStreaming (C#), or RunConfig::with_cancellation (Rust). An async node that passes its token to EmitTokenAsync stops at once, and the run ends Cancelled with a checkpoint you can resume. In Rust, dropping the receiver also works: the waiting emit_token returns Err(StrataError::ChannelClosed), so stop emitting when it does. A sync node blocked in EmitToken / emit_token_blocking can’t see a token, so the engine stops waiting for it and lets it go when the run ends.
Receive run events
Section titled “Receive run events”A run event is a typed record of one thing the engine did: a node started, an attempt failed, a checkpoint was saved. Events exist in every run mode, streaming or not. Register an observer in RunOptions.Observer (C#) or with RunConfig::with_observer (Rust). For one run, use RunEventChannel, an observer whose reader ends after the run’s last event:
var events = new RunEventChannel();var result = await graph.RunAsync("order-2001", Order.New(40), store, new RunOptions{ Metadata = { ["tenant"] = "acme" }, Observer = events,});
await foreach (var e in events.Reader.ReadAllAsync()){ Console.WriteLine($"#{e.Sequence} {e.GetType().Name} {e.NodePath}{(e.Attempt is { } a ? $" #{a}" : "")}");}The events are records in Datalium.Strata.Observability that derive from RunEvent. A switch over them needs a default arm, because later versions add event types.
let (observer, mut events) = RunEventChannel::new();let result = graph .run_with( "order-2001", Order::new(40), store, RunConfig::new() .with_metadata("tenant", "acme") .with_observer(observer), ) .await;
while let Some(e) = events.recv().await { let attempt = e.attempt.map(|a| format!(" #{a}")).unwrap_or_default(); let path = e.node_path.as_deref().unwrap_or_default(); println!("#{} {} {path}{attempt}", e.sequence, kind_name(&e.kind));}
fn kind_name(kind: &RunEventKind) -> &'static str { match kind { RunEventKind::RunStarted { .. } => "RunStarted", RunEventKind::NodeStarted => "NodeStarted", // ... _ => "other", // non-exhaustive: later versions add kinds }}A RunEvent (in datalium_strata::observability) holds the common fields, and its kind is a RunEventKind. The enum is non-exhaustive, so a match needs a wildcard arm.
Both print:
#1 RunStarted#2 NodeStarted Classify #1#3 RetryScheduled Classify #1#4 NodeCompleted Classify #2#5 NodeStarted Approve #1#6 NodeCompleted Approve #1#7 CheckpointSaved#8 Interrupted#9 RunCompletedEvery event carries the thread ID, the run ID, a sequence number (1, 2, 3, … with no gaps), and a timestamp. Node events also carry the node path (Work/Send inside a subgraph) and the attempt. Run-level events have neither.
Write your own observer
Section titled “Write your own observer”A RunEventChannel serves one run. An observer that outlives runs, for example one that forwards every run’s events to a queue, implements IRunObserver (C#) or RunObserver (Rust). It has one method, called once per event:
sealed class ConsoleObserver : IRunObserver{ public void OnEvent(RunEvent runEvent) { var line = runEvent switch { NodeCompleted n => $"{runEvent.NodePath} {n.Result} in {n.Attempt} attempt(s)", RetryScheduled r => $"{runEvent.NodePath} attempt {r.Attempt} failed: {r.Error}", ModelCallCompleted m => $"{m.Model}: {m.Usage.InputTokens} in, {m.Usage.OutputTokens} out", RunCompleted c => $"run {runEvent.RunId} {c.Outcome}", _ => null, // later versions add event types }; if (line is not null) { Console.WriteLine($"[{runEvent.ThreadId}] {line}"); } }}
await graph.RunAsync("order-2002", Order.New(40), store, new RunOptions { Observer = new ConsoleObserver() });struct ConsoleObserver;
impl RunObserver for ConsoleObserver { fn on_event(&self, event: &RunEvent) { let path = event.node_path.as_deref().unwrap_or_default(); let line = match &event.kind { RunEventKind::NodeCompleted { result, .. } => { format!("{path} {result:?} in {} attempt(s)", event.attempt.unwrap_or_default()) } RunEventKind::RetryScheduled { error, .. } => { format!("{path} attempt {} failed: {error}", event.attempt.unwrap_or_default()) } RunEventKind::ModelCallCompleted { model, usage, .. } => { format!("{model}: {:?} in, {:?} out", usage.input_tokens, usage.output_tokens) } RunEventKind::RunCompleted { outcome, .. } => format!("run {} {outcome:?}", event.run_id), _ => return, // later versions add event kinds }; println!("[{}] {line}", event.thread_id); }}
graph .run_with("order-2002", Order::new(40), store, RunConfig::new().with_observer(Arc::new(ConsoleObserver))) .await;[order-2002] Classify attempt 1 failed: gateway timeout[order-2002] Classify Committed in 2 attempt(s)[order-2002] Approve Committed in 1 attempt(s)[order-2002] run 01a18d45-64a0-79ac-afbd-b8d730dbdd45 InterruptedHow the engine delivers events:
- In order, one at a time. The observer gets each run’s events in sequence order, never two at once for the same run. An observer you share between runs must be thread-safe.
- Off the run’s path. Events go into an unbounded queue per run, and a separate task delivers them. A slow observer only makes its own queue grow; it never makes the run wait. In Rust,
on_eventruns on a tokio task, so it must not block: hand slow work to a channel. - Failures are counted, not raised. An exception (C#) or panic (Rust) from
OnEvent/on_eventis caught and counted in thestrata.observer.errorsmetric, and delivery goes on. An observer can’t fail a run. - The run doesn’t wait for delivery.
RunAsync/run_withcan return before the observer has seenRunCompleted. If you need every event, wait forRunCompletedon the observer’s side, asRunEventChanneldoes. - Pairs always close. Every
NodeStartedgets oneNodeCompleted, and everyBranchStartedoneBranchCompleted, even when the node fails or is cancelled.RunStartedis always first andRunCompletedalways last.
The events
Section titled “The events”| Event | When |
|---|---|
RunStarted |
First, every run. Carries the mode (Run, Resume, End, Recover), the node it starts at, the run’s metadata, the graph identity and the time spent waiting for a scheduler. |
Resumed |
A resume, after the old checkpoint is stamped. Who resumed, whether they approved, and what woke the thread (a person, an event or a timer). |
Recovered |
A recovery claimed a thread whose process died. |
NodeStarted |
A node is about to run its first attempt. |
RetryScheduled |
An attempt failed transiently and will run again: the error, whether it timed out, and the delay. |
NodeCompleted |
The engine is done with a node: Committed, Failed, Interrupted or Cancelled, the time over every attempt, the error, and the fallback node if there is one. |
BranchStarted, BranchCompleted |
A parallel group’s branch started, or ended (Succeeded, Failed, Cancelled, with a reason such as LostRace). |
JoinDecided |
A group whose join isn’t All decided it. |
MapExpanded |
A map read its items. Each item then reports as a branch, at <map>/<key>. |
CheckpointSaved |
A checkpoint was written to your store: its status, where it waits, and how long the save took. |
Interrupted |
The run paused at an interrupt boundary, and the checkpoint is saved. |
HopLimitExceeded |
The hop limit stopped a loop. |
LeaseAcquired, LeaseLost |
The run took its thread’s lease, or lost it and stopped. See Run multiple workers. |
ToolCallStarted, ToolCallCompleted |
A tool registry dispatched a call from a node: the tool, the result and the duration. |
ModelCallStarted, ModelCallCompleted |
A node called a model: the model, the provider, the token usage and the finish reason. |
ModelRouted |
A call through a model route ended: which candidate answered, and what happened to each one. |
RunCompleted |
Last, every run: Completed, Interrupted, Failed or Cancelled, the run’s duration, and the error or the cancel reason. |
Run IDs and metadata
Section titled “Run IDs and metadata”Each run gets an ID, a UUID v7 unless you set one with RunOptions.RunId / RunConfig::with_run_id. A resume is a new run with a new ID; the thread ID connects them. The run ID is on events, spans and the node’s context, but not on checkpoints.
Metadata is your own key/value pairs for a run, such as a tenant or a ticket, set with RunOptions.Metadata / RunConfig::with_metadata. The engine:
- writes it on every checkpoint the run saves, at every subgraph level, beside its own keys;
- puts it on
RunStartedand on the run’s span; - gives it to every node, as
NodeContext.Metadata/ctx.metadata().
A resume carries the stored metadata forward, and the resume’s own metadata overrides it key by key. A resume can add or change a key, but not remove one. The engine reserves error, error_kind, attempts, cancel_reason and every key that starts with strata.. A run that uses one, or an empty key, is refused before anything is written: C# throws ArgumentException, and Rust returns Failed with run metadata key '<key>' is reserved.
Events carry no state
Section titled “Events carry no state”No event includes your state, a diff of it, a prompt, a model’s answer, or a tool’s arguments or result. Building a diff of an arbitrary state type means serializing it twice per node, and it would send your domain data, which can include personal data, to every telemetry system by default. To follow the state, read the checkpoints, as Inspect a thread afterwards shows.
Export traces and metrics with OpenTelemetry
Section titled “Export traces and metrics with OpenTelemetry”The engine has one ActivitySource and one Meter, both named Datalium.Strata (StrataTelemetry.Name). Nothing is recorded until something listens. In a hosted app, AddInstrumentation() adds every Strata source and meter to your OpenTelemetry providers. See Send telemetry to OpenTelemetry. Without the host, add the names yourself:
using Datalium.Strata.Observability;using OpenTelemetry;using OpenTelemetry.Metrics;using OpenTelemetry.Trace;
using var tracing = Sdk.CreateTracerProviderBuilder() .AddSource(StrataTelemetry.Name) .AddConsoleExporter() .Build();using var metrics = Sdk.CreateMeterProviderBuilder() .AddMeter(StrataTelemetry.Name) .AddConsoleExporter() .Build();StrataTelemetry.Name covers the engine. Some other Strata packages, such as storage and hosting, have sources or meters of their own, and StrataInstrumentation.ActivitySourceNames and StrataInstrumentation.MeterNames, in Datalium.Strata.Hosting, list them all.
Run the review graph, and the first attempt of Classify exports as this span (part of the output):
Activity.ParentSpanId: f4324e820a09964fActivity.DisplayName: Classify #1Activity.Kind: InternalActivity.Tags: strata.thread_id: order-4001 strata.run_id: 01a18d45-b27c-75e3-8c96-a4598037128d strata.node.path: Classify strata.attempt: 1 error.type: transientStatusCode: ErrorActivity.StatusDescription: gateway timeoutThe engine writes spans and events through tracing, and metrics through the metrics crate. Both are global facades, so there is no instrumentation call: install any tracing subscriber and any metrics recorder, and Strata’s spans and metrics arrive in them. To send spans to OpenTelemetry, add the tracing-opentelemetry layer:
use opentelemetry::trace::TracerProvider as _;use opentelemetry_sdk::trace::SdkTracerProvider;use tracing_subscriber::layer::SubscriberExt;use tracing_subscriber::util::SubscriberInitExt;
let provider = SdkTracerProvider::builder() .with_simple_exporter(opentelemetry_stdout::SpanExporter::default()) .build();tracing_subscriber::registry() .with(tracing_opentelemetry::layer().with_tracer(provider.tracer("my-service"))) .init();
// ... run graphs ...
provider.shutdown().expect("spans flushed");Run the review graph, and the first attempt of Classify exports as this span (part of the output):
Name : Classify #1 ParentSpanId : ae2814816e6f6484 Kind : Internal Status : Error { description: "" } Attributes: -> strata.thread_id: String(Owned("order-4001")) -> strata.run_id: String(Owned("01a18d47-a54b-720a-8f47-4129ac5e2974")) -> strata.node.path: String(Owned("Classify")) -> strata.attempt: String(Owned("1")) -> otel.status_message: String(Owned("gateway timeout")) -> error.type: String(Owned("transient"))The engine also writes tracing events, which are logs in that ecosystem: for example, run started and run completed at info, a scheduled retry and a failed node at warn, and a saved checkpoint at debug. Their targets start with datalium_strata, so a filter directive such as datalium_strata=warn covers them all.
A run makes a tree of spans: run, then node, then attempt. Your handler runs inside its attempt’s span, so a span your own code or an HTTP client opens nests under it. To add to the attempt’s span, use NodeContext.Activity (C#, null when nothing listens) or ctx.span() (Rust).
| Span name | Name a trace viewer shows | Parent | Main attributes |
|---|---|---|---|
strata.run |
strata.run, strata.resume, strata.end or strata.recover |
Whatever span is current when you start the run. | strata.thread_id, strata.run_id, strata.run.mode, the metadata, strata.outcome, strata.cancel_reason, strata.graph.id, strata.graph.version |
strata.node |
The node path, such as Classify or Work/Send |
The run span, or the subgraph node’s attempt span. | strata.node.path, strata.retry_count, strata.node.result |
strata.attempt |
Classify #2 |
The node span. | strata.node.path, strata.attempt, error.type |
strata.branch |
The branch path, such as Gather/Acme |
The group’s attempt span. | strata.node.path, strata.attempt |
strata.tool |
execute_tool <tool> |
The calling node’s attempt span. | gen_ai.tool.name, gen_ai.tool.call.id, strata.tool.result |
strata.model |
chat <model>, or embeddings <model> |
The calling node’s attempt span. | The gen_ai.* attributes below |
strata.route |
route <name> |
The calling node’s attempt span. | strata.route, strata.route.answered |
Every span carries strata.thread_id and strata.run_id. A failed span has an error status and an error.type of transient, fatal or timeout (or the exception type, for a bug). A cancelled run, node or attempt doesn’t: a cancellation isn’t an error. The node span has a retry event for each retry, and the run span a checkpoint.saved event for each save.
The metadata is one attribute per key in C# (strata.metadata.tenant), and one JSON attribute in Rust (strata.metadata). A resume’s spans aren’t linked to the trace of the run that paused: each run has its own strata.run span, and the thread ID connects them.
Metrics
Section titled “Metrics”The main metrics. Thread IDs, run IDs and metadata are never metric attributes, because they have no bound.
| Metric | Type | Unit | Attributes |
|---|---|---|---|
strata.graphs.started |
Counter | {run} |
strata.run.mode |
strata.graphs.active |
Up-down counter (a gauge in Rust) | {run} |
|
strata.graph.duration |
Histogram | s |
strata.run.mode, strata.outcome |
strata.node.duration |
Histogram | s |
strata.node.path, strata.node.result |
strata.node.retries |
Counter | {retry} |
strata.node.path, error.type |
strata.graph.transition |
Counter | {transition} |
strata.node.path, strata.next_node.path |
strata.checkpoint.latency |
Histogram | s |
strata.checkpoint.operation (save, load), strata.checkpoint.status |
strata.observer.errors |
Counter | {error} |
|
strata.tool.requests |
Counter | {request} |
gen_ai.tool.name, strata.tool.result |
strata.tool.duration |
Histogram | s |
gen_ai.tool.name, strata.tool.result |
gen_ai.client.token.usage |
Histogram | {token} |
gen_ai.token.type (input, output), the operation, provider and model |
gen_ai.client.operation.duration |
Histogram | s |
The same, plus error.type |
strata.model.time_to_first_chunk |
Histogram | s |
A streamed call’s time to its first update. |
The same meter also has metrics for the hop limit (strata.graph.hop_limit_exceeded), waits (strata.thread.wakes), recovery (strata.thread.recoveries), leases (strata.lease.acquisitions, strata.lease.losses), the run scheduler (strata.scheduler.*) and model routes (strata.route.*, strata.model.breaker.transitions). In Rust, the names are constants in datalium_strata::observability::metric_names.
Model calls and token usage
Section titled “Model calls and token usage”Every model call made through Strata’s AI package, or through a ModelCallScope in your own client, opens a strata.model span with the OpenTelemetry GenAI attributes: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.response.finish_reasons, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. It records the two gen_ai.client.* metrics and the ModelCallStarted / ModelCallCompleted events. Token counts are what the provider reported, and a count it didn’t report is left empty, never estimated.
Prompts and answers are never recorded, in spans, metrics or events. Neither are tool arguments or tool results.
Inspect a thread afterwards
Section titled “Inspect a thread afterwards”Events and spans are not stored. If nothing listened during a run, they’re gone. Checkpoints stay in the store, with your metadata on them, and the strata CLI reads them from any SQLite, Postgres or Redis store. Find a tenant’s threads with --meta, then look at one:
strata thread list --storage checkpoints.sqlite --meta tenant=acmestrata thread inspect order-5001 --storage checkpoints.sqlite+------------+--------------+-------------+-------------+-------------+----------------------+| Thread ID | Current Node | Sequence ID | State | Waiting For | Updated At |+------------+--------------+-------------+-------------+-------------+----------------------+| order-5001 | Approve | 0 | interrupted | - | 2026-10-30T14:48:27Z |+------------+--------------+-------------+-------------+-------------+----------------------+Thread: order-5001 Status: interrupted Thread state: interrupted Current node: Approve Metadata: tenant=acme, ticket=T-1001 Next node: Execute Checkpoint count: 1 Created at: 2026-10-30T14:48:27Z Resumed at: - State:{ "amount": 40, "log": [ "classify", "approve" ]}The Metadata line shows your keys only. For a failed thread, --json shows the engine’s keys too, such as metadata.error. strata checkpoint log lists every checkpoint of a thread, and strata checkpoint diff shows which state fields changed between two of them. From code, query threads by metadata as Querying threads shows.
Related
Section titled “Related”- Retries and backoff: watch retries with
RetryScheduled. - Call a model: stream a model’s answer, and what a call records.
- Host Strata in a service:
AddInstrumentation()and the drivers’ metrics. - CLI reference: every command and option.
- Demo 27, run observability: .NET, Rust.
- Demo 11, backpressure: .NET, Rust.
- Demo 06, token streaming: .NET, Rust.