Threads and checkpoints

A thread is one execution of a graph. A checkpoint is a saved snapshot of a thread. Strata writes checkpoints to a store, and reads them back to resume, recover, end or inspect the thread. This page explains what a checkpoint holds, when Strata writes one, and how you read them.

Threads

You name each thread with a thread ID, a string you choose, such as an order number. You pass it to every call that acts on the thread: RunAsync / run to start it, ResumeAsync / resume to continue it, EndAsync / end to end it.

A thread can span several runs. It starts, stops at an interrupt boundary, and is resumed hours later, by another process, or by the other language’s engine. The thread ID is what ties those runs together. Starting a run doesn’t check the store for an earlier thread with the same ID, so give each new execution its own ID.

A subgraph’s child isn’t a thread of its own. Its checkpoint is nested inside its parent’s.

Statuses

Every checkpoint has a status. The status of a thread’s latest checkpoint says where the thread stands.

Status	Written when	Can it continue?
interrupted	The thread stops at an interrupt boundary, or waits for an event or a timer.	Yes: a resume, a signal, or a wake when its timer is due.
failed	A node fails after its retries, and there is no fallback node.	Yes: a resume runs the failed node again, from the last committed state.
cancelled	The run’s cancellation token fires, or its deadline passes.	Yes, the same way as failed.
running	Only with the EveryNode policy: before the first node, and after every node that commits. A running thread is either still running, or its process died.	Not by a resume. A dead running thread is recovered (see Durable execution).
completed	Only with the EveryNode policy: the run finished.	No.
ended	You end the thread with EndAsync / end.	No. A later resume returns Failed.

You can end a thread that is interrupted, failed or cancelled, or one left running that no live lease holds. Ending writes an ended checkpoint where the thread stood, and the call returns Cancelled with the reason Ended.

When you list threads, Strata derives a seventh state, resumed: a resume began from the latest checkpoint, under the default Boundaries policy. The store can’t tell whether that run is still going, finished, or died with its process. With EveryNode, a resume writes a running checkpoint instead, so the thread never shows as resumed.

What a checkpoint holds

A checkpoint is a JSON record. Its format is defined by schema/checkpoint.schema.json and is the same in both engines. Its main fields:

Field	What it holds
thread_id	The thread it belongs to.
status	One of the six statuses above.
suspended_node	Where the thread stopped: the boundary it waits at, the node that failed or was cancelled, or, for running, the node that last committed.
next_node	Where the thread goes on: the node a resume or a recovery runs next. Null for completed and ended.
frozen_state	Your state, as JSON.
metadata	String key/value pairs. Your run’s metadata, such as a tenant, is stamped on every checkpoint the run writes. The engine adds its own keys, such as the error of a failed node, the cancel reason, or who ended the thread.
nested	The child graph’s own checkpoint, when the thread stopped inside a subgraph. It can nest further.
decision	The human decision in effect: which boundary was answered, approved or not, by whom, the field overrides, and when. See Human-in-the-loop.
wait	What an interrupted thread waits for, when it is more than a person with no timeout: an event, a timer, or a person with a timeout, and when it is due.
graph	The ID and version of the graph that wrote it, if the graph was compiled with an identity. See Graphs.
created_at, resumed_at	When the checkpoint was written, and when a resume began from it.
schema_version	The version of the record format. A reader refuses a version newer than itself.

Optional fields are left out when they don’t apply. A graph that uses nothing new writes exactly what older versions wrote.

The log and its sequence

A store keeps every checkpoint of a thread, in order, as an append-only log. Nothing is overwritten. Each checkpoint has a sequence: its position in the thread’s log, counting from 0.

The sequence isn’t part of the checkpoint record. The store assigns it when it appends the checkpoint. It is local to one thread, not a global counter.

Because the log is append-only, a resume also adds a row. With the default policy, it appends a copy of the checkpoint it resumed, with resumed_at set. So a thread that waited once and then finished has two rows: the interrupted checkpoint, then the same checkpoint stamped as resumed.

When checkpoints are written

You choose a checkpoint policy when you compile the graph:

Policy	What it writes	Set it with
Boundaries (the default)	Only where a run stops: an interrupt boundary or a wait, a failure, a cancellation, an end. Nothing between nodes. A run that completes writes nothing.	Nothing to set.
EveryNode	Also a running checkpoint before the first node and after every node that commits, and a completed one when the run finishes.	C#: Checkpointing = CheckpointPolicy.EveryNode in CompilationOptions. Rust: CompileOptions::new().with_checkpointing(CheckpointPolicy::EveryNode) with compile_with.

Under Boundaries, a run that goes straight through to the end leaves nothing in the store: it returns Completed with the final state. If its process dies mid-run, the run is lost.

EveryNode lets a thread survive a crash, at the cost of one store write and one state serialization per node. A parallel group and a map count as one node each, and a subgraph follows the outermost graph’s policy. Durable execution explains recovery.

The commit rule

A node’s changes reach the store only after the node finishes. If a node throws, nothing is written for it, and each retry starts from the last committed state. A failed attempt’s partial changes never reach the retry, the fallback node or the checkpoint. So a thread’s log is always a sequence of whole steps.

Conditional and fenced saves

Two writers can act on one thread at once, for example a person approving it while an operator ends it. Two save forms keep both from winning:

• A conditional save (SaveIfLatestAsync / save_if_latest) appends only if the thread’s latest sequence is still the one the writer read. Ending a thread and stamping a resume use it.
• A fenced save (SaveUnderLeaseAsync / save_under_lease) appends only if the writer still holds the thread’s lease. See Durable execution.

Stores

A checkpoint store is a port: ICheckpointStore in C#, the CheckpointStore trait in Rust. Strata ships four adapters in each language: in-memory, SQLite, Postgres and Redis. The engine saves and loads through the port, so your graph code doesn’t change when you switch stores. The in-memory store is for tests and development: nothing survives the process.

Both engines write the same rows. A thread that a .NET service paused in a SQLite file, a Postgres database or a Redis keyspace can be resumed by a Rust service, and the other way round.

The checkpoint store holds execution state only. A conversation’s messages go to a separate history store (IHistoryStore / HistoryStore), and long-term memory to a third, the memory store. State explains what belongs where.

For large state, the SQLite, Postgres and Redis stores have two opt-in options, delta checkpoints and value offload, which shrink what they store. The engine still sees whole checkpoints. State describes both.

Stores never delete anything on their own. Each built-in store has a retention port (ICheckpointRetention / CheckpointRetention) whose PurgeAsync / purge deletes old threads, by default those completed or ended, when your application calls it. Under the default Boundaries policy, a thread that finished after a resume stays resumed, so add that state to the purge if you want those threads deleted too.

Querying threads

Every built-in store also implements a query port, ICheckpointQuery / CheckpointQuery, with two operations:

• ListThreadsAsync / list_threads lists threads, newest first, one page at a time. A ThreadQuery filters by state, by metadata key and value, by when the thread was last updated, by graph ID and version, or by whether its timer is due.
• GetLogAsync / get_log reads one thread’s log, oldest first, from a sequence you give, up to 1,000 entries a page.

This lists the threads waiting for a person, then reads one thread’s log:

C#

var page = await store.ListThreadsAsync(new ThreadQuery
{
    States = new HashSet<CheckpointThreadState> { CheckpointThreadState.Interrupted },
});
foreach (var thread in page.Threads)
{
    Console.WriteLine($"{thread.ThreadId} waits at {thread.SuspendedNode}");
}

var log = await store.GetLogAsync("order-1001");
foreach (var entry in log.Entries)
{
    Console.WriteLine($"#{entry.Sequence} {entry.Checkpoint.Status}");
}

Rust

use datalium_strata::store::{CheckpointQuery, ThreadQuery, ThreadState};

let query = ThreadQuery::default().with_states([ThreadState::Interrupted]);
let page = store.list_threads(&query).await?;
for thread in &page.threads {
    println!("{} waits at {:?}", thread.thread_id, thread.suspended_node);
}

let log = store.get_log("order-1001", None, 100).await?;
for entry in &log.entries {
    println!("#{} {:?}", entry.sequence, entry.checkpoint.status);
}

Inspecting from the command line

The strata CLI reads the same stores. The CLI reference covers every option. Give it a SQLite file path, or a postgres:// or redis:// URL, with --storage.

Command	What it shows
strata thread list	One row per thread: its current node, latest sequence, state, what it waits for, and when it was last updated. Filter with --status, --meta, --graph or --due.
strata thread inspect <id>	The latest checkpoint in full: status, current and next node, graph, lease, wait, decision, metadata, how many checkpoints the thread has, and the state.
strata checkpoint log <id>	The thread’s timeline: one line per checkpoint, with its sequence, status, node, next node and time.
strata checkpoint diff <id> --seq-a <n> --seq-b <m>	The top-level state fields that changed between two checkpoints.
strata checkpoint replay <id> --sequence <n>	The state, the decision in effect, and the node the thread would resume at, as of one checkpoint.

Every command takes --json. The CLI is read-only: it can’t resume or end a thread, and replay doesn’t run the node. See Known limitations. Installation shows how to install the CLI.

See it run

Each demo exists in both languages, with the same folder name:

• 05-hitl-interruption: threads freeze at a boundary in a SQLite file. A new store opened on the same file, standing in for a restarted process, lists the waiting threads and resumes them. C#, Rust.
• 09-cross-language-interop: one engine writes the checkpoint, the other resumes it. C#, Rust.
• 12-pluggable-storage: one scenario against the in-memory, SQLite, Postgres and Redis stores. C#, Rust.
• 20-durable-execution: an EveryNode graph whose process is killed mid-run. Another process finds the running thread and finishes it. C#, Rust.

Next

• Durable execution: recovering a thread whose process died, and thread leases.
• Human-in-the-loop: pausing at an interrupt boundary and resuming with a decision.
• State: what goes in the state, and keeping it small.