Durable execution

Durable execution means a thread outlives the process that runs it. If the process stops, another process, or the same one after a restart, picks the thread up from the store. This page explains what Strata keeps, what it runs again, and what it leaves to your application.

What survives a crash

What a crash costs depends on where the thread is when the process dies, and on the graph’s checkpoint policy.

The thread is…	Boundaries (the default)	EveryNode
Waiting: at an interrupt boundary, an event wait or a timer	Survives. It is in the store.	Survives.
Running a node	The run is lost. Nothing was saved since it started or resumed.	Continues at the node that was running, with the state from the last node that committed.

Under Boundaries, a fresh run that dies leaves no checkpoint at all. A resumed run that dies leaves the thread marked as resumed, and nothing can continue it.

Under EveryNode, a run writes a running checkpoint before its first node and after every node that commits, and a completed one when it finishes. A live run always leaves its thread completed, interrupted, failed or cancelled, so a running thread that nobody works on has lost its process. A parallel group and a map count as one node, checkpointed once after their branches or items join.

A retry’s backoff wait isn’t durable. It is held in memory, so a crash during the wait loses the run like any other crash. See Known limitations.

Each checkpoint is one store round trip. In the benchmark results, a ten-node run under EveryNode takes 3 ms on Redis, 12 to 14 ms on Postgres and 25 ms on SQLite.

Recovering a thread

Recovery continues a running thread at the node it was running. It needs a graph compiled with EveryNode.

• One thread: RecoverAsync in C#, recover or recover_with in Rust.
• Every dead thread: RecoverStaleAsync in C#, recover_stale in Rust. It lists the graph’s running threads, skips the ones a live lease holds, and recovers the rest one at a time.

Before it runs anything, a recovery saves a claim: the thread’s checkpoint again, saved only if the thread hasn’t changed since it was loaded. If two processes try to recover the same thread, one claim wins and the other process runs nothing.

How Strata decides a thread is dead

A thread is stale when it is running and nothing shows that a run still owns it.

• With leases (see Thread leases), a run holds its thread’s lease from before its first save until it returns. A running thread with no live lease has no run. The stale thread is recovered once the dead worker’s lease expires.
• Without leases, the only sign of life is the age of the latest checkpoint. You must set StaleAfter (Rust: with_stale_after) to longer than your slowest node. A node slower than that is run twice. If the recovery runs have no lease and you don’t set StaleAfter, RecoverStaleAsync refuses the call.

A node that crashes every time

A node that kills its process would be recovered forever. Each claim adds one to the strata.recoveries count in the checkpoint’s metadata, and the count goes back to zero when a node commits. Set MaxRecoveries (Rust: with_max_recoveries) and the driver skips a thread that reached the limit. The thread stays running so that someone can look at it and end it. You can end a running thread as long as no live lease holds it.

RecoverAsync and recover don’t apply this limit. If you recover threads yourself, read the count with RecoveryCount.Of (Rust: recoveries_of) and stop at your own limit.

At least once

The node that was running when the process died runs again, from the state it started with. That also happens when the node finished but its checkpoint wasn’t saved yet. In a parallel group or a map, every branch or item runs again.

So Strata runs each node at least once, not exactly once. Anything a node does outside its state, such as charging a card or sending an email, can happen twice unless the node prevents it. Two values on the node context help:

• The idempotency key (NodeContext.IdempotencyKey, Rust idempotency_key()) names one execution of one node in one thread, for example order-1042/Charge#2. It is the same on every retry of the node and when a recovery runs the node again. It is different when the thread reaches the same node again later, in a loop. Pass it to the system you call, so that it can ignore a repeat.
• IsRecovery (Rust is_recovery()) is true when the node may have run before. It means “may have”, not “did”.

The key repeats across a recovery only under EveryNode. Under Boundaries it ends with the run’s ID instead.

Thread leases

Two workers can end up on the same thread: a queue delivers a message twice, a partition moves during a rebalance, or a worker stalls and another takes over. A lease makes sure only one of them writes to the thread.

You turn leases on per run with RunOptions.Lease = new LeaseOptions(owner, ttl) in C#, or RunConfig::with_lease(LeaseOptions::new(owner, ttl)) in Rust. The owner names the process. Every built-in store keeps leases next to the thread’s checkpoints.

• Acquiring. A run takes the lease before its first save. If another live lease holds the thread, the run is refused with Failed (“thread ‘t’ is held by ‘worker-2’ until …”) and writes nothing.
• The token. Each time a thread’s lease is acquired, the lease gets the next number in a per-thread counter.
• Renewing. The run renews its lease in the background every third of the ttl, and releases it when the run returns. A waiting thread holds no lease.
• Fenced saves. Every save checks, in the same atomic step, that the run’s token is still the thread’s latest. A worker that stalled while someone else took the thread can’t write over the new owner’s checkpoints.
• Losing the lease. A run loses its lease when a renewal finds another owner, when a save is refused, or when no renewal has succeeded for a whole ttl. The run stops at once, writes nothing more (not even a cancelled checkpoint), and returns Cancelled with the reason LeaseLost.

Leases protect the checkpoints, not the outside world. A stale worker’s node can still run alongside the new owner’s until its next renewal or save. Use the idempotency key for effects. Leases are also opt-in: a save without a lease is accepted even while a lease is live. Run every worker on a thread with a lease, or none of them.

Wakes and timers

A timer and an event wait are interrupt boundaries with a wake condition. The thread is checkpointed as interrupted, with the time it wakes or times out. It holds nothing in memory and no lease. In the benchmark results, ten thousand frozen threads in SQLite grew the heap by nothing measurable in either engine.

Nothing fires on its own. Something must call WakeDueAsync (Rust: wake_due) on a regular cadence. Each call wakes the timers that are due and times out the waits whose timeout has passed. An event wait is answered with SignalAsync (Rust: signal), and a person’s answer is a resume, as Human-in-the-loop describes.

A wake is a resume, saved only if the thread hasn’t changed. Two processes that try to wake the same thread can’t both succeed.

The hosting drivers

You don’t have to write these loops. The hosting packages run them as drivers:

Driver	.NET (Datalium.Strata.Hosting)	Rust (datalium-strata-host)	Default interval
Wake	AddWakeDriver<TState>()	Host::wake	5 seconds
Recovery	AddRecoveryDriver<TState>()	Host::recover	30 seconds

• The first round runs when the host starts, so a host that comes back after a crash recovers at once.
• A round that fails is logged and tried again after a wait that doubles, up to 5 minutes. A driver never stops the host.
• The recovery driver refuses to start without a graph compiled with EveryNode, and either a lease or StaleAfter.
• When the host stops, a driver lets the run in flight finish. If that run outlasts the shutdown timeout, the thread is left running for a later recovery.

Every replica can run every driver. Wakes and recoveries claim each thread with a conditional save, so two replicas never act on the same thread twice.

What Strata leaves to you

Strata doesn’t spread runs across machines. Each run executes in one process, and Strata has no cluster, no placement and no run queue. You distribute runs with what you already use: Kafka partitions, a queue or a job system. Strata makes a thread safe wherever it runs. The decision and the pattern are in design/47:

1. Route every message about a thread to the same consumer, keyed by the thread ID.
2. Run every thread under a lease, so a rebalance that briefly gives two processes the same thread can’t corrupt it.
3. Compile with EveryNode where losing work matters, and run the recovery driver on every replica.
4. Run the wake driver on every replica.

See it run

Each demo exists in C# and in Rust:

• 19, waits and timers: event waits, durable timers and waking due threads (C#, Rust).
• 20, durable execution: a real process crash, then recovery, with a payment that is charged once (C#, Rust).
• 21, thread leases: two workers on one thread, with fencing (C#, Rust).
• 28, hosting: a host with a wake driver (C#, Rust).

Next

• Threads and checkpoints: what a checkpoint holds, and the two checkpoint policies.
• Human-in-the-loop: pausing a thread for a person, and resuming it.
• Architecture: where the stores and the hosting packages fit.