Skip to content
Datalium StrataDatalium Strata
These docs cover Strata 0.9.0-preview.1.It is a preview, and the API can still change before 1.0.

Run multiple workers

In this guide you run one graph from several worker processes against one shared store. You run every thread under a lease, handle the two results a lease adds, route messages so that one worker usually owns a thread, recover the threads of a worker that died, and see which worker holds which thread.

Strata doesn’t spread runs across processes for you. You bring the queue or broker, and Strata keeps each thread safe wherever it runs. What Strata leaves to you explains the split.

The examples use an invoice graph: Draft runs, the thread waits for approval, then Send runs. It is compiled with CheckpointPolicy.EveryNode and the graph ID invoices.

Before you start

Section titled “Before you start”
  • The packages from Installation. In .NET: Datalium.Strata.Hosting, which brings Datalium.Strata and Datalium.Strata.Storage. In Rust: datalium-strata, datalium-strata-memory, serde and tokio.
  • A store that every worker can reach. The examples use one SQLite file that two workers open. Across machines, use a server store such as Postgres or Redis. See Choose a store.
  • Thread leases explains acquiring, renewing and fencing. This guide puts them to work.

Run every thread under a lease

Section titled “Run every thread under a lease”

Give each run a LeaseOptions with the worker’s name and a ttl. The run takes the thread’s lease before its first save, renews it in the background, and releases it when it returns.

public async Task HandleAsync(string threadId)
{
    var options = new RunOptions
    {
        Lease = new LeaseOptions(_owner, _ttl),
        // ...
    };


    var result = await Graph.RunAsync(threadId, new Invoice(threadId, null, false), Store, options);


    switch (result)
    {
        case GraphExecutionResult<Invoice>.Failed { Message: var why } when why.Contains(" is held by "):
            // Another worker runs this thread. Nothing was written.
            Console.WriteLine($"{_owner}: skipped {threadId}: {why}");
            break;


        case GraphExecutionResult<Invoice>.Cancelled { Reason: CancelReason.LeaseLost } lost:
            // Another worker took the thread over. This run wrote nothing more.
            Console.WriteLine($"{_owner}: lost the lease on {threadId} in {lost.Node}");
            break;


        case GraphExecutionResult<Invoice>.Interrupted waiting:
            Console.WriteLine($"{_owner}: {threadId} waits at {waiting.SuspendedNode}");
            break;


        default:
            Console.WriteLine($"{_owner}: {threadId} ended {result.GetType().Name}");
            break;
    }
}

Pass the same lease to every call that writes to a thread: a run, a resume, a signal, a recovery and an end, and to the drivers. Leases are opt-in. A save without a lease is accepted even while another worker holds the thread, so a single worker without a lease breaks the guarantee for everyone.

In a host, you set this once. In .NET, set Strata:Runs:Lease:Ttl, and every run that StrataRunner and the drivers start takes the lease. In Rust, set runs.lease.ttl, and pass settings.runs.config() to your runs and to the drivers with with_config. Configure Strata from settings shows the section.

Choose the owner and the ttl

Section titled “Choose the owner and the ttl”

The owner names one process, not the service. A live lease refuses every other run, even one with the same owner name, so a distinct name changes nothing for safety. It tells you which process holds a thread, in the refusal message and in strata thread inspect. The hosting default, the host name and the process ID, does that.

The ttl trades recovery time against tolerance for pauses:

  • A run renews every third of the ttl, so a node can take longer than the ttl. A 30-second ttl means one store call every 10 seconds for each run in progress.
  • When a worker dies, its threads stay held until their leases expire. Recovery waits for that, so the worker’s threads continue up to one ttl plus one recovery interval later.
  • A run that can’t renew for a whole ttl, because its process paused or the store was out of reach, loses the lease and stops.

The hosting examples use 30 seconds. The examples here use 3 seconds so that they run fast.

In Rust, renewals run as a tokio task. A node that blocks the only thread of a single-threaded runtime blocks its renewals too, which is how the Rust example below stalls a worker.

Handle a held thread and a lost lease

Section titled “Handle a held thread and a lost lease”

A lease adds two results to a run. The example shows both, with two workers on one SQLite file and a 3-second ttl:

  1. worker-a starts invoice-7 and stays in Draft for 4 seconds. worker-b gets the same thread and is refused. The listing before and after shows that worker-a’s renewals moved the lease’s expiry.
  2. worker-a starts invoice-8 and stalls in Draft past its ttl. worker-b takes the thread and stops it at Draft. When worker-a wakes up, its save is refused.

The .NET example stalls worker-a by giving its run a clock that stands still, so it never renews. The Rust example blocks worker-a’s runtime with std::thread::sleep.

invoice-7: running, held by 'worker-a' until 14:48:21 (token 1)
worker-b: skipped invoice-7: thread 'invoice-7' is held by 'worker-a' until 2026-10-30T14:48:22Z
invoice-7: running, held by 'worker-a' until 14:48:25 (token 1)
worker-a: invoice-7 waits at Draft
worker-b: invoice-8 waits at Draft
worker-a: lost the lease on invoice-8 in Draft
invoice-8's latest checkpoint: Interrupted at Draft, drafted by worker-b

Held. The result is Failed with the message thread '<id>' is held by '<owner>' until <time>. In Rust it is StrataError::ExecutionAborted. No node ran and nothing was written. There is no separate result type, so match the message. Another worker has the thread: if the message was a duplicate, acknowledge it and move on. If it must still be applied, try it again later.

Lost. The result is Cancelled with the reason LeaseLost. The thread now belongs to another worker, and that worker’s checkpoints are its history. Unlike other cancellations, the run saves no cancelled checkpoint, so there is nothing to clean up or retry. When a renewal finds the takeover, the node’s cancellation token fires, so a node that watches its token stops at once.

Each run also reports LeaseAcquired and LeaseLost run events, and two counters: strata.lease.acquisitions, by strata.lease.outcome (acquired or held), and strata.lease.losses. Observability shows how to collect them.

Route each thread to one worker

Section titled “Route each thread to one worker”

Leases make a second worker on a thread harmless. Routing makes it rare. Send every message about a thread, whether a start, an approval or a signal, to the same consumer, keyed by the thread ID:

  • Kafka or Redpanda: use the thread ID as the message key. Every message for the thread lands on one partition, and one consumer in the group reads that partition.
  • A queue with ordered groups, such as Azure Service Bus sessions or Amazon SQS FIFO message groups: use the thread ID as the group ID.

Two workers can still meet on a thread: a partition moves during a rebalance, a queue redelivers a message after its timeout, or a consumer stalls. The lease refuses the second worker, and fencing stops the stalled one. A few held refusals during a rebalance are expected, not errors.

Recover on every replica

Section titled “Recover on every replica”

Compile the graph with CheckpointPolicy.EveryNode where losing work matters, and run the recovery driver on every replica, with the same lease as your runs. Each round lists the running threads, skips the ones a live lease holds, and claims each of the rest with a conditional save. When two replicas list the same dead thread, one recovers it and the other skips it. Recovering a thread explains the claim, and How Strata decides a thread is dead explains why the lease matters.

  • In .NET, AddRecoveryDriver<TState>() uses the lease from Strata:Runs:Lease. In Rust, pass your run config to the driver with with_config.
  • Set MaxRecoveries so a node that kills its process isn’t recovered forever. See A node that crashes every time.
  • Run the wake driver on every replica too. A wake also claims its thread with a conditional save, and it skips a thread another worker holds.

What the drivers need lists what each driver checks before it starts.

See where threads stand

Section titled “See where threads stand”

The store holds every thread, so any worker, or a separate tool, can list them. This lists the running threads of the invoices graph and reads each one’s lease:

static async Task ShowRunningAsync(EfCoreCheckpointStore store)
{
    var query = new ThreadQuery
    {
        States = new HashSet<CheckpointThreadState> { CheckpointThreadState.Running },
        GraphId = "invoices",
    };


    do
    {
        var page = await store.ListThreadsAsync(query);
        foreach (var thread in page.Threads)
        {
            var lease = await store.GetLeaseAsync(thread.ThreadId);
            Console.WriteLine(lease is null
                ? $"{thread.ThreadId}: running, no live lease"
                : $"{thread.ThreadId}: running, held by '{lease.Owner}' until {lease.ExpiresAt:HH:mm:ss} (token {lease.Token})");
        }


        query = query with { Cursor = page.NextCursor };
    }
    while (query.Cursor is not null);
}

ThreadQuery also filters by Metadata (every pair must match), GraphVersion, UpdatedAfter, UpdatedBefore and DueBy.

A page can hold fewer threads than the limit even when more follow. Only a missing cursor (null, or None in Rust) means the end, so always follow it. A running thread with no live lease has lost its run, and the recovery driver will continue it.

The strata CLI reads the same store. While worker-a was in Draft, strata thread list showed the thread:

Terminal window
strata thread list --storage invoices.db --status running --graph invoices
+-----------+--------------+-------------+---------+-------------+----------------------+
| Thread ID | Current Node | Sequence ID | State   | Waiting For | Updated At           |
+-----------+--------------+-------------+---------+-------------+----------------------+
| invoice-7 | Draft        | 0           | running | -           | 2026-10-30T14:49:47Z |
+-----------+--------------+-------------+---------+-------------+----------------------+

strata thread inspect shows who holds it, in the Lease line:

Terminal window
strata thread inspect invoice-7 --storage invoices.db
Thread: invoice-7
  Status:          running
  Thread state:    running
  Current node:    -
  Graph:           invoices version 1
  Lease:           held by 'worker-a' until 2026-10-30T14:49:51Z (token 1)
  Running:         next 'Draft'
  Hops:            0
  Next node:       Draft
  Checkpoint count: 1
  Created at:      2026-10-30T14:49:47Z
  Resumed at:      -
  State:
{
  "drafted_by": null,
  "invoice_id": "invoice-7",
  "sent": false
}

Limit each process with the run scheduler

Section titled “Limit each process with the run scheduler”

A RunScheduler bounds how many runs one process starts at once. It has a number of slots, strict priority with slots reserved for high priorities, and a concurrency limit per tenant, read from the run’s tenant metadata. A run waits for a slot before it loads or writes anything. A run that can’t get one returns NotAdmitted, a kind of Failed, and leaves the thread as it was.

var scheduler = new RunScheduler(new RunSchedulerOptions
{
    MaxConcurrent = 8,
    TenantLimit = 2,
    Reservations = [new SlotReservation(MinPriority: 5, Slots: 2)],
});


var options = new RunOptions
{
    Lease = new LeaseOptions("worker-a", TimeSpan.FromSeconds(30)),
    Scheduler = scheduler,
    Priority = 5,
    Metadata = { ["tenant"] = "acme" },
};

In a host, AddScheduler() registers one from Strata:Scheduler, and the StrataRunner and the drivers use it.

Here the process runs at most 8 runs at once. Two slots are kept for priority 5 and above, and each tenant gets at most two slots. Each replica has its own scheduler: Strata has no limit across processes. The total is the number of replicas times each one’s limit. Demo 29 shows priority, reservations, tenant limits and shedding (C#, Rust).

Make side effects safe to repeat

Section titled “Make side effects safe to repeat”

Leases protect the checkpoints, not the outside world. A stalled worker’s node can keep running next to the new owner’s until its next renewal or save. A recovery runs the interrupted node again. So a node that charges a card or sends an email can run twice.

Pass the node’s idempotency key, NodeContext.IdempotencyKey (Rust: idempotency_key()), to the system you call, so that it can ignore a repeat. At least once explains the key. Demo 20 recovers a real crash and charges the payment once, through a gateway that ignores a repeated key (C#, Rust).

Section titled “Related”