Skip to content
Datalium StrataDatalium Strata
These docs cover Strata 0.9.0-preview.1.It is a preview, and the API can still change before 1.0.

Retry failed nodes

In this guide you make a node that calls a flaky payment gateway. You retry it with exponential backoff, wait as long as the gateway asks when it rate-limits you, fall back to a manual-review node when retries run out, and stop attempts that hang. You also see what Strata saves when nothing recovers the node.

Before you start

Section titled “Before you start”

You need a project with Strata installed. See Installation. The guide builds on Graphs and on the commit rule.

The examples use this state:

sealed record OrderState(
    string OrderId, string CardToken, string Status, string? ReceiptId, bool Confirmed, string? ReviewNote);

Make the node fallible

Section titled “Make the node fallible”

Only a fallible node can fail in a way Strata handles. Add it with AddFallibleNode (Rust: add_fallible_node, or add_async_fallible_node for an async handler). A plain node can’t report a failure:

  • In C#, an exception from a plain node is treated as a bug. It propagates out of RunAsync, even if it is a NodeException.
  • In Rust, a plain node’s handler returns the new state, not a Result, so it has no way to fail.

A fallible node reports a failure as one of two kinds:

  • Transient: worth retrying, such as a timeout, a 503 or a 429. C#: throw NodeException.Transient(message). Rust: return Err(NodeError::Transient(message)).
  • Fatal: retrying won’t help, such as a declined card or bad input. C#: NodeException.Fatal(message). Rust: NodeError::Fatal(message). A fatal failure skips every retry.

In C#, only a NodeException is recovery-managed. Any other exception propagates out of RunAsync, as from a plain node. So catch your client’s errors in the node and translate them:

.AddFallibleNode("ChargePayment", async (order, ctx, ct) =>
{
    try
    {
        var receipt = await gateway.ChargeAsync(order, ctx.IdempotencyKey, ct);
        return order with { ReceiptId = receipt, Status = "charged" };
    }
    catch (GatewayException ex) when (ex.StatusCode is 429 or 503)
    {
        // Worth retrying. Pass the provider's Retry-After on, if it sent one.
        throw NodeException.Transient($"gateway unavailable ({ex.StatusCode})", ex.RetryAfter, ex);
    }
    catch (GatewayException ex)
    {
        // Retrying won't help.
        throw NodeException.Fatal(ex.Message, ex);
    }
})

GatewayException and GatewayError stand in for your own client’s error type.

Every attempt starts from the same committed state. A failed attempt’s partial changes never reach the retry, the fallback node or the checkpoint. In C# this relies on your state being immutable, such as a record changed with with.

Retry with backoff

Section titled “Retry with backoff”

A node’s RecoveryPolicy says what Strata does when the node fails. Attach it with SetRecoveryPolicy (Rust: set_recovery_policy):

var policy = RecoveryPolicy.Retry(2)
    .WithBackoff(TimeSpan.FromMilliseconds(200), TimeSpan.FromSeconds(2), BackoffJitter.None)
    .WithFallback("QueueForManualReview");


// ...
    .SetRecoveryPolicy("ChargePayment", policy)

The number in Retry(2) counts retries, not attempts: the node runs up to 3 times. The default policy, for a fallible node with none set, has no retries, no backoff and no fallback.

Without WithBackoff (Rust: with_backoff), a retry starts at once. With it, the n-th retry waits up to d = initial × 2^(n−1), capped at the maximum. The jitter decides the actual wait:

Jitter C# Rust Wait before a retry
Full, the default BackoffJitter.Full Jitter::Full A random time between 0 and d
Equal BackoffJitter.Equal Jitter::Equal A random time between d/2 and d
None BackoffJitter.None Jitter::None Exactly d

Full jitter spreads out many threads that fail at the same moment, so keep it in production. This guide uses None so the numbers in the output are stable. For a constant wait, set the initial delay equal to the maximum. A maximum below the initial delay is refused: C# throws ArgumentOutOfRangeException from WithBackoff, and Rust’s compile() returns CompilationFailed.

Pass on a provider’s Retry-After

Section titled “Pass on a provider’s Retry-After”

When a provider says how long to wait, such as a 429 with a Retry-After header, pass that hint with the transient failure. C# has an overload, NodeException.Transient(message, retryAfter, inner), which accepts a null hint. Rust has NodeError::transient_after(message, retry_after). The node above uses both.

The hint replaces the computed delay for the next retry. It is capped at the backoff’s maximum and isn’t jittered. With a maximum of 2 seconds, a provider that asks for 30 seconds gets 2. Only the failure being retried counts: a hint from an earlier attempt doesn’t carry over.

A policy without a backoff ignores hints and retries at once. Add a backoff whenever you pass hints.

Fall back to another node

Section titled “Fall back to another node”

WithFallback (Rust: with_fallback) names a node to run once the policy gives up: after a fatal failure, or after the last retry fails. The fallback node runs on the state from before the failed node started. It needs no ordinary edge into it, since the policy is its way in.

The fallback node’s context says why the node failed. Read ctx.LastFailure in C#, or ctx.last_failure() in Rust. In Rust, only an async handler receives a context, so make the fallback an async node.

.AddNode("QueueForManualReview", (order, ctx, _) =>
{
    var failure = ctx.LastFailure!;
    var note = $"{failure.Node} failed ({failure.Kind}) after {failure.Attempts} attempt(s): {failure.Message}";
    if (failure.RetryAfter is { } wait)
    {
        note += $"; asked to wait {wait.TotalSeconds:0} s";
    }


    if (failure.TimedOut)
    {
        note += "; timed out";
    }


    return ValueTask.FromResult(order with { Status = "manual_review", ReviewNote = note });
})

The failure describes the final failure: the node, its path, the kind, the message, the hint, the number of attempts, and whether the last attempt timed out. Only the fallback node sees it, on every attempt if the fallback is itself fallible. It isn’t saved, so if the run stops while the fallback runs and is resumed later, the fallback runs again without it.

Limit each attempt with a timeout

Section titled “Limit each attempt with a timeout”

A hung call never fails on its own. SetTimeout (Rust: set_timeout) limits each attempt:

.SetTimeout("ChargePayment", TimeSpan.FromSeconds(1))

An attempt that runs over is a transient failure with the message node '<name>' timed out after <n> ms. The policy retries it or falls back like any other transient failure, and each retry gets a fresh timeout. The backoff wait between attempts doesn’t count against the timeout.

Strata cancels the handler’s token and stops waiting for it, so pass the token to the calls you make. A sync handler can’t see the token: it runs on, and its result is thrown away. On a plain node, a timeout fails the run, since a plain node can’t have a policy. Rust’s compile() refuses a timeout on an inline node.

Watch the retries

Section titled “Watch the retries”

Every scheduled retry emits a RetryScheduled run event with the real delay. Pass an observer to see the waits:

var events = new RunEventChannel();
var result = await graph.RunAsync(order.OrderId, order, store, new RunOptions { Observer = events });


await foreach (var runEvent in events.Reader.ReadAllAsync())
{
    if (runEvent is RetryScheduled retry)
    {
        Console.WriteLine($"  retry scheduled after attempt {runEvent.Attempt}: wait {retry.Delay.TotalMilliseconds:0} ms ({retry.Error})");
    }
}

Run the graph against a gateway that fails three ways. Both languages print the same events. The C# run prints:

=== 1. Two 503s, then success ===
    [gateway] call 1
    [gateway] call 2
    [gateway] call 3
  retry scheduled after attempt 1: wait 200 ms (gateway unavailable (503))
  retry scheduled after attempt 2: wait 400 ms (gateway unavailable (503))
  Completed: OrderState { OrderId = ord-1001, CardToken = tok_visa_4242, Status = charged, ReceiptId = RCPT-ord-1001-3, Confirmed = True, ReviewNote =  }
=== 2. Rate limited (Retry-After: 30 s) ===
    [gateway] call 1
    [gateway] call 2
    [gateway] call 3
  retry scheduled after attempt 1: wait 2000 ms (gateway unavailable (429))
  retry scheduled after attempt 2: wait 2000 ms (gateway unavailable (429))
  Completed: OrderState { OrderId = ord-1002, CardToken = tok_visa_4242, Status = manual_review, ReceiptId = , Confirmed = False, ReviewNote = ChargePayment failed (Transient) after 3 attempt(s): gateway unavailable (429); asked to wait 30 s }
=== 3. The gateway hangs ===
    [gateway] call 1
    [gateway] call 2
    [gateway] call 3
  retry scheduled after attempt 1: wait 200 ms (node 'ChargePayment' timed out after 1000 ms)
  retry scheduled after attempt 2: wait 400 ms (node 'ChargePayment' timed out after 1000 ms)
  Completed: OrderState { OrderId = ord-1003, CardToken = tok_visa_4242, Status = manual_review, ReceiptId = , Confirmed = False, ReviewNote = ChargePayment failed (Transient) after 3 attempt(s): node 'ChargePayment' timed out after 1000 ms; timed out }

The first run recovers on its third attempt. In the second, the 30-second hint is capped at 2 seconds. In the second and third, the fallback ran. It has no outgoing edge, so the run ends after it as Completed. ReceiptId is empty and SendConfirmation never ran.

When retries run out

Section titled “When retries run out”

Without a fallback, a failure the policy can’t recover ends the run. Strata saves a failed checkpoint and returns:

  • C#: GraphExecutionResult<TState>.NodeFailed, a kind of Failed, with Node, Attempts and Message.
  • Rust: GraphExecutionResult::Failed(StrataError::NodeFailed { node, attempts, message }).

A declined card with RecoveryPolicy.Retry(2) and no fallback makes one attempt, since the failure is fatal. The checkpoint holds the last committed state, names the failed node as the next node, and records the error:

=== 4. Card declined, no fallback ===
    [gateway] call 1
  NodeFailed: 'ChargePayment' after 1 attempt(s): card declined: expired
  checkpoint: Failed, next node ChargePayment
  metadata.attempts: 1
  metadata.error: card declined: expired
  metadata.error_kind: fatal

When the final failure carried a hint, the checkpoint also records it as strata.retry_after_ms.

A failed thread isn’t lost. Once you fix the cause, resume it. The resume runs the failed node again from the last committed state, with its attempts counted from 1, and the signal’s overrides correct the state. A failed checkpoint can be resumed once. See Statuses. Demo 08 resumes its declined order with a new card:

var fix = HumanSignal.Approve("support@shop.example").WithOverride("CardToken", NewCard);
var resumed = ExpectCompleted(await resumeGraph.ResumeAsync("order-1003", fix, store2));

To find failed threads, run strata thread list --status failed. Add --meta error_kind=transient to keep only transient failures. See the CLI reference.

Keep in mind

Section titled “Keep in mind”
  • Side effects repeat. Each retry calls the gateway again. Pass ctx.IdempotencyKey (Rust: ctx.idempotency_key()) to the service you call, as the node above does. The key is the same on every retry of the node, so the service can ignore a repeat. See At least once.
  • The wait is in memory. If the process crashes during a backoff wait, the run is lost like any other crash. See Known limitations.
  • Cancellation isn’t a failure. If you cancel a run or its deadline passes, the backoff wait stops at once. The run ends Cancelled, with no retry and no fallback.
  • Policies are checked when you compile. Compile() refuses a policy on a node that isn’t fallible, or a fallback that doesn’t exist. C# throws InvalidOperationException: recovery policy set for 'ChargePayment', which is not a fallible node. Rust returns CompilationFailed with the same message.
  • Groups and subgraphs are fallible too. A parallel group’s policy re-runs every branch, and a subgraph’s re-runs the child graph.

Test without waiting

Section titled “Test without waiting”

The backoff, node timeouts and the deadline all wait on the run’s clock. In C#, set RunOptions.TimeProvider to a FakeTimeProvider, from the Microsoft.Extensions.TimeProvider.Testing package, and move it forward. In Rust, start the runtime with tokio’s clock paused (start_paused, from tokio’s test-util feature). Demo 08 runs its 4 seconds of backoff this way in milliseconds.

Section titled “Related”