Skip to content
Datalium StrataDatalium Strata
These docs cover Strata 0.9.0-preview.1.It is a preview, and the API can still change before 1.0.

Stop runaway loops

A guarded loop compiles even if its guard never turns false. In an agent or a “refine until good” loop, each pass may be a model call that costs money. In this guide you put a hop limit on such a graph, watch it stop a loop that never ends, read what it leaves behind, and let the stopped thread go on. The hop limit explains what counts as a hop.

Before you start

Section titled “Before you start”

You need a project with Strata added. See Installation. The samples use the in-memory checkpoint store, so you don’t need a database.

Set a limit when you compile

Section titled “Set a limit when you compile”

The sample graph drafts an essay, reviews it, and drafts again while the score is below MinScore. The reviewer stands in for a model call, and it never scores above 7. With MinScore at 8, the guard on the back edge never turns false.

Set the limit when you compile. In C#, pass CompilationOptions with MaxNodeHops to Compile. In Rust, call compile_with with CompileOptions::new().with_max_node_hops(n). There is no limit by default.

using Datalium.Strata.Graph;
using Datalium.Strata.Interruption;
using Datalium.Strata.Storage;


var store = new InMemoryCheckpointStore();


var graph = new StateGraph<Essay>()
    .AddNode("Draft", essay => essay with { Revision = essay.Revision + 1 })
    .AddNode("Review", essay => essay with { Score = Reviewer.Score(essay) })
    .AddNode("Publish", essay => essay with { Published = true })
    .AddEdge("Draft", "Review")
    .AddConditionalEdge("Review", "Draft", essay => essay.Score < essay.MinScore)
    .AddEdge("Review", "Publish")
    .Compile(new CompilationOptions { MaxNodeHops = 20 });


// ...


record Essay(int MinScore, int Revision = 0, int Score = 0, bool Published = false);


static class Reviewer
{
    // A stand-in for a model call. It never scores above 7.
    public static int Score(Essay essay) => Math.Min(4 + essay.Revision, 7);
}

A limit below 1 is refused when you compile. C# throws ArgumentOutOfRangeException (“MaxNodeHops must be at least 1”). Rust returns StrataError::CompilationFailed("max_node_hops must be at least 1").

Choose a limit

Section titled “Choose a limit”

Count the hops on the longest path you expect to be legitimate, then set the limit a few times higher. Here, an essay that needs three revisions takes 7 hops: three drafts, three reviews and the publish. A limit of 20 leaves room for a slow essay and still stops a stuck one.

Keep two things in mind:

  • The limit applies to the thread, not to one run. The count is saved in each checkpoint and carries across pauses and resumes. A loop that waits for a person on every pass still trips at the same hop as one that never pauses.
  • A retry of the same node isn’t a new hop, so retries don’t eat into the limit. A fallback node is a hop of its own.

Detect the trip

Section titled “Detect the trip”

When the next node would pass the limit, it doesn’t start. Its handler never runs, and the run ends with a failed result. With a limit of 20, the 20th node runs, and the node that would be hop 21 is refused.

In C#, the result is GraphExecutionResult<TState>.HopLimitExceeded, with Node, Hops and Limit. It is a kind of Failed, so code that already handles Failed catches it. Put its case before the Failed one. In Rust, the result is GraphExecutionResult::Failed(StrataError::HopLimitExceeded { node, hops, limit }).

var result = await graph.RunAsync("essay-1", new Essay(MinScore: 8), store);


switch (result)
{
    case GraphExecutionResult<Essay>.Completed done:
        Console.WriteLine($"Published after {done.State.Revision} revisions");
        break;
    case GraphExecutionResult<Essay>.HopLimitExceeded tripped:
        Console.WriteLine($"Stopped: node = {tripped.Node}, hops = {tripped.Hops}, limit = {tripped.Limit}");
        Console.WriteLine(tripped.Message);
        break;
    case GraphExecutionResult<Essay>.Failed failed:
        Console.WriteLine($"Failed: {failed.Message}");
        break;
}

The output:

Stopped: node = Draft, hops = 20, limit = 20
hop limit of 20 reached: node 'Draft' would be hop 21

No recovery policy sees the trip, so it isn’t retried.

Read the failed checkpoint

Section titled “Read the failed checkpoint”

Before it returns, the run saves a Failed checkpoint at the refused node, with the last committed state. Its metadata has the message in error, error_kind set to hop_limit, and the count in strata.hops. It has no attempts, because nothing was attempted.

var checkpoint = store.Load("essay-1")!;
Console.WriteLine($"{checkpoint.Status} at {checkpoint.NextNode}");
Console.WriteLine($"error: {checkpoint.Metadata!["error"]}");
Console.WriteLine($"error_kind = {checkpoint.Metadata["error_kind"]}, strata.hops = {checkpoint.Metadata["strata.hops"]}");
Console.WriteLine($"Revision: {checkpoint.FrozenState!["Revision"]}");

The output:

Failed at Draft
error: hop limit of 20 reached: node 'Draft' would be hop 21
error_kind = hop_limit, strata.hops = 20
Revision: 10

With a durable store, strata thread inspect shows the same checkpoint, with the count on its Hops line. The trip is also a HopLimitExceeded run event, it increments the strata.graph.hop_limit_exceeded counter, and the run span gets error.type set to hop_limit, so you can alert on it.

Let a stopped thread go on

Section titled “Let a stopped thread go on”

Resuming the thread as it is trips again at once. The resume re-enters the refused node, which counts as hop 21, so no node runs. To go on, pass a higher limit for that resume: RunOptions.MaxNodeHops in C#, or RunConfig::with_max_node_hops with resume_with in Rust.

The per-run limit replaces the compiled one for that run or resume only, and isn’t saved. The count carries on from strata.hops. A per-run limit can also be lower than the compiled one, for example in a test. On a graph compiled without a limit, it sets one.

Here an editor lowers the bar with a typed override (see Answering with a human signal) and allows ten more hops:

// As it is: it trips again before any node runs.
var again = await graph.ResumeAsync("essay-1", HumanSignal.Approve("editor"), store);
if (again is GraphExecutionResult<Essay>.HopLimitExceeded stillTripped)
{
    Console.WriteLine($"Tripped again: {stillTripped.Message}");
}


// Lower the bar, and allow ten more hops for this resume.
var signal = HumanSignal.Approve("editor").WithOverride("MinScore", 7);
var resumed = await graph.ResumeAsync("essay-1", signal, store, new RunOptions { MaxNodeHops = 30 });
if (resumed is GraphExecutionResult<Essay>.Completed { State: var essay })
{
    Console.WriteLine($"Published after {essay.Revision} revisions");
}

The output:

Tripped again: hop limit of 20 reached: node 'Draft' would be hop 21
Published after 11 revisions

The resume ran three nodes, hops 21 to 23: one more draft, a review that now passes, and the publish.

Limits in subgraphs

Section titled “Limits in subgraphs”

Each compiled graph enforces its own limit over its own nodes. A subgraph node counts as one hop in the parent, and the child graph counts its nodes against the limit it was compiled with. A child with a loop needs its own limit.

When a child trips, the subgraph node fails with a fatal error, so its recovery policy doesn’t retry it. Unless the policy has a fallback node, the parent’s result is NodeFailed for the subgraph node, not HopLimitExceeded. Its message reads like subgraph 'Refine' hit its hop limit: hop limit of 10 reached: .... The child’s checkpoint is kept inside the parent’s failed checkpoint.

The per-run limit reaches the outermost graph only. A child always uses its compiled limit.

What the limit doesn’t cover

Section titled “What the limit doesn’t cover”
  • Paths that can’t end. Compiling doesn’t check that every path reaches an end (STRATA_ERR_NO_TERMINAL isn’t built). The hop limit is the guard for that case. See Known limitations.
  • The graph definition. The exported definition doesn’t record the limit, so strata graph validate can’t see it.
  • Visits to one node. The limit counts every node the graph starts. There is no per-node limit.
  • Time, tokens and cost. To bound time, set a deadline: RunOptions.Deadline in C#, RunConfig::with_deadline in Rust.
  • Threads that were already running. A checkpoint saved before the graph had a limit has no strata.hops, so its thread counts from 0 when it next resumes.
Section titled “Related”