A workflow engine is easy to build for the happy path. The hard part is the unhappy one: a worker dies halfway through a step, a deploy replaces every server while a run waits on a human, someone edits the workflow that run is executing, a provider times out, a retry fires twice. The Verkko orchestrator treats those events as the normal operating condition rather than as exceptions. This article explains the mechanism behind each guarantee, names the failure it removes, and says what it means for the business running on top of it. It also says where the guarantees stop.
The failure model in one table
| What goes wrong | What the engine does | What it means for you |
|---|---|---|
| Workflow edited while a run is waiting | The full definition is pinned onto the run at start | An approved run never resumes into logic nobody reviewed |
| Worker dies mid-run | Lease expires, recover_orphaned_runs() resumes from the last completed step | Work finishes without a person noticing and re-triggering it |
| Approval takes two days | The run parks on a durable timer and releases the worker | A deploy during the wait costs nothing |
| Transient provider error | Classified, retried with exponential backoff and jitter | Fewer pages for blips, no retry storms |
| Retry of a step that already sent something | Side effect is claimed before it fires; a second attempt stops instead of repeating | No duplicate email, payment or CRM write reaches a customer |
| Run loops or overspends | Hard budgets on cost, tokens, steps and wall-clock time | A bad workflow has a bounded blast radius |
1. A run is pinned to the definition it started with
When a run starts, the engine copies the whole workflow definition onto the run record. It does not store a pointer to "the current version". The start event carries the definition's version and hash, so a run can always be tied to the exact logic it executed. Editing a workflow while a run waits on a 48-hour approval therefore cannot change what that run resumes into.
The same pinned copy is what the live progress view reads, so a run in flight keeps showing the plan it is actually executing even after the definition is edited.
Business impact. Change management stays honest. The approver signed off on a specific sequence of steps, and that sequence is what runs. A hotfix to a workflow applies to new runs and never silently rewrites commitments already in flight, which is the property compliance reviewers ask about first.
2. A step log, and state that is never overwritten
After every step the engine appends a record to the run's step log (orchestration_runs/{run_id}/steps/{seq}) and writes the run's progress: context snapshot, current step, step counters, cost and token totals. The run record is updated with a compare-and-set against the expected state. If another worker has moved the run in the meantime, the write is rejected as a stale write and the late worker backs off, because whoever holds the run now has fresher state.
Step outputs are immutable. When a step runs a second time, for example in an approval and redraft loop, the previous output moves into a history list (addressable as step_1.history[0]) and the latest value stays at step_1.output. Context has a hard size cap, 1 MiB by default, matching the document limit of the datastore, so a runaway step fails loudly instead of corrupting the record.
Business impact. Every decision a run made is reconstructable after the fact. When someone asks why a quote was sent or a ticket was escalated, the answer is in the step log, not in a log search across servers. Support and audit stop depending on the memory of whoever built the workflow.
3. Leases, heartbeats and recovery
Only one worker may drive a run at a time. Before it advances a run, a worker acquires a lease (acquire_lease, 120 seconds by default and configurable per engine). If the lease is held by someone else, the call returns without doing any work. The worker renews the lease with a heartbeat after every step and releases it when it exits, on every exit path.
If a worker dies, nothing renews its lease. A run still marked running or pending with an expired lease is, by definition, orphaned. recover_orphaned_runs() finds those runs and continues each one from the last completed step in its step log. It resumes; it does not restart. The entry point is tick(), which is idempotent and safe to call concurrently from any number of processes. It runs from the existing scheduler on a short interval, so there is no dedicated recovery service to deploy, monitor or forget.
Business impact. Infrastructure failure stops being a customer-visible event. A rolling deploy, a spot-instance reclaim or an out-of-memory kill delays a run by roughly one lease window plus a scheduler tick. It does not strand it. You also avoid the quiet failure mode of homegrown automation, where a crashed worker simply means the run stops forever and the first person to notice is the customer.
4. Durable timers instead of sleeping
The engine never calls sleep() to wait. A human approval gate, or a retry backoff longer than a short pause, parks the run in the store with a resume_at timestamp and releases the worker. process_due_runs() later picks up runs whose timer has passed and handles the two cases differently:
- A parked retry backoff is resumed.
- An expired human gate applies the step's
on_timeoutpolicy: approve, reject, escalate or fail. The behaviour when nobody answers is a declared part of the workflow, not an accident of how long a process happened to live.
Business impact. Waiting is free. A run parked for two days holds no thread, no connection and no worker, so approval-heavy processes such as procurement, contract review and refunds scale with the number of decisions, not with the number of idle processes. And because the timeout outcome is explicit, an unanswered approval resolves the way your policy says it should rather than stalling silently.
5. Retries that know what they are retrying
Retry is configured per step:
{"retry_policy": {"max_attempts": 3, "backoff": "exponential",
"initial_delay_ms": 1000, "max_delay_ms": 30000,
"retryable_errors": ["timeout", "rate_limit", "5xx"],
"jitter": true}}
Errors are classified before anything is retried, because retrying the wrong thing is worse than not retrying. Timeouts, rate limits and server errors are transient and are retried. Validation errors route to correction instead. Everything else halts the run: permission denied, not found, budget exceeded, cancelled, a fatal error and, importantly, a duplicate side effect. Jitter spreads retries out so that a provider recovering from an outage is not hit by every run at the same instant.
Business impact. Blips are absorbed without anyone being paged, and genuine faults such as a revoked credential fail fast instead of burning attempts and money. Each halted run carries a classified reason, so triage starts from "permission" or "not found", not from a stack trace.
6. Side effects are claimed before they fire
Resuming after a crash means the step that was in flight runs again. For a read-only step, such as a search or a lookup, that is harmless and the step stays freely retryable. For a step that sends, pays or writes, running it twice is exactly the failure customers see. So before a side-effecting tool executes, the engine records a claim for that run, step and loop iteration. A retry of the same iteration finds the claim already taken and does not call the tool again. It stops the run with a duplicate_side_effect error and preserves full state. The message says plainly that the outcome of the earlier call is unknown and needs manual confirmation.
Two details matter here. The claim is recorded before the call, so the worst case is a missed send that a human can see in the step log, never a duplicate the customer sees. And the tool registry, not the workflow author, decides whether a tool has side effects: an author can mark a step as more cautious but never less. A fresh loop iteration gets its own claim, so an approve-and-redraft loop still runs.
Business impact. The system chooses the failure you can fix quietly over the one you have to apologise for. A worker crash between "email handed to the provider" and "step recorded complete" produces a flagged run for a person to check, not a customer receiving the same message twice. For finance and customer-facing workflows that trade is the right default.
7. Budgets: a bounded blast radius
Every run carries ceilings, checked as it executes: max_cost_eur, max_tokens, max_total_steps (default 200) and max_wall_clock_s. Each step also has max_executions (default 25), because without it an approval and redraft loop that keeps getting rejected never ends. Exceeding a budget fails the run with a classified, non-retryable reason rather than degrading quietly.
Business impact. A misconfigured workflow, or an agent stuck in a loop, has a known worst-case cost before it ships. Finance gets a number instead of an open-ended exposure, and engineering stops needing a manual kill switch for the common runaway.
8. Observability that survives the failures above
The engine emits a typed event stream (run.started, step.started, step.completed, step.failed, step.retrying, run.waiting, run.paused, run.resumed, run.completed, run.failed, run.cancelled). Trace context is W3C-shaped, so it can be handed to OpenTelemetry later without re-instrumenting. Payloads carry hashes of step inputs and outputs, never the payloads themselves, because step data can contain customer personal data and be very large. A hash is enough to tell whether a replay produced the same result.
For people watching a run, the live view is built from persisted state and delivered two ways: an in-process broker for instant updates, and a store poller (every 1.5 seconds by default) because the task holding the browser connection is usually not the task driving the run. Subscriber queues are bounded at 200 frames, so a slow browser cannot pin memory. A full snapshot is sent on connect and on every change, so a reconnect after a deploy or a sleeping laptop redraws the true picture rather than a half-drawn one.
Business impact. Operators can see where a run is and what it is waiting on, including a gate waiting on someone who was never told, which is the main way approval work silently stalls. Logs hold no customer data, which keeps them out of scope for most data-protection reviews.
Where the guarantees stop
- It is not exactly-once. After a crash the in-flight step runs again. Read-only steps are safe to repeat. Side-effecting steps are protected by the claim and stop for a human instead of repeating, which means some failures need a person to confirm what happened.
- Durability depends on the store. Runs persist in Firestore. If it is unreachable at startup the engine falls back to an in-memory store, logs a warning that runs will not survive a restart, and carries on, which suits local development and tests but is not durability. Treat that warning as an incident in production.
- Recovery is bounded by the lease and the scheduler. A dead worker's run resumes after its lease expires and the next tick, so recovery takes seconds to minutes, not milliseconds. The lease is renewed after each step, so lease length should be sized to your longest single step.
- It protects the run, not the tools. A downstream system that is wrong or down is still wrong or down. The engine makes sure the run stops, retries or escalates correctly instead of continuing blind.
The short version for a buyer
Ask any automation vendor four questions: what happens to a run when the server dies, when the workflow is edited mid-run, when a retry follows a send, and when an approval takes two days. For Verkko the answers are: it resumes from the last completed step, it keeps running the version that was approved, it refuses to repeat the send and hands the case to a person, and it parks for free and applies the timeout policy you declared. Those answers are enforced in the engine, not left to convention.
To see this on your own workflows, book a demo. For the governance layer that sits underneath every agent action, read the Baton page.