Model-First Orchestration for Distributed 3D Workflows
An MBSE-inspired reference architecture for governing, versioning, and durably executing distributed workflow definitions.
Distributed 3D processing is rarely difficult because one worker is difficult. It becomes difficult when thumbnail rendering, metadata extraction, analysis, policy checks, and publishing must change independently while callbacks arrive late, twice, or not at all.
For architects evaluating that problem, a useful approach is to treat a workflow definition as a reviewed, validated, immutable, version-pinned executable artefact. This is MBSE-inspired, not a claim that YAML is formal MBSE or that configuration replaces every other authority. MBSE is commonly described as using formalised models to support system requirements, design, analysis, verification, and validation; this design borrows its model-first principle for one bounded concern: orchestration (SEBoK).
The following is an illustrative reference architecture for Mesh-Sync-style 3D model workloads, not a production case study. Its workflow DSL is YAML-encoded, but its safety comes from the surrounding contracts, compiler, durable state, and operational controls~not YAML alone.
The model is important; it is not the whole system
A run’s behaviour is jointly defined by:
- an immutable workflow definition, or compiled-plan digest;
- the DSL schema, compiler, and engine versions;
- versioned action and worker input/output contracts;
- durable run state and transition history; and
- the external domain systems that remain authoritative for their side effects.
For example, a catalogue service owns whether a model is publishable; the orchestrator records that an action was requested or accepted. An object store owns objects and their access policy. Redis may own durable orchestration state in this design. Elasticsearch is a telemetry projection, never the source of run state.
Definitions should be owned by a trusted, reviewed release process. Authors need RBAC; actions, queues, and workers need allowlists; and admission limits should bound fan-out, retries, timeouts, payload size, and expression complexity. Where tenants share the control plane, both execution context and object access need tenant isolation. A model describes only the primitives the platform has deliberately exposed.
That last boundary matters. Adding a stage can be a definition-only change only when a compatible, already-registered action or worker contract exists. A new side effect, queue, result schema, or compensation policy is a platform change as well.
A workflow/control-flow graph, not a disguised list
Calling this structure a DAG is incomplete: success, failure, and decision routes are guarded control-flow transitions. A clearer term is a workflow/control-flow graph. It has an explicit entry point, named success and failure edges, decision edges with a fallback, durable forks and joins, finalisation, and declared terminal outcomes.
Here is a deliberately small DSL example. It assumes the listed actions and workers have already been registered with compatible contracts.
name: model-ingest
version: "2.3.0"
entry: cache-lookup
stages:
- name: cache-lookup
type: action
action: cache.lookup@1
input: { modelId: "${context.modelId}" }
on_success: cache-route
on_failure: failed
- name: cache-route
type: decision
on_evaluation_error: failed
branches:
- when: 'stages["cache-lookup"].result.hit == true'
next: publish
- when: "true" # required fallback
next: render-thumbnail
- name: render-thumbnail
type: worker
queue: rendering
worker: thumbnail.render@3
input: { modelId: "${context.modelId}" }
execution_deadline_ms: 300000
on_success: derive-fork
on_failure: failed
- name: derive-fork
type: fork
branches: [semantic-analysis, technical-metadata]
- name: semantic-analysis
type: worker
queue: analysis
worker: semantic.analyse@4
input: { modelId: "${context.modelId}" }
on_success: derive-join
on_failure: failed
- name: technical-metadata
type: worker
queue: metadata
worker: metadata.extract@2
input: { modelId: "${context.modelId}" }
on_success: derive-join
on_failure: failed
- name: derive-join
type: join
required: [semantic-analysis, technical-metadata]
acceptable_terminal_states: [completed]
on_success: publish
on_failure: failed
- name: publish
type: action
action: catalogue.publish@2
input: { modelId: "${context.modelId}" }
on_success: completed
on_failure: failed
terminal_policy:
fork_branch_failure: request-cancellation-and-fence
late_callbacks: reject-and-record
finalisers: declared-order
finaliser_failure_terminal: failed
finalisers: [audit.run-terminal@1]
terminals: [completed, failed]Interpolation is a distinct, typed runtime binding step. Missing, null, and wrong-type values are different failures; a string scan for leftover ${...} is not definition-time validation. The decision syntax deliberately uses bracket access for the hyphenated stage name. Jexl avoids direct JavaScript evaluation, but it is not thereby a sandbox: safety depends on trusted authorship and the context, functions, and transforms made available to expressions.
flowchart LR
A[entry: cache-lookup] --> B{cache-route}
B -->|hit| P[publish]
B -->|fallback| T[render-thumbnail]
T --> F[fork]
F --> S[semantic-analysis]
F --> M[technical-metadata]
S --> J[join: all completed]
M --> J
J --> P
P -->|success| CI[completed intent]
A -. failure .-> FI[failed intent]
B -. evaluation error .-> FI
S -. branch failure .-> FI
M -. branch failure .-> FI
J -. join failure .-> FI
T -. failure .-> FI
P -. failure .-> FI
FI --> CF[request cancellation of active fork siblings; fence late callbacks]
CI --> Z[finalise in declared order]
CF --> Z
Z -->|completed intent and all finalisers succeed| C[completed]
Z -->|failed intent or finaliser failure| X[failed]A fork records which branch entries it released. A join with all persists each branch completion and atomically releases its successor once all required branches are in acceptable terminal states. It does not use an in-memory Promise.all: workers may complete by asynchronous callbacks after the process that initiated the fork has gone away.
In this example, failed is a terminal intent, not an immediately exposed terminal result. A cache-lookup failure, a cache-route decision-evaluation error, either derive-branch failure, an unsatisfied derive-join, or a publish failure atomically sets that intent. On a derive-branch failure, the engine requests cooperative cancellation of any active sibling and fences its attempt before finalisation; it records the request and any acknowledgement, but does not assume remote work has stopped. A sibling can therefore still consume remote resources or complete its external effect. Its later non-finaliser callback is rejected from changing the run and recorded as late after terminal intent. The same reject-and-record rule applies to every ordinary callback that arrives after either terminal intent.
After an intent is recorded, finalisers run once and sequentially in their declared order. The engine records every finaliser outcome and attempts later finalisers even if an earlier one fails. It exposes completed only when the intent was completed and all finalisers succeeded; a failed intent or any finaliser failure exposes failed. Thus finalisation failure is retained in durable history rather than silently converting a completed run into success.
Useful stage states are queued/waiting, running, retry-scheduled, completed, failed, timed-out, skipped or cancelled, and finalising; the run also records a non-terminal completed or failed intent while finalisers execute. The plan must state which transitions are legal, which terminal intent a failure route reaches, and how finalisation resolves it.
Compile and admit definitions before starting runs
The compiler should produce a canonical compiled plan and digest, not merely parse YAML. Its checks have different jobs:
| Layer | Questions it answers | What it cannot establish |
|---|---|---|
| Schema/admission | Are stage shapes discriminated? Are required fields, bounds, and allowed properties valid? | Whether an action is safe or useful |
| Graph semantics | Are names unique; all route, branch, handler, and finaliser references resolvable; entry reachable; cycles absent; decision fallback present; forks/joins well formed? | Whether an external effect succeeds |
| Contract/type | Does the action/worker registry offer the named version, and do declared input/output schemas compose? | Whether a future payload satisfies its business rules |
| Runtime binding/result | Do actual values exist, distinguish null, and satisfy type/result schemas? | Whether a side effect was correct in the outside world |
JSON Schema is valuable for shape; it does not prove business correctness or side-effect safety. Graph compilation should traverse every edge namespace~not just on_success~and reject unreachable stages, cycles, a join with the wrong branch set, and a decision without a deterministic fallback. The resulting plan makes the engine’s control flow inspectable and gives review tooling a stable object to visualise.
Pin versions before dispatch
A version string is not enough if its contents can change. At run creation, resolve an approved definition to an immutable definition or compiled-plan digest, then persist that digest alongside DSL/compiler/engine versions and every action/worker contract version. Dispatches and callbacks carry the same identity.
Activation creates a new immutable version; it never edits an active version in place. New runs may select the new approved plan while old plans stay available until their runs drain. Rollback means activating a previous immutable plan for new runs, not rewriting history. During that drain window, workers must accept pinned job and input-contract versions; callback ingress and result validation must accept pinned callback and result-schema versions, or tested, versioned adapters must bridge them.
Delivery is at-least-once, so identity is part of the protocol
BullMQ is useful as a queue boundary, but queue retries are not the entire execution policy. Its attempts setting concerns failing job processing; a missing application callback does not automatically become a retry (retries). Stalled-job handling is also a separate worker concern (stalled jobs).
This reference design places a dispatch adapter between BullMQ and the remote worker. BullMQ owns retries for adapter processor exceptions and stalls while establishing remote acceptance. A BullMQ job has already been assigned to a processor when that processor throws or stalls; that fact does not establish that the remote worker accepted the envelope. The adapter sends a deterministic remote execution ID and idempotency key, and records remote acceptance before completing the queue job. If an adapter crash leaves acceptance uncertain, a redelivery queries or repeats the acceptance handshake with that same identity; it must not create a new remote attempt merely because BullMQ redelivered the job. Once acceptance is recorded, the adapter completes the BullMQ job and BullMQ is no longer a retry owner for that remote side effect.
The orchestration layer owns application-level missing callbacks and persisted execution deadlines after remote acceptance. It may create a new, fenced stage attempt only after applying the cancellation-and-fencing policy to the prior attempt; it does not also ask BullMQ to retry the accepted dispatch. This makes the owners non-overlapping: BullMQ retries pre-acceptance dispatch processing, while the orchestrator retries post-acceptance execution progress. Assume at-least-once delivery for jobs and callbacks. Every job and callback envelope should contain a run ID, stage ID, attempt and fencing token, definition digest, contract version, deterministic job/event ID, deadline, and result schema version. Workers and actions must be idempotent or receive an idempotency key; BullMQ makes the same recommendation for jobs that may be retried (idempotent jobs).
On callback, the state store compares run, stage, attempt, and fencing token before accepting a result. A matching duplicate can be acknowledged without re-releasing a successor. A stale or reordered callback from an earlier attempt is recorded or rejected, but cannot overwrite the current result. Once terminal intent exists, ordinary callbacks are rejected from state transition and recorded as late, even if their token otherwise matches. TLS and HMAC can authenticate a correctly verified canonical payload; they do not provide freshness, replay prevention, or idempotency. Add timestamp/nonce replay windows, key rotation, attempt binding, strict output schemas, and redaction of sensitive values in telemetry.
One atomic record is not a distributed transaction
Redis Lua scripts execute atomically with respect to Redis, which is enough to compare and transition one run record. The following is simplified Lua pseudocode, not a complete implementation:
-- Expected state and attempt/fencing token are supplied by the callback.
if record.state ~= expected_state or record.fence ~= expected_fence then
return "duplicate-or-stale"
end
record.state = "completed"
record.result = validated_result
record.fence = next_fence
persist(record)
if join_is_now_satisfied(record) and not record.successor_released then
record.successor_released = true
persist_scheduling_outbox(record.run_id, record.join_successor, next_fence)
end
return "accepted"It cannot atomically coordinate Redis, BullMQ, MinIO, worker side effects, callbacks, and Elasticsearch. Persisting a scheduling/outbox record in the winning transition narrows the state/enqueue gap; a dispatcher retries that record with a deterministic job ID, and reconciliation repairs stranded records. The same pattern emits telemetry from accepted transitions rather than hoping direct writes all succeed.
Timeouts need three clocks: a queue deadline, an execution/heartbeat deadline, and an end-to-end run deadline. Persist them. One distributed claimant transitions an expired current attempt to timed-out; a timeout does not stop remote work. Before retrying an effect, fence or cooperatively cancel the prior attempt and reject late results. BullMQ documents an application-level timeout pattern separately from job retries (timeout jobs); an interval scanner alone is not sufficient without durable claims, fencing, and recovery after crashes.
Cache and observe at the right boundary
A cache key such as modelId-v1 is version-keyed, not content-addressable. For reusable stage results, compute a key from canonical input bytes plus stage implementation/contract version, parameters, and tenant identity where relevant. Store an absolute expiry in validated metadata and check it on read, or configure a matching MinIO lifecycle expiration rule (lifecycle expiration). Object metadata named “TTL” does not expire an object by itself.
Only an explicit not-found is a cache miss. Denied access, service outage, corruption, and invalid JSON should surface as distinct outcomes. Validate cached results against the current output contract, route a valid hit around the worker, and consider per-key locking or request coalescing to avoid a stampede. Cache access controls must not become a cross-tenant data path. MinIO’s putObject API accepts metadata, but metadata is not a lifecycle policy (API).
For observability, write a durable transition/event outbox and asynchronously bulk-index a sanitised projection into Elasticsearch. Elasticsearch bulk requests have item-level outcomes that the indexer must handle (Bulk API); search is near-real-time rather than an immediate read-after-write guarantee (near-real-time search). Use deterministic event IDs, retries with backoff, backpressure, and a failure destination. Metrics, traces, alerts, and ingestion-health monitoring are separate concerns. Do not put raw model data, secrets, or uncontrolled high-cardinality payloads in the projection.
Decide whether to own this control plane
| Approach | Good fit | Cost or limit |
|---|---|---|
| Explicit application code/state machines | Small, stable, mostly linear flows | Change couples orchestration to deployment |
| Typed, code-first workflow library | Engineers want compiler help and code review | Less approachable for governed declarative changes |
| Established durable workflow engine | Timers, recovery, history, human waits, and tooling matter | Operational and conceptual adoption cost |
| Event choreography | Independent services can converge without central flow control | Global progress, retries, and compensation are harder to see |
| Custom DSL and engine | Workflow variety, governance, constrained deployment, or unusual DSL needs outweigh ownership cost | The team owns timers, joins, history, migration, compatibility, recovery, and tooling |
Do not build this DSL for small stable linear flows, when most changes require new code anyway, or where latency leaves no room for a control plane. It is also a poor fit for non-idempotent, uncompensatable effects; untrusted authors without isolation; loops, dynamic fan-out, or human waits beyond the metamodel; teams unable to operate a control plane; or cases where an established orchestrator already fits.
Verify failure paths before migration
Treat the compiler and runtime as products. Use golden and property tests for validation and compilation; malformed graphs, decision fallbacks, joins with failure/cancellation, and contract mismatch tests; and integration tests for duplicate, stale, and reordered callbacks. Exercise crash windows before and after state, enqueue, and event writes; timeout fencing; cache outage, corruption, and expiry; rolling worker/version upgrades; reconciliation; and operator re-drive.
Migration starts by inventorying side effects and establishing contracts and idempotency. Shadow or replay representative runs with effects disabled, then canary only new runs. Pin and drain old definitions, retain rollback and reconciliation paths, and make the operational status of every run explainable from durable history.
Conclusion
For changes expressible through existing compatible primitives, a model-first workflow can make orchestration more reviewable and modifiable. It does not make YAML the sole authority, and it does not remove distributed-systems failure modes. It introduces a language and a durable control plane~both of which must be versioned, secured, tested, and owned.