From Pipelines to Governed Contracts: Keeping Worker Platforms Coherent

Worker fleets rarely fail all at once. They drift.

The first worker sends JSON to a queue, processes it, and writes a result. The next workers add different payloads, callbacks, large-file timeouts, another language, and field names that look alike but mean different things. Nothing is broken in isolation; the system simply becomes harder to reason about every week.

That is the problem this MeshSync-inspired target architecture addresses. Its thesis is not that schemas alone make a platform safe. A heterogeneous worker platform remains coherent when messages are governed contracts, with generated bindings where they earn their keep, explicit orchestration, validation, observability, and an honest distinction between design, implementation, testing, and operation. MeshSync’s public organisation and product landing page provide the domain context for this case study: 3D assets, processing, storage, and marketplace-facing work (GitHub, landing page).

Status of this case study ~ 28 August 2026

State	What public evidence supports
Design artefact	The target architecture, examples, and proposed workflow inventory in this article.
Implemented	Not established for the worker platform or its individual capabilities.
Integration-tested	Not established.
Deployed / operated	Not established; this article makes no production-traffic, scale, or outcome claim.
Planned	Illustrative workers and workflows, subject to admission and deployability checks.

“Production-oriented” below means an intent to make the design operable if it is built, not evidence that it is deployed. This is the worker-side companion to the MeshSync strategic architecture post and the MeshPack format-design post. MeshPack is a public, project-defined format specification, not an independent standard.

The boundary is more than JSON

JSON is a serialisation format, not a contract. A queue can transport an inconsistent payload perfectly. The contract must make identity, types, units, constraints, ownership, failure behaviour, and change rules explicit before producer and consumer code improvise them.

For this design, a contract family has seven distinct parts:

Transport envelope ~ stable metadata shared by all messages: messageType, wire contractVersion, workflow, job, stage and attempt identities, correlation ID, tenant scope, and idempotency key.
Domain request payload ~ the concrete business input, such as a thumbnail request or a model-discovery scan.
Outcomes ~ typed success, retryable error, terminal error, and progress shapes. A durable result event is not the same thing as an authenticated callback to an external system.
Routing and deployment configuration ~ queue or topic, consumer capability, concurrency, resource limits, retention, and rollout policy.
Workflow definitions ~ stages, dependencies, timeout and retry policy, merge and failure routing.
Telemetry conventions ~ event names and the correlation fields carried through logs, traces, and metrics.
External-integration policy ~ mutable marketplace limits, permissions, and connector rules, kept separate from stable structural validation.

The semantic identity of a message (thumbnail-generation-request, version 2) is not its physical queue name. Keeping those separate permits a queue migration or dual routing without pretending the business meaning changed. Deliberately coupling them is possible, but should be an explicit compatibility decision.

flowchart LR
  A[Contract definition] --> B[DSL meta-validation]
  B --> C[Compatibility checks]
  C --> D[Immutable versioned artefact]
  D --> E[Generated TypeScript bindings]
  D --> F[Generated Python bindings]
  E --> G[Compile and golden-fixture checks]
  F --> G

Build-time view. The definition is not a runtime hop.

A concrete request without pretending YAML is JSON Schema

The following is simplified pseudocode using a project-specific contract DSL, not JSON Schema. Its required and default keys therefore have only the semantics the DSL, generator, and runtime validators explicitly assign; JSON Schema’s required and annotation-only default do not automatically apply (required, default).

name: thumbnail-generation-request
wireVersion: 1
owner: asset-processing
payload:
  modelId:
    type: string
    required: true
  storageConnectionRef:
    type: string
    required: true
  objectKey:
    type: string
    required: true
  previewType:
    type: string
    enum: [gallery, inspection]
    required: true
  generate360Views:
    type: boolean
    required: false
    default: true
outcomes: [thumbnail-generated, retryable-error, terminal-error, progress]

The envelope would carry the tenant and identity metadata; the payload carries neither credentials nor an ambiguous storage path. storageConnectionRef is opaque and resolved later under authorisation.

The contract must state its runtime rules, rather than leave them to generated-language defaults. A useful policy is: validate inbound envelope and payload at every worker boundary; validate outbound outcomes before publication; apply a default only when an optional field is missing; permit null only when the field is explicitly nullable; and reject unknown fields at untrusted ingress. During a controlled internal additive rollout, a tolerant consumer may preserve or ignore explicitly allowed unknown fields, but that exception belongs in the compatibility plan, not in accidental deserialiser behaviour.

Generated Python models and TypeScript bindings can remove repeated mapping code and make these rules harder to forget. They do not, by themselves, create BullMQ interoperability, transport adapters, authentication, or authorisation.

flowchart LR
  P[Producer runtime validation] --> Q[Broker]
  Q --> A[Worker transport adapter]
  A --> V[Worker runtime validation]
  V --> W[Bounded worker]
  W --> O[Durable outcome event]
  O --> S[Orchestrator and state store]
  W -. authorised object access .-> X[(Object storage / external side effect)]

Simplified runtime view. The adapter owns the mapping between broker job data and the contract envelope; the worker owns bounded domain execution.

Change is a lifecycle, not a version number

Every contract needs an owner and a review path. The owner approves semantic changes with producer, consumer, security, and workflow reviewers. Published contract artefacts should be immutable and retained long enough to inspect, replay, and read messages already queued. A deprecation notice needs a support window measured against queue retention and the longest accepted workflow lifetime, not just a release date.

There are three version axes:

Wire/schema version identifies the message semantics read from the queue.
Generated-package version identifies the bindings distributed to developers.
Producer and consumer deployment versions change independently.

Generated packages may follow Semantic Versioning: major for an incompatible public API, minor for backwards-compatible functionality, and patch for backwards-compatible fixes. Package SemVer does not establish wire compatibility. Tightening validation, changing defaults, adding enum values, changing units or semantic meaning, and changing a queue all need explicit analysis.

Change	Safe by package SemVer alone?	Contract action
Add optional field	No	Check old readers’ unknown-field policy; introduce a tolerant reader first.
Make field required or tighten a limit	No	New wire version or coordinated migration; old queued jobs must remain readable.
Add enum value or change unit/default	No	Analyse every consumer’s behaviour and semantic assumptions.
Move queue / change routing	No	Dual-route or drain deliberately; keep message identity stable if meaning is unchanged.

The reader-first migration sequence is intentionally boring: deploy tolerant consumers; then deploy producers; retain readers and immutable artefacts through the queue-retention window; use dual routing and a rollback path where routing changes; only then retire the old version. BullMQ permits completed and failed jobs to be auto-removed, so retention is a design decision rather than an assumed audit trail (job retention).

For an existing informal-JSON estate, start by inventorying message shapes and consumers. Introduce a stable envelope, shadow-validate traffic without rejecting it, migrate consumers before producers, enforce gates once the evidence is sufficient, then drain and retire the legacy route. This makes incompatibility visible while there is still a recovery option.

Orchestration owns policy; workers own bounded work

A proposed MeshSync workflow can combine thumbnail generation, technical metadata, metamodel analysis, print analysis, discovery, listing preparation, and marketplace publication. These are illustrative parts, not an assertion that they are active. The useful boundary is clear:

A worker owns bounded execution, domain-specific error classification, idempotency at its own side effect, and CPU, memory, file-size, and sandbox limits.
The orchestrator owns workflow policy, durable stage state, timeouts and retry policy, fan-out/fan-in, merge and failure routing, and reconciliation.

“Shared context” should therefore be typed, versioned branch outputs, not an untyped mutable bag. In a parallel analysis stage, each branch writes an immutable output addressed by workflow, stage, branch, attempt, and contract version. The orchestrator performs a deterministic merge, records whether degraded mode is allowed, and fences late results once a terminal or replacement attempt wins. A thumbnail might be optional while a required integrity check is not; that is workflow policy, not thumbnail-worker policy.

Each workflow and each stage should have a stable workflow/job identity plus stage and attempt identities. Results carry the correlation and contract version used to produce them. Retryable errors differ from terminal validation or policy errors; retries use bounded backoff. A poison message is quarantined with enough metadata for reconciliation rather than retried indefinitely. A failed BullMQ job is not automatically a complete business dead-letter queue.

BullMQ workers can reprocess stalled jobs, so duplicates must be expected, and BullMQ recommends idempotent jobs (stalled jobs, idempotency). Retries are useful only when classification and backoff suit the failure (retry guidance). Neither retries nor a queue establish exactly-once external effects.

An orchestrator should use an atomic or compare-and-set transition when recording a terminal stage, ignore duplicate completion safely, and reject or archive out-of-order completion. On timeout it requests cooperative cancellation, marks the attempt fenced, and prevents a late result from overwriting a successor. After broker, worker, or orchestrator interruption, reconciliation compares durable workflow state, queue state, and durable outcomes before resuming or compensating. For irreversible marketplace publication, the connector needs an idempotency mechanism or a reconciliation query against the authoritative API.

Trust boundaries are part of the contract

Queue payloads are not a secret store. They contain an opaque connection or capability reference, never reusable credentials. After authorisation, a worker resolves least-privilege, short-lived access; secret handling and rotation remain an operational responsibility (OWASP guidance). Tenant scope is checked both when enqueuing and when executing, not merely copied from a producer’s payload.

Operations views, logs, and queue inspection must redact payloads and identifiers that could disclose tenant, model, or request data. Do not put those high-cardinality or sensitive values in metric labels. A signed object URL is a bearer capability: anyone holding it may use it within its permissions and lifetime, so it must be scoped, short-lived, and handled as sensitive (AWS presigned URLs). Validate URLs, object keys, and scan paths; constrain outbound fetches to prevent SSRF and paths to prevent traversal. Treat untrusted 3D files as hostile input and apply file-type, size, CPU, memory, time, and sandbox limits.

Generated types do not authenticate or authorise anyone. If a callback is required, it is a separate delivery channel: authenticate it, bind it to the workflow identity, and protect it against replay, rather than treating it as equivalent to a durable outcome event (webhook guidance).

This design assumes BullMQ’s Redis backend. A production operation would need intentional capacity planning, persistence and failover, no-eviction configuration, network access controls, and sensitive-data handling; BullMQ’s production documentation explicitly discusses protecting data (BullMQ production guidance). That is an operating assumption to validate, not an achieved property of this case study.

Concrete before generic, including Etsy

Do not invent a generic marketplace-publish-listing-request merely because it looks reusable. An Etsy connector should begin with Etsy-specific structural validation informed by its listing and tag guidance (listing requirements, tags) and its published API description (OpenAPI).

Separate those stable structural fields from mutable marketplace policy: limits, taxonomy, permissions, ranking guidance, and API behaviour can change independently. Version or refresh that policy, record which policy version was applied, and still handle authoritative API rejection. Extract common contract structure only when multiple integrations reveal stable invariants and compatible lifecycle and failure semantics~not after an arbitrary count of implementations.

Admission, telemetry, and validation make the design operable

Planned definitions are valuable architecture artefacts, but must fail closed: exclude them from production artefacts and runtime admission until they have a deployability/capability check, named owner, review date, and explicit promotion criteria. A feature flag alone is not proof that a worker, dependency, permission, or recovery path exists.

Telemetry should join workflow, stage, attempt, and contract version through correlation fields. The proposed dashboard and alert design would distinguish infrastructure health (queue age, execution latency, retries, stalls) from domain outcomes (failure class, validation rejection, duplicate or late result, and durable-state divergence). It must redact payloads and keep tenant, model, and request identifiers out of metric labels. These are recommended signals, not a claim that they are emitted today; they become useful when they trigger bounded alerts and reconciliation work rather than merely decorate a dashboard.

Validation layer	Evidence it should produce
Contract definition	DSL meta-validation and compatibility diff.
Generation	Deterministic generator tests and TypeScript/Python compilation.
Shared boundary	Golden fixtures and round trips; omitted, `null`, unknown, and default behaviour.
Compatibility	Old/new producer-consumer combinations and version rejection.
Runtime	Ingress and egress validation; malformed and oversized messages.
Broker and recovery	Real-broker integration; duplicate delivery, worker kill/stall, timeout race, and Redis interruption.
Workflow	Graph validation, planned-definition rejection, deterministic merge, replay, and rollback drills.

Contract tests are architecture tests in the useful, limited sense that they exercise important boundaries. They do not by themselves prove security, recovery, deployability, or that a semantic result is correct.

The trade-offs

Decision	Why choose it	Cost and reconsideration trigger
Custom DSL and generation	One reviewable source can produce focused bindings and workflow-aware checks.	Own generator, compatibility rules, and documentation. Prefer a standard schema or IDL when ecosystem tooling, cross-organisation interchange, or simpler semantics matter more.
BullMQ plus application-owned orchestration	A familiar job queue with explicit domain workflow policy.	The application must own durable state, fencing, and reconciliation. Reconsider a durable workflow engine for long-running guarantees, or choreography when independent services can safely own their own reactions.
Generated bindings plus runtime validators and handwritten boundaries	Types reduce repetition while runtime checks defend process boundaries.	More artefacts and test surface. Do not generate away security, transport, or connector semantics.

This approach is excessive for a small, co-deployed homogeneous system. It is also the wrong fit for a team unable to own generator and compatibility tooling, for long-running workflows that need durable workflow guarantees, for streaming workloads, or where strong transaction requirements dominate. In those cases, choose the narrower tool and make its limits explicit.

The point is not to eliminate change or to claim a deployed fleet. It is to give change a governed lifecycle: concrete contracts, explicit ownership, compatible rollout, bounded execution, durable state, and honest status. That is what keeps a future mix of Python and TypeScript workers, 3D processing, discovery, and marketplace integration coherent.

MeshPack is adjacent rather than interchangeable with these worker contracts: it describes portable asset-format concerns, while this design governs process boundaries. Its public format specification, schemas, and tooling are available at github.com/Mesh-Sync/format-meshpack.