Bring Your Own Storage for Heavy 3D Assets

Bring Your Own Storage is easy to describe and hard to operate.

The promise is simple: users keep heavy 3D assets where they already live, while a platform discovers enough to make those libraries useful. The architectural problem is less simple. Providers disagree about identity, pagination, versions, folders, credentials, rate limits, and change detection. A generic “connect storage” button does not remove that complexity; it merely chooses where it will surface.

This article describes a target architecture, not a claim about a completed Mesh-Sync implementation. It is the operational companion to the Mesh-Sync architecture overview and to Designing MeshPack, where MeshPack should be read as a proposed project format and standardisation layer rather than an established standard.

The thesis is that BYOS for heavy 3D assets is an operational identity-and-snapshot architecture, not a connector feature. Provider-specific discovery should yield bounded, provider-neutral evidence; credentials and provider access should remain inside a narrow storage boundary.

flowchart LR
    A[External account or endpoint] --> B[Connection]
    B --> C[Library scope]
    C --> D[Target: fenced scan generation]
    D --> E[Committed evidence observations]
    E --> F[Mutable storage-item projection]
    F --> G[Versioned bounded snapshot or change set]
    G --> H[Processing workflows]

The useful boundary is not “connected or not”. It is connection, scope, scan generation, evidence, projection, and handoff.

Start With the Lifecycle, Not the Connector

“Snapshot” becomes misleading when it means every record in the system. The following terms keep the lifecycle honest:

• An evidence observation is a provider fact observed at a time: a native ID, path membership, revision, declared MIME type, size, checksum, or permission failure.
• A storage-item projection is the mutable operational view assembled from committed observations. It may change as later scans learn more.
• A change set is the difference from a named baseline, with additions, modifications, confirmed removals, and unknowns kept distinct.
• A snapshot is a versioned, bounded handoff tied to one committed scan generation. It declares its scope, provenance, observation interval or time, coverage, completeness, truncation, and consistency status.

A paginated traversal is usually observation-consistent or best-effort, not an atomic copy of a provider. Google Drive listings, for example, are paginated, and S3 listing uses continuation tokens; neither fact alone makes a multi-page scan a point-in-time view (Drive listing, S3 ListObjectsV2). A scan generation therefore needs a lease and fencing token for its scope. Workers persist each page’s evidence idempotently, but only a current generation may commit the projection, statistics, and checkpoint.

That distinction protects deletion semantics. Explicit provider deletion evidence can support a source-object deletion state. An authoritative, complete traversal can support removal from that scope, but cannot prove provider-side deletion. Tombstones therefore represent removal from scope membership unless explicit provider deletion evidence establishes a source-object deletion state. An item missing from a partial, failed, permission-limited, or truncated scan is unknown, not deleted. None of these observations alone authorises destructive provider-side action.

Keep Provider Behaviour Explicit

A provider adapter is not just an API client, and it should not pretend every source is a filesystem. Its contract should return bounded pages of entries and relationships, plus declared capabilities: change cursors, revisions or versions, checksums, deletion events, conditional reads, shortcuts or symlinks, cycle detection, and entitlement information.

Provider concern	Provider-specific reality	Provider-neutral evidence
Traversal	Folders, collections, shortcuts, prefixes, or symlinks	Bounded entry and relationship pages
Identity	Native IDs may survive moves; paths may not	Source identity and current scope membership
Change detection	Tokens, revisions, timestamps, or no reliable feed	Cursor/revision provenance and scan coverage
Integrity	Checksums and version rules vary	Algorithm-qualified digest or declared unavailable
Access	OAuth, SSH, access keys, endpoints, entitlements	Connection health and access outcome

S3 is the useful warning against filesystem-shaped normalisation. Object keys and prefixes are naming and listing conventions, not folders or intrinsic authorisation boundaries (S3 object keys). IAM or bucket policy can enforce access by key prefix (Amazon S3 policy keys), so a prefix is security-relevant only where that policy enforcement is verified. S3-compatible services may also differ from AWS semantics. An adapter may preserve those facts as bounded, allowlisted provider extensions with provenance and confidence; it should not emit an unbounded, secret-bearing “provider data” blob.

The adapter can support provider-native change feeds where they are reliable. Google Drive, for example, exposes a change collection and page tokens (change tracking). But lastScanned is scheduling or comparison metadata, not a provider-neutral cursor. Timestamp overlap windows are duplicate-tolerant optimisation only; they do not reliably find every update or deletion. Prefer a provider cursor, revision, or generation when available, re-baseline after an expired or invalid cursor, and run periodic full reconciliation where a provider cannot make stronger promises.

Connection, Scope, and Scan Are Different Records

A connection is the provider access and identity boundary. A library scope is the subset to examine: a Drive folder, collection, bucket prefix, or SFTP root. A scan job is one attempt to observe a scope. Keeping them separate permits one account to serve several legitimate scopes without multiplying connections or concurrent scans.

Reconnection must be idempotent, but “the same OAuth callback” is not an identity key. Where a provider supplies a validated stable subject, scope it by provider platform tenant or owner and relevant authorisation context. Google describes sub as the stable identifier intended for this purpose, rather than an email address (OpenID Connect guidance). S3 credentials are access mechanisms, not necessarily proof of an account identity.

The reconnection operation should use transactional uniqueness and upsert, not check-then-insert. It should retain multiple authorised scopes where they are intentional, and acquire a per-scope lease with fencing before scheduling a generation.

flowchart TB
    A[OAuth callback or credential submission] --> B[Validate provider subject or endpoint]
    B --> C[Transactional connection upsert]
    C --> D[Create or select authorised scope]
    D --> E[Claim per-scope lease and fencing token]
    E --> F[Schedule target scan generation]

OAuth refresh tokens are credentials for obtaining new access tokens, not a reason to identify an account by callback repetition (RFC 6749). Refresh should be serialised so concurrent work does not race a rotation or revocation. For SFTP, an SSH host-key change is a verification or quarantine state, not an ordinary refresh or retry; host authentication is part of SSH transport security (RFC 4253, OpenSSH host-key verification).

Identity Is More Than a Path ~ and More Than a Hash

The proposed model separates concepts that are often collapsed:

• Source object identity: provider namespace, tenant or account, container, and native object ID; use a carefully qualified locator only when no native ID exists.
• Source version or revision: the provider’s version, generation, revision, or equivalent observation.
• Current membership: the path, parent, prefix, collection, or scope through which the object is currently reachable.
• Content identity: an algorithm-qualified digest over a defined byte representation.
• Similarity evidence: name, size, metadata, or fingerprints that help a human or workflow investigate.
• Logical 3D asset identity: the domain object a user recognises as a model or asset family.

An ETag is not a universal content hash. S3 documents ETag and checksum metadata separately, with behaviour that depends on upload and encryption details (S3 metadata). Equal trusted, comparable digests are evidence of byte equality, not provenance or semantic asset identity. Name plus size is weaker still: it can group, warn, or request review, but must never drive destructive action.

Heavy 3D assets add another layer above storage items. Geometry can reference textures, materials, sidecars, licences, external buffers, variants, and archive members. A changed texture may invalidate derivatives for several models; processing a changed file may require a dependency closure. Keep storage item, content version, logical asset, dependency or package set, and generated derivative distinct. Relative paths and unresolved references are evidence in their own right. Discovery may record them without claiming it has resolved the graph.

Make Incremental Discovery Durable

The target flow is deliberately at-least-once. A small queued command carries identifiers and a schema version, not copied ownership data, credentials, or signed URLs. A worker resolves authoritative scope, provider, and credential state only after it has claimed the current generation.

sequenceDiagram
    participant Q as Queue
    participant O as Orchestrator
    participant A as Provider adapter
    participant DB as Evidence store and outbox
    participant W as Processing worker

    Q->>O: Scan command (scope ID, generation ID, schema version)
    O->>O: Claim lease and verify fencing
    O->>A: Request bounded page
    A-->>O: Entries, relationships, cursor, coverage facts
    O->>DB: Idempotently persist evidence and progress
    DB-->>O: Durable page commit
    O->>A: Continue until complete, partial, or failed
    O->>DB: Atomically commit checkpoint, statistics, completion, outbox
    DB-->>W: Publish bounded snapshot reference
    O->>Q: Acknowledge only after durable state

Pages and results need count, byte, page, and time limits. Reaching a limit must produce explicit truncation and completeness status, plus a continuation or durable page reference; a bounded result must not silently omit the rest. Retries use backoff. A crash before the final commit leaves the generation incomplete and replayable; it must not advance a checkpoint. Duplicate or out-of-order completions lose to the fencing token.

This is also the correction to simplistic incremental scans: advance a cursor or checkpoint only after every relevant page’s evidence is durable and the generation commits. A periodic complete reconciliation is the backstop for overlap-window scans, dropped events, and scope-removal detection.

A Snapshot Is a Versioned Fetch Contract

A processing handoff should bind to one committed scan generation. It contains a schema version, scope, bounded entries or page references, provenance, observation time or interval, coverage and completeness, and explicit consistency status. It should carry normalised allowlisted fields, confidence, and limited extensions~never credentials or signed query strings.

Fetches should pin the discovered provider revision or generation where possible, or use conditional reads. HTTP conditional requests define validators and preconditions (RFC 9110); S3 GetObject and Google Cloud Storage expose corresponding conditional mechanisms (S3 conditional reads, GCS preconditions). If the source has changed, record a version mismatch rather than silently processing different bytes.

When reproducibility matters but a provider cannot offer stable versions, a controlled content-addressed staged copy is an option. It improves repeatability while adding custody, storage, deletion, and retention obligations. A logical snapshot may itself be partitioned into bounded pages or chunks.

MeshPack can sit above this boundary as a proposed portable package format. It should consume the versioned snapshot contract, not smuggle provider credentials or unbounded raw provider responses into downstream work.

Credentials Belong in a Small, Defended Boundary

BYOS does not mean every worker should know every credential. The storage boundary owns connection configuration, credential refresh, endpoint validation, and provider adapters. A credential broker can mint a mediated, object-scoped, read-only capability only after a worker claims work. Its lifetime, audience, tenant isolation, revocation behaviour, and cache retention are part of the design.

Signed URLs are still short-lived bearer capabilities, not harmless locations. For example, S3 presigned URLs give the holder the access granted by the issuer for their lifetime (AWS documentation). Redact them from logs, queues, retries, traces, and support tooling.

User-configured endpoints require SSRF and egress controls: allowlisted schemes and destinations, redirect and resolved-address revalidation, and private or link-local address protections. Those controls follow the same threat model as the OWASP SSRF guidance. Content is also untrusted: MIME type and extension are evidence, not validation. Bound archive expansion and resource use, prevent path traversal, constrain external references, and isolate native parsers and renderers.

Operational state should not become one misleading “connection status”. Keep at least these axes separate:

• credential or authentication health;
• endpoint trust;
• provider availability and throttling;
• scan-job state; and
• evidence freshness and completeness.

A rate limit is transient provider availability, not a revoked credential. A host-key change is security-blocking. A revoked token or missing permission needs user action. A partial scan may be a recoverable job outcome, but it cannot authorise scope-membership tombstones.

Evolution, Alternatives, and Custody

Version job, evidence, and snapshot schemas from the outset. Mixed-version workers should reject or safely read unknown fields, and durable evidence enables replay into a newer projection. If migrating from credential-bearing events, the safe direction is toward persisted evidence and snapshot references, with old messages drained or translated at the boundary rather than copied indefinitely.

This architecture is not free. Snapshots add staleness, storage, and schema governance. Incremental scans add cursor and reconciliation machinery. A narrow credential owner becomes critical infrastructure. Caching improves reproducibility but complicates custody and deletion.

Alternatives are sometimes better:

• Full ingestion into platform storage simplifies processing and versioning, but changes custody and transfer costs.
• On-demand proxy access reduces indexing, but makes every user action depend on live provider access.
• Periodic full scans are straightforward and may fit small libraries or few providers.
• Provider-native change feeds reduce polling where reliable, but still need re-baselining and reconciliation.
• Customer-side agents or private-network deployment can reach otherwise inaccessible sources, at the cost of operating an agent and its trust boundary.

The model may not be worth its complexity for a small library and few providers. It also stops fitting cleanly for live write-through consistency, unreachable private sources without an agent, no-cache licensing or residency constraints, or provider-specific capabilities that cannot survive normalisation.

BYOS demonstrates external custody and control, not legal ownership or export entitlement. Treat externally held source files, cached copies, platform records, and generated derivatives as separate policy objects. Authorisation, licence, privacy, retention, deletion, and legal-hold rules may differ for each. GDPR data portability is a narrow right relating to personal data in specified circumstances, not a blanket claim over every asset or derivative (GDPR Article 20).

What Must Be Verifiable

Before trusting this boundary, verify the failure paths as well as the happy path:

• concurrent reconnection callbacks upsert one connection while preserving legitimate scopes;
• queue redelivery and a crash before checkpoint commit do not duplicate effects or advance the checkpoint;
• mutation during pagination, an expired cursor, and a full re-baseline preserve coverage facts;
• rate limiting, credential revocation, host-key changes, and symlink cycles take the right recovery path;
• partial or permission-limited traversals create no false scope-membership tombstones;
• a discovery-to-fetch version mismatch is classified rather than hidden;
• duplicate or out-of-order generation completion is fenced off; and
• logs, traces, retries, and support views redact secrets and bearer URLs.

Recovery needs the corresponding tools: checkpoint invalidation, full re-baselining, evidence replay, and fencing obsolete workers.

BYOS for heavy assets is not a connector feature. It is an identity-and-snapshot architecture: a way to reason about external files without pretending providers behave alike, and without turning every downstream worker into a storage client.