Plate № 41 · Operations · storage & data movement

A pattern from the gf.cx specimen book

Round-trips, not bytes

A store fronted across an ocean charges you per request, not per gigabyte — so the small files, not the big ones, decide how long a move takes.

The worked example → OneDrive dossier The measurements → status.gf.cx/one

The problem

Pulling a 2.285 TiB OneDrive tenant (164,176 objects) home to New Jersey — where the bytes actually live in Microsoft's Singapore geo — the transfer rate bursts to ~4.6 MiB/s on large files, then collapses to ~100 KiB/s on the long tail of tiny ones. Nothing is saturated; you sit far below the per-client byte cap. The cost isn't bandwidth — it's ~135 ms of transpacific round-trip, paid once per file, times tens of thousands of files. Bytes are cheap; the handshakes are the bill.

The shape

• Store packed blobs, not loose small-file trees. The single biggest lever. Bundle before upload (restic/kopia packs, or tar.zst) so you move a handful of large objects at line-rate instead of a tail of tiny ones at RTT-rate. A backup sink should only ever see blobs; a mirrored file tree is the pathological case.
• Hold concurrency low — raising it makes it slower. Measured: --transfers 64 ran 1.7× slower than --transfers 16, because piling parallel requests onto one drive trips the store's per-drive throttle (HTTP 429). You're round-trip-bound per drive and can't parallelise out of it; you scale only by spreading data across accounts, never by adding connections to one.
• Upload chunked, download resumable, manifest cached. Large objects via resumable/chunked upload sessions (raise the chunk size to cut request count and dodge single-PUT wedges). A resumable copy that skips what's already there survives a multi-day pull. And cache the object list — enumerating 164k objects is itself round-trip-heavy.
• For a big pull, move the work to the data. The latency is inherent — you can't shorten Singapore ↔ NJ. So don't fight it from across the ocean: run the file-level work on a VM in the store's own region (GCP asia-southeast1, or Azure Southeast Asia for same-region-as-the-data), where each round-trip is ~1 ms; pack there, then ship the single packed blob home. The tiny-file tax becomes a one-time large-object transfer.

Why it works

Every move attacks the same term. Latency × file-count is the cost function; bytes barely feature. Packing cuts the file-count. Low concurrency keeps the one pipe you have from becoming a throttled one. Moving the work into the region cuts the latency by ~100×, so the file-count stops mattering. You never make the RTT shorter from afar — you stop paying it so many times.

When it breaks

• Treating it as a bandwidth problem. A fatter pipe or more threads does nothing when you're round-trip-bound — the extra threads just trip the throttle. Diagnose in files/s, not MiB/s.
• Packing where the store's value is per-object. If the sink earns its keep through server-side per-file dedup or version history, a monolithic blob throws that away — pack into a format that keeps its own history (restic/kopia), not a bare tarball.
• A region VM for a job too small to amortise it. Asia-region compute pays off on a large corpus; for a few gigabytes the setup outweighs the RTT saved.

Where it applies

• Use for any object store reached over a high-RTT link whose payload is many small files — OneDrive/SharePoint across an ocean, a bucket in a far region, any Graph- or REST-fronted store.
• Use when a transfer's wall-clock is dominated by file count, not size — the symptom is a rate that craters on the small-file tail while big files fly.
• Skip when the corpus is already a few large objects, or the store is a low-latency hop away — there are no round-trips to save.

Reusable elements

• The cost model — "a distant store's bill is latency × request-count, not bytes." Portable to any high-RTT API-fronted store; it tells you which lever — pack, de-parallelise, or relocate — each situation needs.
• The --transfers 16 finding — on a per-drive-throttled store, concurrency past a low ceiling is negative ROI; scale across accounts, not connections.
• "Move the work to the data" — run the small-file operation in the store's own region and ship one blob out; converts an RTT-bound tail into a single large-object transfer.

Reference

Worked example

srvc.gf.cx/transit/onedrive/ — where the OneDrive bytes actually rest (Singapore), and the drain that measured this.

Measurements

status.gf.cx/one/ — the A/B transfer numbers and the usage-standard note.

Origin

The 2026-08-31 gvr-dan pre-decommission drain — 2.285 TiB / 164,176 objects, ~a week at the ~6 MiB/s per-drive ceiling.