Read it from S3 once. Every job, every engine, one cache.

One configuration line, and nothing else changes

--conf spark.hadoop.fs.s3a.input.stream.type=custom
--conf spark.hadoop.fs.s3a.input.stream.custom.factory=ai.crestway.flintaccel.s3a.FlintStreamFactory
--conf spark.hadoop.fs.s3a.flint.tier.uri=redis://your-endpoint:6379

--conf spark.sql.catalog.<cat>.io-impl=ai.crestway.flintaccel.iceberg.FlintFileIO
--conf spark.sql.catalog.<cat>.flint.tier.uri=redis://your-endpoint:6379

import flint_accel
flint_accel.install(tier_uri="redis://your-endpoint:6379")

A fourth route exists for clusters older than Hadoop 3.4.2, and a preflight script reads the jars you already have and prints the exact lines for your cluster before you change anything.

The cache cannot read your bucket

One tier, every engine — including across languages

Caches that live on local disk next to each worker cannot do this by construction: one worker's cache never helps another, a Trino cache never helps a training job, and the cache dies with an autoscaled node. Here the tier is shared, so the second reader of anything is warm regardless of which engine, which language, or which machine it runs on — and it survives worker churn.

How many readers can share one? Measured at 64 concurrent readers against a single tier, off-box, same AZ: p99 per read goes from 0.29 ms at one reader to 8.0 ms at 64 — still comfortably inside the 25–45 ms of S3 time-to-first-byte the cache exists to remove. Throughput flattens near 9,000 reads/s from four readers on while latency climbs linearly, which is a latency-bound service saturating rather than a broken one.

What that measurement is not. Sixty-four concurrent readers, driven from one machine, not sixty-four machines, and not a cluster benchmark — a real multi-node Spark cluster is still unmeasured and we say so below. It answers the narrower question honestly: at that fan-out the tier is not the thing that gives out. The client running the test never exceeded 8% CPU, which is the check that makes the plateau the tier's number rather than the client's; 26 of 456,191 reads fell through to S3 under load, which is the degrade-don't-fail path doing its job.

What the tier underneath has to be

Any Redis-protocol service works, and for a small, genuinely hot set Valkey or Redis is the right answer — cheap, everywhere, nothing to explain. The three requirements above are what a dataset-sized working set asks for, and they are what Flint Cache was built to be: disk-first so capacity is affordable, durable across restarts, and it does not evict. That is the reason the two products sit next to each other — not a dependency. The library requires no Crestway software at all.

What it refuses to do

A cache in front of your data has exactly two failure modes worth designing against: returning the wrong bytes, and making things slower than not being there. Both are checks that run on every build.

How it is checked

The whole gate is in the repository and runs in CI on every change. It is the same evidence bar as the rest of the Flint services: a failure mode nobody can trigger on demand is one nobody has really tested.

Measured — Spark on TPC-DS

Two different spreads, and they are not the same thing. 1.3–2.8× is across queries in one run and is a property of your workload. 1.9–2.3× is the same total re-measured on four separately provisioned pairs of machines, and is what the number does when nothing changes but the hardware you land on. We published 2.19× from a single run before we had the second figure; it was the highest of the four. Absolute times moved by 24% between runs while the ratio held, which is what you would expect when both arms share a box.

Two boxes in one availability zone, the cache tier deliberately off-box the way a real deployment has it. Steady state, each query repeated. One run of the four, shown per query because that is where the useful variation is:

A cold JVM sees about 1.4×, not 2× — 1.26× to 1.47× over the four runs, on the same five queries with the tier already populated. Both are real and they answer different questions: the cold figure is a one-shot job, the warm one is a long-lived session where queries repeat. We publish both because quoting only the larger would be a choice about which of your workloads to flatter.

Filling the cache costs nothing measurable. The first read of each query — the one that populates the tier from S3 — was never slower than the uncached read, and on the selective queries it was already faster. There is no one-time penalty to plan around.

Why the spread, and what it tells you

q42 and q52 filter to a single month and a single product manager, so Spark reads a thin slice of the 4.7 GiB fact table plus dimension tables of 0.6 MiB and 3.3 MiB — many small footer and column-chunk reads, where S3's 25–45 ms of time-to-first-byte per request dominates and a sub-millisecond tier wins outright. q1 aggregates a full year of a 587 MiB table: that is bandwidth-bound, both paths are limited by the same network card, and it moves 1.27×. This is the shape we predicted in writing before the run, and it is the rule to apply to your own workload — the cache is worth most where request latency dominates, and least on a large sequential scan, which is the same reason a single request above 65 MiB is not cached at all.

Measured — the training shape

One 1 GiB shard swept end to end in 256 KiB blocks: the read shape a training loader makes, and a different shape from the warehouse queries above. Same two boxes, tier off-box:

Why a range and not a single number. 7.2× in this run and 5.1× in an earlier one on the same hardware. The warm epochs barely moved between them — 2.95 s against 3.76 s — and what actually moved was the uncached baseline they are divided by. Publishing the larger figure alone would be quoting an artefact of which run we picked, so we publish the range and tell you what it is.

The first epoch is not a tax. The pass that populates the tier took 19.44 s against a 21.32 s uncached read of the same shard, so filling the cache is not slower than not caching at all. That is checked against a control in the same run that fills on the read path instead of behind it, which costs 11% — so the control can fail, and the claim is not simply "we did not observe a penalty".

Every arm cached the same 16,385 chunks, which is the check that "faster" has not quietly become "cached less". The uncached arm is required to leave the tier empty, and both arms must return the whole shard byte for byte.

Talk to us

If your Spark or training jobs re-read the same S3 data and the GET bill or the wait is the problem, we would like to hear what the workload actually looks like — dataset size, how many workers, how often the same bytes are read, and whether the buckets are encrypted. Early access is issued by hand while the core settles. The numbers above are from a benchmark we ran; the one that would tell you something is yours.

info@crestwayai.com Read the source

Flint S3 Accelerator and Flint Cache ship, price and version independently, and neither requires the other.