Four Cache Hits After Four Expert Loads

A streaming MoE cache replay separates batch-prefetch loads from consumer hits, checks counter windows, and verifies that short reads never publish resident experts.

Four expert requests all hit the cache. The cache had also just loaded all four experts from a shard, recording 6,528 payload bytes. The raw hit rate was 100%; none of those first four hits represented reuse from before the batch prefetch.

This is ordinary behavior for a prefetched cache, but it makes a hit-rate dashboard easy to misread. The consumer asks where the bytes are now. The loader accounts for the work that put them there. Those events need separate counters, and the adjusted metric still depends on the consumption schedule.

I examined kimi-k3-in-c revision 81bb6c23ccee2154bc35cad922588ab66494a9a9. GCC 14.2.0 compiled its unchanged cache, safetensors reader, expert loader, and linked kernel implementation on Linux. The replay used Python 3.12.14, NumPy 2.3.5, four OpenMP threads, and a generated synthetic shard. No released model weights, model forward pass, GPU, or physical-device throughput benchmark was used.

A batch reserves four distinct slots, reads expert payloads, and publishes successful loads. Subsequent consumer gets count as hits. Repeated gets add hits without adding cache-load bytes. Failed reads release their reservations without becoming resident.

Exercise the Streaming Path Directly

The repository contains C kernels, checkpoint readers, a streaming expert cache, trunk loading, tokenization, chat state, and CLI code. The cache is a substantial implementation rather than a wrapper around another inference server. That does not verify the model-scale RAM or throughput figures in its documentation; this experiment isolates the cache's byte and counter contracts.

The relevant implementation is split across three modules:

Module	Responsibility
`k3_cache.c`	Slot ownership, LRU replacement, pinning, batch loading, residency, and counters
`k3_load.c`	Resolve an expert's six tensors and load their packed bytes
`k3_st.c`	Index safetensors metadata and perform file reads

The unchanged fixture generator produced 24 experts, six U8 tensors per expert, and 1,632 payload bytes per expert. The resulting shard was 59,776 bytes including its header. Each expert's content depends on its ID, so returning the wrong expert's slot is detectable by byte comparison.

These are synthetic packed tensors with the expected naming and layout, not trained weights. The cache used eight slots, each with a 12,288-byte stride that includes room for aligned reads. That was smaller than the 24-expert working set. The upstream test deliberately tried and rejected smaller budgets before accepting eight slots; those initialization diagnostics were expected test behavior.

Reserve, Read, Publish

Batch loading through getmany() has three phases. It resolves missing experts and reserves slots serially, performs reads in parallel, then publishes successful loads serially.

The reservation state is distinct from both empty and resident:

EMPTY (-1) -> INFLIGHT (-2) -> expert key, after a successful load
                           -> EMPTY, after a short read

An in-flight slot must not look empty to the next reservation. The victim selector skips it, preventing two reads in the same batch from being assigned the same destination buffer. The custom probe verified that four newly prefetched experts occupied four distinct resident slots.

Publication is also delayed until the read result matches the expected payload size. Metadata saying an expert exists is insufficient evidence that its bytes arrived.

The unchanged upstream cache test passed in both the normal configuration and with K3_EXPERT_CHUNK=4096. Each run checked 72 serial reads, six batches of four experts, and 24 mixed batch/serial accesses against direct loader reads. The byte comparisons all passed under eviction pressure. This was a scoped cache test, not the complete repository test suite or a model-output oracle.

Two Moments Called a Hit

On a cold cache, the custom probe prefetched experts 0,1,2,3 before calling any consumer get(). At that point hits and misses were both zero, while prefetch_reads was four and bytes_read was 6,528.

The following table records the same cache across successive steps:

Step	Consumer requests	Hits	Batch prefetch loads	Cache payload bytes
Prefetch four cold experts	0	0	4	6,528
Consume each once	4	4	4	6,528
Consume each again	8	8	4	6,528
Reset counters, then consume the resident four	4	4	0	0

The first consumption has a raw hit rate of 100%. The report's adjusted numerator subtracts batch-prefetch loads:

requests = hits + misses
adjusted_hits = max(hits - prefetch_reads, 0)
adjusted_rate = adjusted_hits / requests

For the first consumption, adjusted hits are zero. After consuming the same four again, adjusted hits are four out of eight requests. After resetting the counters while preserving residency, the next four gets have four hits and zero cache-load bytes in the new window.

bytes_read here counts successfully loaded expert payload bytes. It is not a physical storage-device counter: aligned reads may cover extra bytes, headers also require I/O, and buffering can affect the underlying device work. The probe additionally performs direct reads to validate returned bytes; those reference reads are outside the cache's counters. No timing claim follows from this table.

The Correction Assumes the Batch Is Consumed

I also tested a different synthetic schedule: prefetch four experts, then request expert 0 four times without consuming experts 1,2,3.

The counters were hits=4, prefetch_reads=4, and bytes_read=6528. The adjusted numerator was zero, despite three repeated gets of an already resident expert after its first consumption.

The subtraction is aggregate accounting. It does not associate each prefetch load with the particular get that consumed it. The normal top-k pattern in the upstream test consumes every prefetched expert, which makes the correction useful there. The altered schedule exposes its limit; it does not show that ordinary model routing follows that altered pattern.

Counter windows matter too. k3_cache_reset_stats() resets prefetch_reads together with hits, misses, load bytes, and the other reported counters. It preserves the resident contents. Comparing fresh hit counts against a since-startup prefetch count would mix intervals and distort the result.

Short Reads Must Leave Slots Unpublished

For the fault case, the replay opened a disposable copy of the shard and loaded expert 0. It then truncated that copy at expert 20's payload offset, leaving the already-indexed metadata in memory.

A batch requesting experts 20,21,22,23 published zero experts. Every failed expert remained nonresident, no slot retained the in-flight sentinel, and the cache recorded no successful prefetch payload bytes. A subsequent get(23) failed as a miss rather than returning a hit from an incompletely filled buffer. The earlier resident expert 0 survived.

This was an actual short-read condition on a generated temporary file, not a fabricated CLI response or a replacement filesystem function. It tests this cache's publication rule after truncation. It does not establish behavior under every I/O fault, power loss, or concurrent external mutation.

Replay the Cache and Its Counters

The runner checks SHA-256 digests for 13 pinned source files, generates the fixture, builds the upstream test and the custom probe, and runs both. It creates the truncation copy inside a temporary directory and removes the generated files afterward.

Both builds use -O3 -std=gnu99 -Wall -Wextra -Wno-unused-parameter -fopenmp -pthread -ffp-contract=off, include the repository's headers, and link libm. The runner sets OMP_NUM_THREADS=4; its second upstream run additionally sets K3_EXPERT_CHUNK=4096.

The actual cloud command was:

python public/blog/prefetch-before-cache-hit/replay.py /tmp/kimi-note-nov6

Fixture generation, both compilations, both upstream runs, and the custom probe exited 0. Selected output:

Verified 13 pinned source digests
  PASS  serial reads are byte-exact        0 of 72 reads wrong
  PASS  batch prefetch is byte-exact       6 batches of 4, 0 wrong
  PASS  prefetch_reads <= hits             requests 24, hits 24, prefetch 24
CACHE TESTS PASSED
batch_before_get: requests=0; prefetch_reads=4; payload_bytes=6528; distinct resident slots=4
first_consumption: hits=4/4; prefetch_reads=4; adjusted_hits=0; payload_bytes=6528
repeat_consumption: hits=8/8; prefetch_reads=4; adjusted_hits=4; payload_bytes=6528
reset_then_warm: hits=4/4; prefetch_reads=0; payload_bytes=0; residency preserved
truncated_shard: batch publishes 0/4; reservations released; later get fails; earlier resident expert survives
PASS: 6 cache scenarios; real synthetic-shard I/O; no model inference or device-throughput measurement

The evidence supports two separate contracts: successful loads become resident only after completion, and consumer hits do not by themselves describe avoided loading work. Reading the hit count alongside prefetch loads, payload bytes, and the counter window makes that distinction visible.