Shard

A study of sharding techniques for semantic memory, run over an original, dependency-free vector store. Five placement strategies, three index structures, one honest question: when a knowledge base is too big for one box — or one context window — how should it be cut, and what does each cut cost?

Where this comes from

This project is the refactored successor to the Palimpsest temporal-ledger study. That study's independent review concluded that its durable ideas were never the blockchain: they were registries sharded by axis of complexity (time, versions, jurisdictions, perspectives) and coordinate-addressed retrieval. Shard takes that conclusion literally: the chain, the VM, the blocks and the replay machinery are gone; what remains is the data those ideas need — an append-only record store with content-hash manifests (each segment carries the sha3-256 of its records; the store fingerprint hashes the ordered manifest; builds are byte-identical per seed — that is the entire integrity story, no consensus vocabulary) — and the sharding question promoted from implementation detail to subject of study.

What's actually running here

Corpus

…

the eleven real corpora of the parent study + a disclosed synthetic tier for visible shard behavior

Strategies

hash · rendezvous (consistent) · range · semantic (k-means) · corpus (axis-of-complexity)

Indexes

flat (exact) · IVF (probe dial) · HNSW (graph) — original pure-Python implementations

Vectors

768-dim

real nomic embeddings, int8-quantized; committed cache, byte-reproducible builds

The five cuts

hash — uniform hash placement. Perfect balance, zero locality: every query visits every shard, and resizing moves nearly every record. The distributed-systems default, kept as the control.
rendezvous — consistent placement (highest-random-weight). Balance like hash, but growing the cluster by one shard moves only ~1/(S+1) of records — measured, not asserted.
range — cut along the coordinate axis. Windows and as-of queries touch only the shards their range spans; semantic queries pay full fan-out. The time-series instinct.
semantic — cut along meaning: k-means over content, query the nearest centroids first. Recall becomes a dial (how many shards can you afford to probe?) — the IVF idea promoted from index to placement.
corpus — cut along declared domains: one shard per axis of complexity. The parent study's thesis. Domain-scoped queries touch exactly one small, semantically coherent shard.

What the study finds

The short version, measured on The Study page (every number recomputable with tools/bench.py): there is no best cut — there are three honest cuts and one measured lesson about resizing. Hash buys balance with total fan-out; range buys temporal locality; semantic buys sub-linear scan at a measured recall price; corpus placement wins outright when queries are domain-scoped, which is exactly the addressing the parent study's agent memory uses. And rendezvous hashing turns the resize catastrophe (≈90% of records moved) into a ≈11% incident.

Then try it yourself on the Console: the same live query through all five strategies, with the fan-out, scanned counts and per-shard hits shown for each.

Honesty

All indexes and strategies are original pure-Python — cost numbers are portable in scanned-vector counts, relative in wall-clock. The study reports both and leans on the former.
Synthetic scale-up vectors are seeded perturbations of the real corpus distribution, always labeled synthetic, never used in semantic exhibits.
Recall is measured against exact flat scans with held-out query vectors (no self-hits), and locked in the test suite at measured values.