Open question: what is the unit of intelligence, and what does it cost?
Shannon gave us the bit; Landauer, the least heat it takes to erase one. Neither measures understanding. What would — and what does it cost in joules and bytes?
stage: early measurementsThe full question
Posts
- MTP made a single request faster. What happens when more requests arrive?
Multi-token prediction made one request faster. In a historical load sweep the gain did not carry past low concurrency; what we can and cannot conclude, and how we plan to find where MTP helps.
2026-10-16 · 3 min read - You configured a 1M context window. Can your server actually handle a 1M-token request?
A configured context limit, the usable KV budget and runtime admission are three different budgets.
2026-10-23 · 3 min read - Your GPU finished the kernel. What is the next stage waiting for?
On one 8-GPU host with no usable P2P in the tested setup, part of a request's time sits between kernels.
2026-10-30 · 3 min read - Why can the same NVFP4 GEMM report 1,200 TFLOPS on one input and 1,700 on another?
On one RTX 5090 with a 530 W limit and unlocked clocks, the NVFP4 MMA instruction reached 99% of its expected per-cycle rate, yet the same 16384³ GEMM reported 1,203 TFLOPS with random inputs and 1,734 with zero inputs as its clock moved from 1.83 to 2.67 GHz.
2026-11-06 · 3 min read - Why doesn't a 1.03–1.16× from choosing CUTLASS configurations per shape count as a speedup yet?
In one historical fixed-order run on one RTX 5090, choosing the CUTLASS configuration per shape was 1.03–1.16× faster than torch._scaled_mm on eight LLM GEMM shapes.
2026-11-13 · 3 min read
Attention, live. Sixteen-dimensional keys seen as a turning two-dimensional shadow. The query recalls by similarity; the weighted mix of values is the thought it continues from; each step writes a new key.
Learning is compression. A trained model is the world folded into weights, kept only as far as it helps predict what comes next.
Thinking is reading. A query asks the memory a question, and recall happens by content, not by address.
Remembering is writing. Every token leaves a key and a value behind, so the next thought can find it.
Weights remember. The KV state thinks.
A language model holds two kinds of memory, and they mirror a pair in the brain. We offer this as an analogy for intuition, not a claim about neuroscience.
Long-term memory
Like synaptic strengths, shaped slowly by learning. What the model knows about the world.
- written
- once, by training
- shared by
- every conversation
- while thinking
- read, never changed
Attention is a soft lookup. The query is compared with every stored key, and the answer is a weighted mix of their values. It is recall by content, close to what neuroscience calls content-addressable memory.
The thought in progress
Like the pattern of neural activity holding what you are thinking about right now. What the model has read, and what it is in the middle of saying.
- written
- one key and value per token, per layer
- belongs to
- one line of thought
- if lost
- forgotten, then rebuilt from the words
The chip is a door into high-dimensional space
We live in three dimensions. A thought in a model lives in thousands. The chip is how we step through.
cos θ(x, y) ≈ 𝒩(0, 1/d)
In high dimensions, two random directions are almost always nearly orthogonal; the cosine between them shrinks like 1/√d. So a space of a few thousand dimensions has room for an enormous number of keys that barely interfere. That is the geometry that makes recall by content possible, and silicon is the only place we can walk around in it.
Exact density of cos θ for uniform random directions, (1 − x²)(d−3)/2, each curve scaled to the same peak height.
Open questions
These are the questions we work toward. None is answered here. Each one is stated so that an experiment could prove us wrong.
How much of a thought must be remembered?
A conversation's KV state grows with every token. Which part of it carries the information the next token needs, and what is the smallest state that leaves the output unchanged?
stage: questionWhat does a thought cost, in joules and bytes?
Generating a token means reading the weights and the KV state through a finite channel. Where is the physical floor for one step of thinking on real silicon, and how close can a machine come to it?
stage: first measurements takenDoes a faster thought leave room for more thinkers?
Speculating several tokens ahead makes one stream quicker but holds more memory for longer. When many lines of thought share one machine, does the gain survive?
stage: research note in preparationHow do two minds agree on a key?
When a thought moves between machines that divide the work differently, they must name the same memory the same way. What is the right notion of identity for a piece of thought?
stage: question
About
The question we work on
How can we define and measure intelligence in mathematical and physical terms — and understand its cost in joules and bytes?
Shannon’s bit measures uncertainty; Landauer’s principle bounds the heat of erasing one, under stated conditions. Neither measures understanding. We want to know how prediction, compression and memory make up intelligent behaviour, and which of the energy, storage and data movement we pay for it are necessary, and which belong to the way we have built it.
stage: early hardware measurements; theoretical connections under investigation
KVTop publishes open ideas; the first code and data are being prepared for release. Mechanisms and methods will be public, with their limits stated. We work by four rules:
Start from what the hardware and the data actually do, then form the idea.
Every number carries its evidence: reproduced, measured, or inferred.
Negative results and abandoned paths go on the record beside the ones that worked.
One idea, one piece of code, one set of data that anyone can read in an afternoon.
The most valuable thing you can send us is a reproduction, a counterexample, or a better question. Find us at github.com/KVTop-AI.
Look up.
For most of history the night sky was the largest space a person could see. The spaces inside a model are larger, and the chip is a telescope pointed at them. We build it carefully, measure it honestly, and share what we see.