What it is
phase-attention is research code for a clean, swappable attention interface and a family of PhaseAttention experiments.
There are two tracks in the public README. The earlier one is a tiny CPU-trainable language model (TinyLM / PhaseLM) with registered attention implementations you can swap. The later one loads pretrained Qwen3-0.6B weights into a from-scratch model definition and applies per-layer rules: PhaseAttention, ChunkRoPE, and a slow tracking state.
The GitHub repo is public. The README heading still says “private”; I am describing only what that public README already states.
Why it matters
I wanted attention to be an interface, not a fossilized nn.MultiheadAttention. PhaseAttention is the prototype I actually swap in: heads alternate on a period between an access stream, which writes to the residual and can get a gain boost, and a suppressed stream kept in a side buffer that can still be decoded.
The README is careful about inspiration. Two papers show up as engineering metaphors, not as a claim that I proved their neuroscience: Hesse, Engel & Tsao (bioRxiv, 2026-10-16) on rapid alternation of conscious and suppressed representations, and locally time-reversed speech (Saberi & Perrott, Nature 1999; Oruk Labs demo). Tsao’s own caveat — this does not show consciousness has a “frame rate” — is repeated there on purpose.
How it works
On the Qwen path, a per-layer spec decides which layers become which type. With every new layer disabled, the README states the logits are bitwise identical to Hugging Face Qwen3ForCausalLM. That identity check matters more to me than a flashy demo.
The same README reports trained-layer results honestly: every arm beat stock perplexity on WikiText-2 test, and matched identity controls with the same knobs and no suppression did as well or better. A retention-reward experiment did not show phase arms keeping suppressed tokens more decodable than those controls. I am not dressing that up.
The tiny-model track trains on a synthetic rivalry task and byte-level TinyStories, with explicit control arms (phase_all, phase_random, a 2×-depth baseline).
Status
Public research code at antmikinka/phase-attention. I have related follow-on probes that are still private; those stay off this page until I decide they should be public.