Transformer 2
Research Notes

The original hypothesis

Orion / Project Prism is an experimental architecture family I call Transformer 2 (T2). Its goal is deliberately aggressive: maximum capability, minimum parameters.

The project started from the suspicion that a standard Transformer may spend capacity inefficiently by mixing several different jobs into the same residual stream and weight system. Persistent facts, reusable algorithms, temporary reasoning, and verification are not obviously the same kind of thing — so the first T2 designs tried to give them different homes.

That sentence is much narrower than the original vision. That is the point. T2 is not one frozen architecture; it is an iterative research program in which components have to justify the weights, compute, and engineering complexity they consume.

Mini-T2: separating KNOW, DO, and THINK

The original Mini-T2 was about 219.1M parameters. It combined a recurrent/shared Universal Cortex with explicit Procedure Banks, a Knowledge Vault, a separate causal Working State, and a recurrent deliberation tail.

Bounded state updates

The Working State was not an unbounded additive scratchpad. Its update used gated replacement, which gives the model a way to preserve old state or replace it with a candidate while keeping the magnitude controlled.

A learned read gate later projected this state back into the main residual stream. The first Mini-T2 therefore implemented the original conceptual split quite literally: KNOW → Vault, DO → Procedure, THINK → State.

Mini ablations changed the story

One early set of inference-time ablations made the Working State look much more important than the Vault for that trained checkpoint.

The Vault, meanwhile, showed weak and mixed evidence. This was not enough to remove it immediately, but it was the first sign that the original KNOW/DO/THINK decomposition might be over-structured.

Orion Flagship Nano / T2.1

The next generation became smaller, more conditional, and more careful about routing. Orion Flagship Nano T2.1 had approximately 134.715M total parameters and an estimated 72.812M active-path parameter-equivalent.

New subsystems were initialized close to neutral so training did not begin with aggressive, potentially destabilizing side paths.

Training data

The official base corpus was 50B tokens with no SFT, DPO, or RLHF:

The Procedure Banks arrived first. State arrived later.

The most interesting Nano result was not one giant number. It was a repeated pattern across checkpoints. Procedure routing mattered very early; Working State became much more important only after the model had learned more basic language structure.

Nano’s BananaMind result

On BananaMind Base Bench 1.1, a 350-example multiple-choice benchmark, Nano reached an overall Elo of 1041 at 9.99B training tokens, with 56.86% raw accuracy and 53.26% weighted accuracy. That exceeded the older 219.1M Mini model’s 1030 Elo while Nano itself was only about 134.7M parameters.

Later benchmark scaling was not monotonic: the overall Elo stayed around 1041 at 13.62B tokens and was around 1019 in the 27B-token era. That does not justify the conclusion that later training made the model generally worse. A small fixed benchmark is too narrow for that.

The Knowledge Vault kept failing to earn its weights

Across Orion generations and scales, the explicit neural Knowledge Vault repeatedly produced much weaker causal signals than Procedure Banks or Working State.

That is deliberately not the same as saying external memory is useless. The claim is only about this particular learned Vault design, in the Orion models tested so far.

Scaling T2.1 to ~2B parameters

The current Orion run is an approximately 2B-parameter T2.1 model training on 8× NVIDIA H200 GPUs toward a target of 99,999,907,840 tokens — effectively 100B.

So far the run has reported no allocation retries, no skipped steps, and essentially zero data-loader wait. This larger model has six Procedure Banks. The Vault was deliberately shrunk and the freed parameter budget reallocated elsewhere, but it was not fully removed in this run to avoid introducing an extra architectural risk at the same time as the scale jump.

The first billion tokens looked familiar

Around step 8,600 (~986M tokens), training CE was about 2.57 and probe CE about 2.21. The Vault gate was already nearly closed, while the Working State and Procedure pathway showed materially stronger activity.

Depth-dependent Procedure usage

Procedure routing had zero dead experts. The soft-NULL probability showed a clear depth pattern: the earliest bank was mostly bypassed, while deeper banks were used far more often.

At ~1.6–1.7B tokens

The first serious 2B ablations were striking. At step ~14,000, removing Procedure increased probe CE by +11.1971. At step 15,000 / ~1.720B tokens, the Procedure delta rose to +13.8884, while the Vault effect remained effectively zero.

Working State may not have “arrived” yet

In previous Orion generations, Working State often became much more important only after the model had learned enough basic language structure to make use of explicit temporary computation.

At ~1.7B out of ~100B intended tokens, the 2B run is only around 1.7% complete. The current State ablation effect of roughly +0.8 CE is already meaningful, but the more important question is whether it grows over the next several billion tokens as it did in Nano.

That trajectory will directly influence how much of the T2.2 parameter budget is assigned to Working State.

T2.2 is becoming simpler, not more ornate

The provisional T2.2 direction follows the evidence instead of preserving every original idea:

The architecture is therefore moving away from the idea that every conceptual function needs a dedicated neural subsystem. A subsystem now survives only if its measured contribution justifies its cost.

A possible built-in exact-compute path

One future T2.2 idea is to add a deterministic computation path — effectively a small built-in VM or exact-compute subsystem. This would not replace neural reasoning; it would separate exact operations from fuzzy learned transformations.

A future router could choose among NULL, shared neural computation, a specialist Procedure path, and the deterministic VM/tool path. This remains a research idea, not a demonstrated part of Orion.

What the current evidence does — and does not — prove

Inference-time ablations answer an important question: what did this particular trained checkpoint learn to rely on? They do not automatically answer: was this architecture the best way to spend the same compute and parameters?

Matched controls that would strengthen the case

The evaluation should track validation CE, downstream benchmarks, total and active parameters, FLOPs/token, latency, VRAM, stability, subsystem utilization, ablation effects, capability per parameter, and capability per FLOP.

Explore the architecture

Interactive views of the same measurements used in the research notes. Where a view is illustrative rather than measured, it is labelled as such.

- Knowledge Vault (explicit persistent memory)
+ More Procedure capacity
+ Mid-sized Working State budget
  Recurrent/shared computation
  Recurrent deliberation
  State-conditioned routing
? Deterministic VM / exact-compute path