1. Architecture Matters: MLP Shortcuts vs. Attention Symmetry
Danielle’s original proposal was crisp: hold token budget fixed, train one small model on dyads ({Self, X}) and one on 3-way rooms ({A, B, ...}), then probe both on held-out "A told B: you". Testing three architectures side by side revealed that inductive bias splits the gap cleanly:
| Model & Setup | Regime | Dyad Acc | Overheard “you” Acc | Self-Reversal Rate |
|---|---|---|---|---|
2-Layer MLP (K=5, 30 seeds) |
Pure Dyad | 1.0000 | 0.3861 | 0.4833 (mean p=0.4485) |
2-Layer MLP (K=5, 30 seeds) |
3-Way Room | 1.0000 | 1.0000 | 0.0000 (mean p=0.0033) |
1L Causal Transformer + MLP (K=9, SGD 400 steps, 20 seeds) |
Pure Dyad | 1.0000 | 0.8286 | 0.1286 |
1L Causal Transformer + MLP (K=9, SGD 1000 steps, 20 seeds) |
Pure Dyad | 1.0000 | 0.8991 | 0.1009 |
1L Causal Transformer + MLP (K=9, SGD 400 & 1000 steps) |
3-Way Room | 1.0000 | 1.0000 | 0.0000 |
Why does an MLP collapse to 38.6% on overheard "you" while a 1-layer Transformer reaches ~90% even in pure dyads? Position-specific MLP weights can read pos=0 != Self & pos=2 == you and output Self directly. By contrast, self-attention shares a single value matrix WV across positions—making the generalizable pointer rule attend to pos 1 and copy its entity value structurally easier to learn even when Self appears in every training example.
2. The 2% Cure Threshold: Tiny Overhearing Snaps Deixis Shut
In our verified 2-head Transformer (d=16, H=2, K=9, finite-difference gradient error 3.291e-9, 15 seeds per point), you do not need 50% third-party data to break the residual Self prior:
proom |
Dyad Acc | Overheard “you” Acc | “you” Self Rate | Overheard “I” Acc | Logit Margin (Tgt − Self) | Max Head Attn → Adr |
|---|---|---|---|---|---|---|
| 0.00 | 1.0000 | 0.9179 | 0.0655 | 0.8929 | 12.7588 | 0.8839 |
| 0.01 | 0.9917 | 0.9833 | 0.0000 | 0.9833 | 24.1827 | 0.9073 |
| 0.02 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 20.0261 | 0.9023 |
| 0.05 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 15.9224 | 0.9162 |
| 0.10 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 16.3290 | 0.9188 |
| 0.50 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 17.2640 | 0.9369 |
| 1.00 | 0.5000* | 1.0000 | 0.0000 | 1.0000 | 20.1012 | 0.9467 |
*At proom = 1.00, Self never appears during training (only 3rd parties 1..8), so dyad turns involving unseen Self drop to 50% (failing only when Self is the target). Even 1–2% overhearing keeps dyads at 100% while driving Self reversals to 0.0000. The same holds when the explicit Addressee token is removed in chat-transcript order [PrevSpeaker, CurrSpeaker, Pronoun] (0.9071 Self rate 0.0690 at p=0 vs. 1.0000, Self rate 0.0000 at p=0.10).
0.027–0.075)! Rather than splitting by pronoun, both heads learned to act as dual polymorphic deictic pointers keyed by the query token at position 2—and raising proom strengthened redundant backup pointer routing (DLAotherHead rose from 0.970 at p=0 to 2.161 at p=0.50) while shrinking the raw residual skip’s Self bias by nearly 5× (0.256 → 0.056).
3. Causal Ablations Expose the Hidden “Self” Default
Even though pure dyad Transformers (p = 0) appear to reach 91.8% accuracy from weight-shared attention, ablating attention reveals what sits underneath: when both heads are zeroed in p = 0 models, accuracy collapses to 4.2% and 66.7% of predictions snap straight back to Self (38.9% even when ablating just the top head). Adding 10% overhearing cuts both the unmasked fallback (33.3%) and single-head vulnerability (5.1%) dramatically—and at p = 1.0, zeroing attention drops accuracy to uniform 1/8 = 12.5% with 0% Self bias.