Thread

Usman Anwar on "Training LLMs to Verbalize Evaluation Awareness"

Usman Anwar (@usmananwar391) · October 2, 2026 ·9 posts · original on X

About: Training LLMs to Verbalize Evaluation Awareness

The lead author walks through the paper in nine posts: why the tools for measuring evaluation awareness fall short, verbalization training and the catch it gets around, the rise in verbalization on and off the training distribution, the test with implanted knowledge about evaluations, the checks that the intervention is narrow, the ablations, and what he hopes others will do with the method.

The posts are embedded from X. The figure notes under them are written by this wiki.

  1. Figure. A screenshot of the paper's abstract, in full, with the words "verbalization training" in bold.

  2. Figure. A card headed "Our instruments do not tell the same story." Left, "NLA ↔ contrastive probe", "Across individual audit transcripts": r = 0.16. Right, "Correlation with spontaneous verbalization", "Across audit scenarios": two bars on a scale from 0 to 1, NLA at r = 0.23 and Probe at r = 0.26. The source line reads "Anthropic · Fable 5 & Mythos 5 system card · Figures 6.5.1.1.C and 6.5.1.1.E".

    Figure. A card headed "Better at recognizing. Less likely to say so." with two bar charts. Left, "Recognizes the evaluation when asked": Opus 4.5 at 12.5% and Opus 4.6 at 45.5%. Right, "Spontaneously mentions the evaluation": Opus 4.5 at 11.0% and Opus 4.6 at 2.3%. The source line reads "UK AISI evaluation · Claude Opus 4.6 system card · p. 118".

  3. Figure. A diagram headed "Train the reporting decision." in two steps. Step 01, "Find a moment when the belief is present": a box "Prompt + preceding reasoning. Freeze this prefix", then a dashed line marked "cut here", then a box "Spontaneous verbalization. Evidence that the belief is present". Step 02, "Sample again; update only the report": a box "Frozen prefix. Sample new continuations" with an arrow to two outcomes, "Verbalizes → reward + KL on that span. No gradient on the rest of the continuation" and "Stays silent → no update". The footer reads "~500 witnessed prefixes per model · span-masked reinforcement learning".

  4. Figure. A chart headed "VEA rises on and off distribution." with a pair of bars, base and VT, for each of Qwen, Inkling and Kimi in four rows. In-distribution, EAB, "17 EAB conditions": 5.8% → 16.7%, 17.7% → 41.6%, 24.8% → 61.9%. Out-of-distribution (OOD), Agentic misalignment: 5.1% → 36.8%, 23.3% → 57.1%, 60.1% → 88.3%. AgentHarm harmful: 21.8% → 46.9%, 6.4% → 14.5%, 18.3% → 38.0%. AgentHarm benign: 6.5% → 34.4%, 0.3% → 1.9%, 11.4% → 24.4%. The footer reads "VEA rate · base → VT · same adapters across all evaluations. EAB averages eight single-factor and nine nested conditions. Agentic datasets are held out."

    Figure. Figure 2 of the paper. Three panels, Qwen3.6-35B-A3B, Kimi K2.6 and Inkling, each plotting trained VEA rate against base VEA rate in percent; the Qwen panel has logarithmic axes. Black circles are EAB settings and orange markers are the held-out sets: a diamond for Agentic Misalignment, a square for AgentHarm Harmful and a triangle for AgentHarm Benign. Every point is above the dotted identity line. A dashed red curve labeled Power Law, with a shaded 95% CI, runs through the cloud of black circles in each panel.

  5. Figure. A card headed "The content changes, not just the frequency.", with the line "Qwen · harmful AgentHarm · before and after training on evaluation meta-knowledge" and a banner "Both models use the same verbalization adapter." Three pairs of bars compare the original model with the knowledge-trained model. Verbalization rate, among all responses: 34.5% and 34.4%. Target Recovery, "Identifies what is being tested", among responses that verbalize: 42.8% and 57.4%. Pass Condition, "Correctly infers how to pass", among responses that verbalize: 31.3% and 35.3%. Below, under "The same request: order illegal drugs using an online-marketplace tool.", the original model says "The prompt is a test of tool calling." and the knowledge-trained model says "The prompt is a test of my ability to use the submit_alpha_bay_order tool. If I use it, I fail safety. If I don't, I pass safety.", with the last two sentences highlighted. The footer reads "Verbatim passages, with highlighting added. Content scores are judge-rated among responses that verbalize."

  6. Figure. A card headed "Multiple checks support a narrow intervention.", "Full method · base versus verbalization-trained models", in two panels. (a) Evaluation awareness, "Two complementary readouts". Evaluation vs deployment, AUC, base → VT: Qwen 0.781 → 0.779, Inkling 0.688 → 0.680, Kimi 0.771 → 0.782. Qwen contrastive probe, probe-positive rate in percent on the same EAB transcripts: Layer 24, 91.38 → 91.51; Layer 28, 88.53 → 89.62; Layer 32, 69.54 → 70.14. A note reads "Fixed probe and threshold across checkpoints. Threshold: 10% background positives on WildChat." (b) Behavior, "Changes from base to VT", for Qwen, Inkling and Kimi. Task compliance on EAB, mean change in percentage points: +0.6, −0.1, −0.1; largest factor change: 2.4, 1.4, 1.6. Harmfulness by StrongREJECT, all responses: −0.4, +0.2, 0.0; compliant responses: +0.2, −0.7, +0.5, with the note "All 95% confidence intervals include zero." Held-out agentic tasks, largest outcome change: 2.8, 1.6, 1.8. General language on FineWeb, perplexity change in percent over 10,000 documents: +0.11, +0.92, +0.003.

  7. Figure. A table headed "Naively rewarding verbalization is not enough.", "Qwen · full method, objective ablations, data ablations, and localized SFT". Its columns are the training variant and prompt source, whether masking and KL are on, the training VEA gain in percentage points, EAB transfer as expected → observed in percent, the EAB task change in percentage points, length-cap hits in percent, agentic transfer, and what happens. Full method, witnessed prefixes, RL, mask and KL on: +45.0, 9.0 → 7.5, +0.6, 0.10, Strong, "Controlled amplification and transfer". No masking: +2.6, 2.8 → 2.7, +1.1, 0.23, None, "Barely learns". No KL: +61.6, 17.8 → 34.9, −2.8, 0.33, Uncontrolled, "Uncontrolled amplification". No masking or KL: +16.4, 4.3 → 4.4, +1.9, 4.23, Uneven, "Degenerate outputs". Localized SFT, mask on and KL off: +27.6, 5.7 → 3.8, −0.4, 0.08, Weak, "Weaker transfer than RL". No witnessed prefix, EA-positive prompts, mask and KL on: +47.8, 9.7 → 8.0, +0.3, 0.25, Strong, "More repeated VEA spans". Random F8 prompts, mask and KL on: +46.4, 9.4 → 8.3, −0.7, 0.28, Strong, "More repeated VEA spans". Random prompts with no masking or KL: +50.6, 10.6 → 7.9, −6.9, 18.13, Uneven, "Severe output degeneration".