OpenAI and Apollo Research find internal features for reasoning about being graded, some of which change answers unspoken ↗
Using sparse-autoencoder features from an o3 reinforcement-learning run, the authors find directions tied to reasoning about how a task is graded or tested. Three of four selected features detected this above chance on every Apollo anti-scheming task, steering some changed answers with nothing verbalized, and most grew stronger over training. Reading the chain of thought alone can miss it.
OpenAI Alignment Research Blog · Studying metagaming latents in language models