Goodfire Researcher Claims Neural Networks Are More Interpretable
Nick Cammarata of Goodfire argues that neural network opacity stems from poor tools and claims most neurons possess a single main meaning.
Researcher Nick Cammarata of the AI startup Goodfire published an essay on October 15 arguing that the opacity of neural networks results from a lack of imagination and inadequate tools rather than inherent complexity. To support this claim, Cammarata reinterpreted a neuron previously believed to arbitrarily group cars and foxes, identifying its actual function as detecting pointy objects.
Applying this methodology to GPT-2 XL, Cammarata estimated that roughly 70% of neurons in a random sample of 500 have a single main meaning. This represents a sharp increase from the 1.7% interpretability rate he observed while working at OpenAI in 2023.
Reactions to the findings are mixed. An anonymous OpenAI employee described the discovery as a potential major moment in scientific history. However, Neel Nanda of Google DeepMind cautioned that it remains difficult to guarantee a complete picture of model behavior and questioned if the method can scale to the millions of neurons present in frontier models.
