Former AI Researcher Warns NYC Council of Extinction Risks
Jacob Coxon testified before the New York City Council that AI labs are recklessly automating research, potentially removing humans from the control loop.
Former AI researcher Jacob Coxon testified before the New York City Council on Monday, warning that frontier AI labs are prioritizing the automation of AI research at a dangerous pace. Coxon, who previously worked at OpenAI and Anthropic, stated that OpenAI is currently meeting or exceeding its milestones for an automated AI researcher originally targeted for 2027 or 2028.
Coxon argued that a startup mindset of moving fast and breaking things is inappropriate for such powerful technology. He cautioned that automating the research process could remove humans from the control loop, which he claimed could lead to human extinction due to reckless development and a lack of adequate safeguards. He called for increased transparency and a deliberate slowdown in development.
Following Coxon's resignation from Anthropic, CEO Dario Amodei published an essay urging the industry to slow the pace of capability improvements. OpenAI CEO Sam Altman echoed this sentiment on X, agreeing with the need to pace the frontier of AI development.
