In a wonderful act of self-criticism, one of the architects of the transformer technology that powers chatgpt, cloudAnd virtually every major AI system told an audience of industry leaders this week that artificial intelligence research has become dangerously narrow — and that it is moving beyond its creation.
lion joneswho co-authored the seminal 2017 paper "all you need is attention" And coined the name too "Transformer," Gave an unusually candid assessment in ted ai conference Tuesday in San Francisco: Despite unprecedented investment And with the influx of talent in AI, the field has become siled around a single architectural approach, potentially blinding researchers to the next big breakthrough.
"Despite the fact that there has never been more interest and resources, money and talent, this has narrowed the scope of the research we are doing," Jones told the audience. He argued that he was the culprit "excessive pressure" From investors seeking returns and researchers struggling to stand out in a crowded field.
The warning takes on particular significance given Jones’ role in AI history. transformer architecture He helped Google develop what has become the foundation of the generative AI boom, enabling systems that can write essays, draw pictures, and engage in human-like conversations. his paper is done Cited over 100,000 timesMaking it one of the most influential computer science publications of the century.
Now, as Tokyo-based CTO and co-founder Sakana AIJones is clearly abandoning his creation. "Earlier this year I personally decided that I would significantly reduce the time I spend on transformers," He said. "I’m obviously exploring and looking for the next big thing now."
Why more AI funding has led to less creative research, according to a Transformer pioneer
Jones presented a picture of the AI research community based on what he called a paradox: more resources leading to less creativity. He said researchers are constantly checking to see if they have gone "get caught" by competitors working on similar ideas, and by academics choosing safe, publishable projects over risky, potentially transformative projects.
"If you’re doing standard AI research right now, you have to assume that there are probably three or four other groups doing something similar, or maybe exactly the same," Jones described an environment where "Unfortunately, this pressure hurts science, because people are rushing their papers, and this is reducing the amount of creativity."
He drew an analogy from AI itself – the "exploration vs exploitation" Trade-offs govern how algorithms find solutions. When a system exploits too much and explores too little, it finds mediocre local solutions while missing out on better alternatives. "We’re almost certainly in the same situation right now in the AI industry," Jones argued.
The implications are serious. Jones recalled the period just before the emergence of Transformers, when researchers were endlessly tweaking recurrent neural networks – the previous dominant architecture – for incremental gains. Once transformers arrived, all that work suddenly seemed irrelevant. "How much time do you think those researchers would have spent trying to improve recurrent neural networks if they had known that something like Transformers was right around the corner?" he asked.
He’s worried the field is repeating the same pattern. "I’m worried that we’re at a point right now where we’re just focusing on one architecture and just sorting it out and trying different things, where a breakthrough may be right around the corner."
How the ‘All You Need to Pay Attention’ paper was born not from pressure, but from freedom
To illustrate his point, Jones described the conditions that allowed transformers to emerge in the first place – a sharp contrast to today’s environment. He said, the project was "Very organic, bottom up," born from "Talking at lunch or writing randomly on the whiteboard at the office."
seriously, "We didn’t really have a good idea, we really had the freedom to spend time and work on it, and more importantly, we had no pressure from management," Jones recounted. "No pressure to work on a particular project, publish multiple papers to advance a certain metric."
Jones suggested that freedom is largely absent today. Even researchers were recruited at huge salaries – "In some cases, literally a million dollars a year" -Can’t feel empowered to take risks. "Do you think they feel empowered to try out their wilder ideas and more fantastical ideas when they start their new position, or do they feel extreme pressure to prove their worth and, once again, go for the low hanging fruit?" he asked.
Why one AI lab is betting that research freedom is worth more than million-dollar salaries
Jones’ proposed solution is intentionally provocative: turn on "Explore Dial" And share findings openly, even at competitive costs. He acknowledged the irony of his situation. "It might seem a little controversial to hear one of the Transformers writers stand up on stage and say that he’s absolutely sick of them, but that’s fair enough, right? Except for seven people, I’ve been working on them longer than anyone else."
But Sakana AIJones said he is attempting to recreate that pre-revolutionary environment with minimal pressure to pursue nature-inspired research and publications or compete directly with rivals. He passed a mantra to the researchers from engineer Brian Cheung: "You should only do research that wouldn’t happen if you didn’t do it."
An example is Sakana "continuous thinking machine," Which involves brain-like synchronization in neural networks. One employee who presented the idea told Jones that she would have faced suspicion and pressure from previous employers or academic positions not to waste time. In Sakana, Jones gave him a week to research. This project was so successful that it made headlines neuripsA major AI conference.
Jones even suggested that freedom in hiring was better than compensation. "This is a really great way to get talent," He said of the exploratory environment. "Think about it, talented, intelligent people, ambitious people will naturally seek out this kind of environment."
Transformers’ success could block AI’s next breakthrough
Perhaps most provocatively, Jones suggested that Transformers may be victims of their own success. "The fact that current technology is so powerful and flexible… has stopped us from looking for a better one," He said. "It makes sense that if current technology were worse, more people would be looking for a better one."
He was careful to make it clear that he was not dismissing ongoing Transformer research. "There is still a lot of important work to be done on the current technology and there will be considerable value in the coming years," He said. "I’m just saying that given the talent and resources we currently have, we can afford to do much more."
His final message was one of cooperation over competition. "In fact, from my point of view, it is not a competition," Jones concluded. "We all have the same goal. We all want to see this technology advance so that we can all benefit from it. So if we could all collectively turn up the explore dial and then openly share what we find, we could reach our goal much faster."
High risk of AI exploration problem
The comments come at a critical moment for artificial intelligence. The industry is grappling with growing evidence that larger transformer models are simply being manufactured may be approaching diminishing returnsLeading researchers have begun to openly discuss whether the current paradigm has fundamental limitations, with some suggesting that continued progress toward more capable AI systems will require architectural innovation – not just scale.
Jones’ warning suggests that finding those innovations may require dismantling the incentive structures that have driven AI’s recent boom. with Billions of dollars are flowing into AI development every year And because of secrecy among laboratories and fierce competition that promotes rapid publication cycles, the exploratory research environment he described is fast slipping away.
Yet his insider perspective carries unusual weight. As someone who helped create the technology that now dominates the field, Jones understands both what it takes to achieve successful innovation and what the industry risks by abandoning that approach. His decision to move away from Transformers – the architecture that built his reputation – adds credibility to a message that might otherwise seem like a contradictory position.
Whether AI power players will heed the call remains uncertain. But Jones offered a stark reminder of what’s at stake: The next Transformers-scale breakthrough may be just around the corner, pursued by researchers with the freedom to explore. Or while thousands of researchers rush to publish incremental improvements to the architecture, it may lie undiscovered, in the words of Jones, one of its creators. "Absolutely sick."
After all, he’s been working on Transformers longer than almost anyone else. He will know when it is time to move on.
