As large language models reach diminishing returns from simply scaling up data and computing power, AI companies like OpenAI are pioneering new training techniques that make models “think” in more human-like ways. This approach, exemplified by OpenAI’s new “o1” model, aims to overcome the delays and challenges that arise from conventional scaling efforts, which require vast resources and can encounter unpredictable outcomes.
Once a major proponent of massive pre-training with extensive data, OpenAI co-founder Ilya Sutskever recently acknowledged the limitations of the “bigger is better” philosophy. He emphasized that future advancements in generative AI may hinge more on targeted, innovative scaling rather than sheer size and power. OpenAI’s o1 model, developed with methods to enhance “inference” processing, can solve complex tasks by considering multiple possible answers before selecting the best one, simulating a step-by-step thought process. This approach has shown significant promise; OpenAI’s Noam Brown noted that giving a model even a short time to think through decisions has yielded similar performance gains to traditional scaling efforts.
The move has spurred competitors like Anthropic, Google DeepMind, and Elon Musk’s xAI to explore similar techniques. The change is also causing AI hardware demands to shift from Nvidia’s training chips toward chips optimized for inference, potentially diversifying competition in the AI chip market.
Top venture capital investors are closely watching these developments, as the shift could alter the trajectory of investments in AI infrastructure, potentially favoring distributed, cloud-based inference systems over large pre-training clusters.
