Apparently the secrets behind GPT-4 have been revealed, and the results have left many surprised. Although the company had kept key details about GPT-4’s internal structure and architecture secret, recent rumors indicate that the model is not a major technical innovation, but rather a combination of eight smaller models. Despite this, GPT-4 is still considered the best existing language model today.
A startling revelation
George Hotz, founder of the autonomous driving startup Comma.ai, was the first to leak this information. According to Hotz, GPT-4 is not a massive monolithic model like its predecessors GPT-3 and GPT-3.5. Instead, it is a combination of eight smaller models, each with 220 billion parameters. This leak was later endorsed by Soumith Chintala, co-founder of PyTorch at Meta, and Mikhail Parakhin, Microsoft Bing’s AI lead.
The details behind the model
Although not yet officially confirmed, this leak reveals key information about the GPT-4 model and raises questions about the current state and future of artificial intelligence. The strategy used by OpenAI in combining eight smaller models within GPT-4 is known as “expert blending” and is not something new or invented by OpenAI. In fact, Google engineers such as William Fedus and Trevor Cai had successfully implemented this technique in 2021.
The mystery behind GPT-4
OpenAI’s decision to keep the details of GPT-4 secret has been the subject of both praise and controversy. On the one hand, the lack of information has generated excessive expectations and has fueled the conversation around the model. On the other hand, it has allowed OpenAI to maintain its reputation and aura of mystery. The company had already cemented its status in the AI ​​field with the success of its ChatGPT model, which led them to hide the less impressive aspects of GPT-4 and suggest that it was a revolutionary advance.
The hidden goals of OpenAI
Hiding the details of GPT-4 has allowed OpenAI to achieve three key goals. First, it has stimulated people’s imaginations and encouraged speculation about the power of the model. This has reinforced the OpenAI narrative about the need to plan and regulate artificial general intelligence (AGI). Second, it has prevented open source initiatives and competitors like Google from copying the techniques they were supposed to have invented or discovered. However, this leak reveals that GPT-4 is not as revolutionary as previously believed, but rather a combination of smaller models.
A brake on the advancement of AI?
This leak raises the question of whether OpenAI and the industry in general are running out of ideas in the field of AI, as Hotz suggests. He also questions whether progress in the field of AI is being as fast as it is made out to be. Although the GPT-4 is still an impressive model in terms of performance and ability for writing and programming tasks, the revelation that it is a combination of smaller models may affect the perception of its technological advancement.
In The Algorithmic Bridge article we already have our first impressions on the subject:
- Dave Hitz points out that the performance of GPT-4 is what matters and that the architecture is not relevant as long as it works well.
- Alberto Romero explains that how GPT-4 performance was achieved matters and criticizes the way OpenAI has handled the situation.
- Pouria Mistani suggests that the model combining approach is not evidence of a move towards artificial general intelligence (AGI).
- Pär Winzell highlights the acceleration of the AI ​​field and raises concerns about the uncontrolled development of GPT-4 level models in the wrong hands.
- Alberto Romero shares the concern about the accelerated development of the field and emphasizes that the advances in AI may be being exaggerated.
Let’s see how the story will unfold over the next few weeks.
