Llama 3 vs GPT-4: Meta Challenges OpenAI on AI Turf

The model punches above its weight class and shows promise as an emerging challenger. LLMs will also continue to expand in terms of the business applications they can handle. Their ability to translate content across different contexts will grow further, likely making them more usable by business users with different levels of technical expertise. One of the most popular infographics displays GPT-3 as a dot next to a big black hole named GPT-4.

ChatGPT has broken several records within a few days of its release, which shows its capabilities. It was OpenAI’s GPT-3.5 that was used to power ChatGPT, which is now ChatGPT App the most popular AI chatbot in the world. But things are always progressing in the tech industry, so it’s no surprise that GPT-3.5 now has a successor in GPT-4.

GPT-4

They can both respond to prompts like questions or requests, and can provide responses very similar to that of a real person. They’re both capable of passing exams that would stump most humans, including complicated legal Bar exams, and they can write in the style of any writer with publicly available work. ChatGPT-4 is the newest model of OpenAI’s chatbot, known generally as ChatGPT. ChatGPT is powered by artificial intelligence, allowing it to answer your questions and prompts far better than previous chatbots. ChatGPT uses a large language model powered by a GPT (Generative Pre-trained Transformer) to provide information and content to users while also being able to converse.

There, we revealed OpenAI’s high-level approach to the architecture and training cost of GPT-4 in relation to various existing models. From GPT-3 to 4, OpenAI aims to scale up by a factor of 100, but the problem lies in the cost. Dense Transformer is the model architecture used by OpenAI GPT-3, Google PaLM, Meta LLAMA, TII Falcon, MosaicML MPT, and other models. We can easily list over 50 companies that train LLM using this same architecture.

We learn that the picture inputs are still in the preview stage and are not yet accessible to the general public. These tests are useful for gauging level of understanding rather than IQ. The fourth generation of GPT (GPT-4) has improved context understanding and intelligent reaction times in complicated corporate applications.

The major problem with these pre-trained models is that they only support the native entities they were trained for… and these entities are rarely useful in real life projects. Most companies want to use NER to extract custom entities like job titles, product names, movie titles, restaurants, etc. The only solution was to create a huge dataset for these new entities through a long and tedious annotation process, and then train a new model. Everytime one wanted to support a new entity, the only solution was to annotate and train again.

This is where Meta’s Llama family differs the most from OpenaAI’s GPT. Meta releases their models as open source, or at least kind of open source, and GPTs are closed. This difference in openness significantly impacts how you work with and build products upon each. No one outside of OpenAI knows the details of how it’s built because it’s a closed-source model.

What is Gen AI? Generative AI explained

It officially released LLaMA models in various sizes, from 7 billion parameters to 65 billion parameters. According to Meta, its LLaMA-13B model outperforms the GPT-3 model from OpenAI which has been trained on 175 billion parameters. Many developers are using LLaMA to fine-tune and create some of the best open-source models out there. Having said that, do keep in mind, LLaMA has been released for research only and can’t be used commercially unlike the Falcon model by the TII.

That would make GPT-4o Mini remarkably small, considering its impressive performance on various benchmark tests. Therefore, when GPT-4 receives a request, it can route it through just one or two of its experts — whichever are most capable of processing and responding. Instead of piling all the parameters together, GPT-4 uses the “Mixture of Experts” (MoE) architecture. Previous AI models were built using the “dense transformer” architecture. ChatGPT-3, Google PaLM, Meta LLAMA, and dozens of other early models used this formula.

This is important for hardware vendors who are optimizing their hardware based on the use cases and ratios of LLM in the next 2-3 years. They may find themselves in a world where every model has powerful visual and audio capabilities. Overall, the architecture is sure to evolve beyond the current stage of simplified text-based dense and/or MoE models. It is said that the next model, GPT-5, will be trained from scratch on vision and will be able to generate images on its own.

Its performance in extracting pertinent information from biological texts has been demonstrated by its scores of 69.0% on the MMLU Medical Genetics test and 57.3% on the MedMCQA (dev) dataset. A token is selected from the output logits and fed back into the model to generate the logits for the next token. This process is repeated until the desired number of tokens is generated.

• Though outgunned in funding, Claude 2’s advanced capabilities suggest it can go toe-to-toe with even well-funded behemoths (though it’s worth noting that Google has made several large contributions to Anthropic).
• However, GPT-4 may have shown how far the MoE architecture can go with the right training data and computational resources.
• “OpenAI is now a fully closed company with scientific communication akin to press releases for products,” says Wolf.
• Such an AI model would be formed of all of these different expert neural networks capable of solving a different array of tasks with formidable expertise.
• Plus, with the different versions of models available out there, comparing them can be tricky.

Interestingly, Google has allowed a limited group of developers and enterprise customers to try out a context window of up to a whopping one million tokens via AI Studio and Vertex AI in private preview. The one to compare, as Huang walked through during his keynote, was how to train the 1.8 trillion parameter GPT-4 Mixture of Experts LLM from OpenAI. On a cluster of SuperPODs based on the Hopper H100 GPUs using InfiniBand outside of the node and NVLink 3 inside of the node, it took 8,000 GPUs 90 days and 15 megawatts of juice to complete the training run. To do the same training run in the same 90 days on the GB200 NVL72, it would take only 2,000 GPUs and 4 megawatts. If you did it across 6,000 Blackwell B200 GPUs, it would take 30 days and 12 megawatts.

The ability to produce natural-sounding text has huge implications for applications like chatbots, content creation, and language translation. One such example is ChatGPT, a conversational AI bot, which went from obscurity to fame almost overnight. GPT-1 was released in 2018 by OpenAI as their first iteration of a language model using the Transformer architecture. It had 117 million parameters, significantly improving previous state-of-the-art language models. In simpler terms, GPTs are computer programs that can create human-like text without being explicitly programmed to do so. As a result, they can be fine-tuned for a range of natural language processing tasks, including question-answering, language translation, and text summarization.

The architecture may have simplified the training of GPT-4 by allowing different teams to work on different parts of the network. This would also explain why OpenAI was able to develop GPT-4’s multimodal capabilities independently of the currently available product and release them separately. In the meantime, however, GPT-4 may have been merged into a smaller model to be more efficient, speculated Soumith Chintala, one of the founders of PyTorch. The MoE model is a type of ensemble learning that combines different models, called “experts,” to make a decision.

Some reports suggest that OpenAI’s flagship LLM includes 1.76 trillion parameters while Google LLC’s Gemini Ultra, which has comparable performance to GPT-4, reportedly features 1.6 trillion. GPT-4o is multimodal and capable of analyzing text, images, and voice. For example, GPT-4o can ingest an image of your refrigerator contents and provide you with recipes using the ingredients it identifies. Free ChatGPT users can also upload documents for GPT-4o to analyze and make inferences or summaries. It’s a specialized Llama 2 model additionally trained on 500 billion tokens of code data.

The basic idea behind guessing decoding is to use a smaller, faster draft model to pre-decode multiple tokens and then feed them as a batch to the oracle model. If the draft model’s predictions for these tokens are correct, i.e., agreed upon by the larger model, then multiple tokens can be decoded in a batch, saving a significant amount of memory bandwidth and time for each token. It is worth noting that we assume high utilization and maintain a high batch size.

The world of artificial intelligence is on the cusp of another significant leap forward as OpenAI, a leading AI research lab, is diligently working on the development of ChatGPT-5. This new model is expected to be made available sometime later this year and bring with it substantial improvement over its predecessors, with enhancements that could redefine our interactions with technology. The 30B-Lazarus model has been developed by CalderaAI and it uses LLaMA as its foundational model.

Google has focused on commonsense reasoning, formal logic, mathematics, and advanced coding in 20+ languages on the PaLM 2 model. It’s being said that the largest PaLM 2 model has been trained on 540 billion parameters and has a maximum context length of 4096 tokens. In 2021, global data center electricity use was about gpt 4 parameters 0.9 to 1.3 percent  of global electricity demand. As the capabilities and complexity of AI models rapidly increase over the next few years, their processing and energy consumption needs will too. It’s estimated that the energy consumption of data centers on the European continent will grow 28 percent by 2030.

ARTIFICIAL INTELLIGENCE

Before discussing the trade-offs faced by OpenAI and the choices they have made, let’s start with the basic trade-offs of LLM reasoning. As for why they didn’t use full-model FSDP, it may be because of the high communication overhead. Although most of OpenAI’s nodes have high-speed network connections between them, not all nodes do. We believe that the bandwidth between at least some clusters is much lower than others. Furthermore, the attention mechanism shares approximately 55 billion parameters.

The Eliza language model debuted in 1966 at MIT and is one of the earliest examples of an AI language model. All language models are first trained on a set of data, then make use of various techniques to infer relationships before ultimately generating new content based on the trained data. Language models are commonly used in natural language processing (NLP) applications where a user inputs a query in natural language to generate a result. However, if it turns out to be true massive amounts of data of ChatGPT-4 might be nearly 571 times greater as compared to the training data size of 175 billion parameters of GPT-3. ChatGPT-4 also will be utilized for multiple language applications such as text summarization, code generation, classification, language interpretation, chatbot, and grammar rectification.

Theoretically, considering data communication and computation time, 15 pipelines are quite a lot. However, once KV cache and cost are added, if OpenAI mostly uses 40GB A100 GPUs, such an architecture is theoretically meaningful. However, the author states that he does not fully understand how OpenAI manages to avoid generating “bubbles” (huge bubbles) like the one shown in the figure below, given such high pipeline parallelism. It is very likely that OpenAI has successfully borne the cost of these bubbles. In each forward propagation inference (generating one token), GPT-4 only needs to use about 280 billion parameters and 560 TFLOPs. In comparison, a pure dense model requires about 18 trillion parameters and approximately 3,700 TFLOPs of computation for each forward propagation.

• This might not be the biggest difference between the two models, but one that might make the biggest difference for most people.
• 100 trillion parameters are a low estimation for the count of neural connections in the human brain.
• After the release of ChatGPT by OpenAI, the race to build the best LLM has grown multi-fold.
• In 2022, LaMDA gained widespread attention when then-Google engineer Blake Lemoine went public with claims that the program was sentient.
• Based on the memory bandwidth requirements, a dense model with one billion parameters cannot achieve this throughput on the latest Nvidia H100 GPU server.

Preferably, the ChatGPT model is trained to ask the users to explain queries when the user requests a vague answer. However, the current updated model tries to guess the intent of the user. The ChatGPT model has ChatGPT been instructed to reject inappropriate requests, but at times also answers unsafe instructions or questions. This membership will offer priority access to the AI chatbot even during peak hours to consumers.

This is equivalent to 2-3 literature books, which GPT-4 can now write on its own. On the other hand, GPT-3.5 could only accept textual inputs and outputs, severely restricting its use. You can foun additiona information about ai customer service and artificial intelligence and NLP. GPT-3.5 has a large dataset measuring in at 17 terabytes, which helps it provide reliable results. Insiders at OpenAI have hinted that GPT-5 could be a transformative product, suggesting that we may soon witness breakthroughs that will significantly impact the AI industry.

If there is no software advantage in inference and manual kernel writing is still required, then AMD’s MI300 and other hardware will have a larger market. The batch size gradually increases over a few days, but in the end, OpenAI uses a batch size of 60 million! Of course, since not every expert sees all the tokens, this actually means that each expert processes 7.5 million tokens per batch.

At the same time, smaller and slightly less capable models can handle many of the tasks companies and individuals throw at them. Microsoft’s research division has added a major new capability to one of its smaller large language models, a big step that shows less expensive AI technology can have some of the same features as OpenAI’s massive GPT-4. It is a standalone visual encoder separate from the text encoder, but with cross-attention. After pretraining on text only, it is further fine-tuned on an additional 2 trillion tokens. If an application requires minimal latency, we need to apply more chips and divide the model into as many parts as possible.