In a follow-up to my previous exploration “Evaluating the Accuracy of Large Language Models: A Deep Dive into Response Time and Relevance”, I’ve delved into the realm of OpenAI‘s latest advancements again.
The Experiment Setup
This time, I benchmarked exclusively OpenAI models, namely GPT-3.5 Turbo, GPT-4, and the new GPT-4 Turbo Preview, employing the new embedding models “text-embedding-3-small” and “text-embedding-3-large“, as well as revisiting the classic “text-embedding-ada-002“, where I’ve compared these new results with my previous findings of about a month ago, for intriguing insights.
The detailed Hardware Specifications and the Benchmark Details from my previous test provide a backdrop for understanding these results.
The major findings from this recent benchmark are as follows:
GPT-3.5 Turbo: Fast, Sacrifices Faithfulness
Fastest Response Time: GPT-3.5 Turbo, when coupled with “text-embedding-ada-002” and a chunk size of 128, still leads in Response Time with 0.81 seconds. However, this speed comes at a cost: both Faithfulness and Relevancy are noticeably lower.
GPT-4 shines in quality, showcasing deliberate, thoughtful responses
Slowest Response Time: GPT-4, using the “text-embedding-3-small” with an overly large chunk size of 4096, recorded the slowest Response Time (3,89 seconds, still a very respectable time overall, though). This confirms that the “small” embedding model may not be optimized for such large chunk sizes. Despite this, it delivered some of the highest scores in faithfulness and relevancy.
GPT-4 excels in Faithfulness and Relevancy
Top Performer in Faithfulness and Relevancy: GPT-4, when utilizing the “text-embedding-3-large” with a chunk size of 1536, emerged as the top performer. This indicates a significant correlation between the choice of embedding model and the quality of output in terms of Faithfulness and Relevancy.
Dynamic performance: ‘text-embedding-ada-002’ dip with GPT-3.5, improves with GPT-4
Comparative Performance of “text-embedding-ada-002”: Interestingly, the performance of “text-embedding-ada-002” has dipped compared to the previous month when used with GPT-3.5 Turbo. However, its combination with GPT-4 shows an improvement, presenting a curious dynamic in the interaction between the model versions and embeddings.
GPT-3.5 Turbo: Embedding choice inconsequential; GPT-4 prefers ‘large’
Choice Between “text-embedding-3-large” and “text-embedding-3-small”: For GPT-3.5 Turbo, the choice between these two embedding models results in similar performance levels. However, GPT-4 and the new GPT-4 Turbo Preview display significantly better performance with the “large” embeddings, almost by a factor of two.
These findings, displayed in the above tables, underscore the nuanced relationship between the choice of Embedding Models, Chunk Szes, and the Performance of different LLM versions.
However, it’s crucial to remember that these tests are representative of my specific hardware and testing conditions.
Therefore, while they offer valuable insights and guidance, they should not be considered universally definitive. Different hardware configurations and use cases might yield varying results, highlighting the importance of contextual application of these findings.
In the ever-evolving landscape of large language models and their applications, such benchmarks offer a snapshot of capabilities and limitations, guiding users in making informed decisions tailored to their specific needs and conditions.
In the context of AI for Business, it’s crucial to make informed decisions regarding the tools and strategies that are most suitable for the current business landscape. Selecting the right AI tools and approaches can significantly impact a business’s competitive edge.
Discover in the video below how GPT-4 is still performing a bit better than GPT-4-Turbo.
FAQs for “Evaluating LLM Accuracy With New OpenAI Embeddings”
What was the purpose of this new experiment?
The experiment aimed to evaluate the accuracy and response times of different OpenAI models, specifically GPT-3.5 Turbo, GPT-4, and the new GPT-4 Turbo Preview, using the latest embedding models.
Which OpenAI models were benchmarked in this study?
The study focused exclusively on OpenAI models: GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo Preview.
What are the new embedding models tested in this experiment?
The new embedding models tested include “text-embedding-3-small”, “text-embedding-3-large”, and a revisit to the “text-embedding-ada-002”.
How do the new findings compare to previous results?
The study compares the results with previous findings to provide insights into performance changes and advancements in model accuracy and response times.
What were the major findings from the recent benchmark?
Key findings include the fastest response time with GPT-3.5 Turbo but lower faithfulness and relevancy, the high performance of GPT-4 in faithfulness and relevancy, and the dynamic performance of the “text-embedding-ada-002” across different model versions.
How does the choice of embedding model affect performance?
The choice of embedding model significantly impacts the quality of output, with “text-embedding-3-large” showing better results with GPT-4, indicating a notable correlation between embedding model selection and output quality.
What does the term ‘faithfulness’ refer to in this context?
In this context, ‘faithfulness’ refers to the accuracy and reliability of the model’s responses concerning the input data.
Why is the ‘chunk size’ important in these benchmarks?
Chunk size affects the processing speed and quality of responses; different sizes can lead to variations in response times and accuracy, making it a crucial factor in model performance.
Can these benchmark results be generalized for all hardware and use cases?
No, these results are specific to the author’s hardware and testing conditions. Performance may vary with different hardware setups and use scenarios, underscoring the importance of contextual application.
What implications do these findings have for AI in business?
These benchmarks offer guidance for selecting appropriate AI tools and strategies, impacting a business’s competitive edge by informing decisions tailored to specific needs and conditions.