Salesforce has launched the first-ever LLM (Large Language Model) benchmark for customer relationship management (CRM). The benchmark aims to aid businesses in assessing various LLMs for enhanced CRM functionalities.
This pioneering tool provides a structured framework for evaluating the performance of LLMs across four critical areas: accuracy, cost, speed, and trust and safety. It addresses the specific needs of sales and service applications, such as prospecting, lead nurturing, and summarising sales opportunities and service cases. Moreover, the inclusion of a publicly accessible leaderboard allows businesses to easily compare and select the LLM that best fits their CRM requirements.
The need for a specialised CRM benchmark
Until now, LLM benchmarks have predominantly focused on academic and general consumer applications, offering little relevance to business-specific needs. They often lack thorough evaluations by seasoned professionals, failing to properly consider essential factors like accuracy, speed, cost, and trust. This gap has left many CRM users without a reliable method to evaluate the efficacy of AI-driven CRM solutions. Businesses need clarity on how LLMs perform on these metrics for tailored use cases to make informed decisions.
“As AI continues to evolve, enterprise leaders are saying it’s important to find the right mix of performance, accuracy, responsibility, and cost to unlock the full potential of generative AI to drive business growth,” said Silvio Savarese, EVP & Chief Scientist, Salesforce AI Research. “Salesforce’s new LLM Benchmark for CRM is a significant step forward in the way businesses assess their AI strategy within the industry. It not only provides clarity on next-generation AI deployment but also can accelerate time to value for CRM-specific use cases. Our commitment is to continuously evolve this benchmark to keep pace with technological advancements, ensuring it remains relevant and valuable.”
A deeper look at the metrics and strategic advantages
Developed by Salesforce AI Research, this new benchmark utilises real CRM data and incorporates evaluations by expert practitioners. This approach allows companies to strategically integrate generative AI into their CRM systems by focusing on:
- Accuracy: This includes subcategories such as factuality, completeness, conciseness, and adherence to instructions, enhancing the value of the modelโs outputs to improve customer interactions.
- Cost: The cost is categorised into high, medium, and low tiers, representing the operational costs associated with different CRM applications, helping companies manage their budgets effectively.
- Speed: Assessing the model’s promptness and efficiency, faster response times lead to better customer service and streamlined operations.
- Trust and Safety: Ensuring the protection of sensitive data, compliance with data privacy laws, and the model’s ability to prevent bias and maintain security.
These metrics provide businesses with the tools to compare LLMs, choose the most suitable option, and implement more effective AI solutions in their CRM systems.
With Salesforceโs Einstein 1 Platform, clients can select from existing models or integrate their own to address unique business challenges. Clara Shih, CEO of Salesforce AI, emphasised, “Business organisations are looking to utilise AI to drive growth, cut costs, and deliver personalised customer experiences, not to plan a kid’s birthday party or summarise Othello. Our customers have been asking for a purpose-built way to evaluate and select from among the proliferation of new AI models, and we are thrilled to introduce the world’s first LLM benchmark for CRM to help them navigate the complex landscape of models. This benchmark is not just a measure; it’s a comprehensive, dynamically evolving framework that empowers companies to make informed decisions, balancing accuracy, cost, speed, and trust.”
