Larger language models are not necessarily better as companies seek returns on AI investments
Small language models – streamlined versions of larger ones – are expected to gain popularity
LARGE language models (LLMs) – artificial intelligence (AI) systems such as ChatGPT that process vast amounts of text data to comprehend and generate human language – have been touted as a game changer with the potential to make workers far more productive, or even replace them in certain jobs.
But, powerful as they may be, industry watchers believe that LLMs will need to be leaner and less expensive to run before they are more widely adopted in commercial use cases.
Sri Ambati, founder and chief executive of AI startup H2O.ai, noted that LLMs have been effective at demonstrating that computers can answer questions about a wide range of topics.
However, applying LLMs to a call-centre AI agent would be akin to asking someone with a doctorate to answer the phone quickly and effectively, he said.
Moreover, the cost of AI applications powered by LLMs can be prohibitive, Ambati added.
In his estimation, a single LLM-powered AI use case costs anywhere from half-a-million to a million dollars. If a company has a 1,000 use cases spread across different functions, such as marketing and customer support, it could be looking at spending up to a billion dollars just to apply AI.
Intuitively, he explained, it makes sense to create specialised models for companies to use, instead of creating ever-larger LLMs that can be more expensive to operate.
Enter small language models (SLMs), which are essentially streamlined versions of LLMs.
H2O.ai’s own SLM, dubbed Danube, has been trained with six trillion tokens – words, sub-words or characters that together form the data that the model was trained on.
Versions of the model were then created with four billion, two billion and 0.5 billion parameters, or variables that can be fine-tuned to make it more accurate. This is significantly fewer than the number of parameters in larger and more complex models. For instance, OpenAI’s GPT-4 model is estimated to have about 1.8 trillion parameters.
Ambati said that his company has trained small models with AT&T’s call-centre data to create a call classification model. He noted that the US telco opted to try this solution after already using LLMs.
“We still have to do data cleaning, data curation and labelling, and synthetic data (generation) sometimes, and distilling,” he pointed out, adding that once that is done, AT&T has been able to achieve “dramatic” cost savings and faster responses from the model.
Good things in small packages
Similarly, NTT Research president and CEO Kazuhiro Gomi said that in his experience, companies would like to adopt trained AI for specific tasks in larger processes such as accounting.
At NTT’s R&D Forum 2024 in November, the company demonstrated AI agents powered by the company’s LLM, dubbed Tsuzumi.
Among its selling points are that it is lightweight and can run on a single graphics processing unit (GPU), unlike other LLMs, which may have to be run on multiple GPUs.
“(Tsuzumi) is lighter, therefore you can put that on different hardware devices,” noted Gomi, adding that different devices with varying levels of performance may be needed for different tasks.
While a smaller model may not be able to compete with more sophisticated models by companies such as OpenAI in all tasks, he believes that they can be fine-tuned to operate in specific environments.
One of the demos showed how a Tsuzumi-powered AI agent could run on a local Windows desktop and take natural language instructions to help someone procure pens. It could order the pens and fill out lengthy procurement forms for the order based on pre-written natural language instructions.
Gomi said that companies will need to take three steps when they first consider adopting AI. They will need to define what they want to achieve, how they want to adopt AI and how they want to collect the data to build the models.
“That’s a common challenge that most of the companies are facing,” he said, adding that companies are getting better at making such decisions. Things should also improve as they begin to see their investments in AI pay off.
AI Singapore senior director of AI products Leslie Teo said that initially, LLM creators expected application makers to call on models online. This meant that large models could be hosted on powerful servers and their responses could be served to users over the Web.
However, he added that small models are necessary, especially in cases where users may be in areas with patchy Internet connectivity.
He noted that one of Indonesian tech company GoTo’s smallest AI models is only 80 kilobytes in size, and is used to listen for users saying its “wake word” to trigger its AI assistant.
Teo said that even laptop and chip manufacturers, such as Dell and Qualcomm, have reached out to ask if AI Singapore can optimise its SEA-LION LLM, or South-East Asian Languages in One Network, to run locally on neural processing units, or NPUs.
NPUs are cheaper, lower-powered semiconductor chips that have been optimised to do AI calculations, and are a requirement on new Windows AI PCs. Such chips are expected to run AI assistants using minimal power and without accessing the Web.
“I think this is how things are going to be used. We won’t have one big model; we will have many small models and they all work together in a system,” he said.
The national AI programme will soon produce a version of its SEA-LION LLM with 70 billion parameters, but Teo added that he does not expect users to adopt this model.
Instead, AI Singapore will further optimise its AI model in 2025 to a size of about one billion to three billion parameters, which he hopes that companies will use to generate economic value.
Security concerns
Aside from the benefits of running on cheaper hardware and using less power, SLMs will also find use in more air-gapped environments, where computers do not have access to the Internet.
Cybersecurity provider Ensign Infosecurity’s head of Ensign Labs Tan Ah Tuan said that the company has adopted AI in Apollo, its threat-hunting platform, which generates reports with an LLM.
The report tells clients what vulnerabilities their networks may have, how potential attackers can compromise the network, and what clients can do to fortify their networks.
He added that because the solution is deployed on critical information infrastructure that are air-gapped from the Internet, the LLM has to be optimised to run offline.
Furthermore, because the company has fine-tuned its LLM to only reference threats detected in a client’s network, the LLM does not hallucinate, or generate erroneous responses common to that of other cloud-based LLMs such as OpenAI’s ChatGPT and Google’s Gemini.
Already, he said, the version of Meta’s LLM, Llama 3, that the company has customised can produce reports that are better than what employees could previously write manually. To ascertain this, the company concurrently had employees write more than 100 reports, as well as generate them using the LLM.
“We have a criteria that when the client reads (the report), they’re able to copy and paste it in an e-mail and send it to their boss without having to do any edits,” he said, adding that the lack of clarity with human-written reports was an issue for clients. Tan noted that in his experience, newer larger models do not necessarily produce significantly better results. “Our goal was to be better than humans. We don’t need it to be perfect.”