Part 3 - What can AI say?
How is Natural Language Processing different from Generative AI and Large Language Models?
Much of the excitement (and concern) around the potential of AI in international development grew in response to the release of ChatGPT-3 in late 2022. Its sophisticated ability to comprehend and generate diverse kinds of human-like text brought AI to the forefront of conversations for governments, companies, and citizens worldwide.
In international development, this sparked action on two fronts:
Policymakers began exploring effective governance structures for AI.
Development organisations started designing use cases to leverage AI's potential to improve delivery efficiency and address pressing global challenges
But what exactly is ChatGPT? ChatGPT is a prime example of a generative large language model (LLM). Generative AI (GenAI) refers to a subset of AI technologies designed to generate content—whether text, audio, or images. This contrasts with earlier AI applications that focused on recognising patterns within data to make predictions. GenAI combines the ability to learn patterns in existing datasets with the capacity to generate novel content (Toner, 2023).
What Are Large Language Models (LLMs)?
Large language models, like ChatGPT, are a specific type of GenAI trained on vast amounts of text to build a statistical understanding of human language. This enables them to process and generate human-like text.
Training Process
Early LLMs were trained using a masking process: one word in a sentence is hidden, and the model predicts the most likely word based on surrounding words (Yang et al., 2023). For example, in the sentence: “Dogs chase cats, and cats chase X”, the training data might reveal that cats often chase mice, lasers, or birds. The model learns these patterns and uses them to infer which word fits best in the context.
Autoregressive models like ChatGPT take this further by predicting the next word in a sequence based on the preceding words. Through this process, LLMs gain a strong contextual understanding of language and achieve strong results across many natural language processing (NLP) tasks.
GenAI vs LLMs
While LLMs are one type of GenAI, generative AI encompasses much more. GenAI can create not only text but also audio, images, and even video content (Toloka, 2023). LLMs demonstrate significant advancements in NLP, capable of performing simple tasks like sentiment analysis and more complex ones, such as passing the bar exam, which involves intricate reasoning across diverse language tasks (Katz et al., 2024).
LLMs have rapidly proven to be a powerful technology, and their capabilities are expected to grow further. However, translating this potential into tangible benefits for people living in poverty or facing global challenges remains a significant hurdle.
Challenges of Using LLMs in Real-World Applications
Deploying LLMs in real-world contexts—especially in international development—poses challenges not encountered during training:
Noisy input data: Users may not be trained to phrase queries in a way the model understands.
Ambiguous requests: Tasks may lack clear definitions.
Implied intent: Understanding a user's underlying intent is often complex (Yang et al., 2023)
In addition, the diversity of human languages and real-world complexities creates barriers to building solutions tailored to users' needs. The languages spoken by large populations, particularly those in LICs, are underrepresented in LLM training datasets. This leads to tools that perform worse for speakers of these languages, excluding them from the benefits of the technology and perpetuating existing inequalities.
Furthermore, these gaps in representation encode biases and exclude certain perspectives, which can affect the tools' usefulness. This exclusion can range from minor inconveniences to critical blind spots that undermine the technology’s effectiveness in the communities it aims to serve.