A two-year-old has never read a textbook, yet after hearing only a few hundred thousand sentences, they’ve grasped the basics of their native language. A modern language model needs trillions of words and entire data centers full of energy to reach a comparable level of fluency. That gap in scale is now driving a growing community of cognitive scientists and AI researchers, as a recent MIT Technology Review report shows: if humans learn so much more efficiently, what exactly is the brain doing differently from a transformer model?
Key takeaways
- Children hear roughly 100 million words by preschool age and about 300 million by adulthood once reading is included, yet speak fluently after a fraction of that exposure.
- Meta trained Llama 3.1 on 15 trillion tokens, and researchers say current frontier models now use multiples of that figure.
- The annual BabyLM Challenge has teams train language models specifically on child-sized datasets of just 10 to 100 million words.
- A model called GPT-BERT achieved strong results on grammar benchmarks in 2024 despite training on orders of magnitude less data than large commercial models.
- Researchers are weighing several explanations: innate language capacities, multimodal learning, active exploration of the environment, and social interaction with caregivers.
A gap of several orders of magnitude
Michael C. Frank of Stanford University illustrates just how large the difference really is: to replicate a milestone that a child reaches in its first year of life in a living room, AI researchers essentially have to burn down a forest and process nearly all human knowledge available on the internet. An older model like GPT-2, trained on about 30 million words, still produced largely nonsensical output, according to researchers cited in the report. A toddler who has heard a similar number of words is, by that point, already speaking in correct, if simple, sentences.
Four competing explanations
The classical theory, tracing back to Noam Chomsky, assumes innate grammatical structures hardwired into the human brain. That view has come under pressure as purely statistical language models demonstrate that many grammatical patterns can apparently be inferred from large volumes of text without any built-in language rules. Brenden Lake of Princeton University takes a different approach: in his SAYCam project, toddlers wore head-mounted cameras for two and a half years, generating footage used to train multimodal models on just 61 hours of video that could genuinely recognize objects and link them to words. His own finding immediately tempers the achievement, though: ‘We don’t end up with a two-year-old.’ Alison Gopnik of UC Berkeley, meanwhile, stresses the active role children play, choosing for themselves which stimuli to explore and which cause-and-effect relationships to test, while language models so far learn purely passively from fixed text corpora. Elizabeth Bonawitz of Harvard University shows in her research that children weigh information differently once they recognize an adult is deliberately trying to teach them, reasoning not just about the facts but about the teacher’s intent.
The BabyLM Challenge as a testing ground
To put these theories to a practical test, linguist Alex Warstadt of UC San Diego launched the BabyLM Challenge in 2022. Teams compete to train language models specifically on child-sized datasets of 10 or 100 million words, far less than the trillions of tokens used by large commercial systems. In the 2024 round, a hybrid model called GPT-BERT achieved notably strong results on grammar benchmarks despite training on orders of magnitude less data than much larger comparison models, as documented in the associated research paper. One detail turned out to be surprising: the intuitive assumption that models would learn better if fed simple language first and more complex language later, mirroring how children are spoken to, known as curriculum learning, performed worse than expected in the experiments.
Why this is more than academic curiosity
The question of data efficiency has long since stopped being a niche linguistics topic; it touches a very practical problem for the entire industry. High-quality, freely usable text data from the internet could, according to several researchers, be largely exhausted by the 2030s, a bottleneck that also connects to the growing debate over the rising share of AI-generated text on the open web, which is itself becoming increasingly unusable as training material. More efficient learning methods modeled on how children acquire language could therefore do more than solve an academic puzzle; they could also help train better language models for languages with limited available text data, from Serbian to Sámi. That data-efficient training could also matter commercially was hinted at recently by the case of startup Inherent, whose comparatively small model Faraday claims, in figures not yet independently verified, to match much larger systems on scientific replication tasks.
Context and outlook
Research has not yet delivered a single breakthrough that closes the gap between how children and machines acquire language, more a mosaic of partial explanations, each illuminating a piece of the puzzle without any one being sufficient on its own. Gopnik suspects that fundamental progress may only come with a new generation of models succeeding today’s transformer architecture, one that builds in developmental psychology from the ground up. Until then, the finding remains a useful reality check for the AI industry: raw compute and ever-larger datasets are one path to more capable models, but clearly not the only one nature knows.