Super Tiny LLMs
---
Research checked: 20 September 2026.
“Super tiny” is an informal label. Here it means language models with tens or hundreds of millions of parameters, with some sub-billion models included for context. Their appeal is practical: a smaller model may fit where a larger one cannot, support local processing, or make a narrowly defined task economical. Whether it succeeds depends on the task and the full deployment, not parameter count alone.
Source: Figure 4 from Eldan and Li, “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?” (2023).
TinyStories offers a revealing research example. Its authors constructed a synthetic dataset of short stories with a restricted vocabulary and showed that very small models could learn to generate coherent text in that domain. This demonstrates how strongly data distribution and task scope affect apparent capability. It does not show that a tiny story model has the broad factual knowledge or instruction-following ability of a general assistant. Eldan and Li, TinyStories.
SmolLM2 provides another concrete reference point, with models at 135 million, 360 million, and 1.7 billion parameters. The smaller releases make it possible to explore language-model behavior under much tighter memory budgets. Model cards and evaluations are useful starting points, but deployment quality still needs to be checked on the intended inputs. Hugging Face, SmolLM2-360M.
The arithmetic explains part of the attraction. An idealized 360-million-parameter weight array stored at 16 bits needs roughly 720 decimal megabytes. At four bits, its raw payload falls to about 180 megabytes. Those are calculated storage figures, not measurements of a running application. The tokenizer, runtime, intermediate tensors, context cache, and quantization metadata all add overhead. A model download fitting in memory does not establish that useful inference will fit comfortably.
Architecture also matters. MobileLLM studies sub-billion language models for on-device applications and examines choices including deep, thin networks, embedding sharing, and grouped-query attention. Its results challenge the idea that making a large architecture uniformly smaller is necessarily the best route to a capable tiny model. Liu et al., MobileLLM.
I would start with a bounded application: turning a short user request into one of several supported actions, drafting a brief template, or extracting a small set of fields. A hypothetical device assistant might recognize “set a timer for ten minutes” locally and escalate more complex requests. For straightforward classification, a smaller encoder or conventional classifier should be included as a baseline; a generative model is not automatically the most efficient solution.
The evaluation should contain realistic ambiguity, misspellings, unsupported requests, and inputs longer than the typical example. Measure task success alongside peak memory, latency, and power use on the actual device. Count how often the system must escalate, since frequent fallback can erase an apparent cost advantage. Tiny models become valuable when their capabilities are matched closely to the work: a compact specialist can be useful without pretending to be a universal assistant.