What are small language models?

Small language models, or SLMs, are language models designed with fewer parameters and lower computational requirements than the large models used at the frontier. They can summarise, extract information, rewrite text, classify content and answer focused questions, often with lower latency, memory use and energy per token.

There is no official parameter threshold at which a model becomes “small”. The term is relative to the hardware, period and comparison. A model that fits on a phone is clearly small beside a data-centre model, but two providers may apply the label differently. The useful questions are how much of the model is active, what hardware it needs and whether it completes your task reliably.

A language model can be useful without being enormous

Language models learn numerical patterns from text and use them to predict tokens. Larger models can hold more parameters and often perform well across a wider range of unfamiliar tasks. They also require more memory and computation to train and serve.

Small models trade some breadth for efficiency. Careful training data, distillation and focused fine-tuning can preserve strong performance on common or narrow tasks. Microsoft’s Phi-3 technical report, for example, describes a 3.8-billion-parameter model designed to be capable enough for deployment on a phone. Its benchmark results belong to that model and test set; they do not prove every small model rivals a larger one.

“Small” therefore describes a resource class, not a quality guarantee. A compact model can be excellent at extraction and poor at difficult reasoning, just as a large model can be unnecessary for routine formatting.

The comparison date should be stated too. What counted as compact several years ago may look ordinary now as hardware and model design improve.

Why small models usually need less energy

In a conventional dense transformer, generating a token requires calculations across most of the model’s weights. Fewer parameters generally mean fewer values to move through memory and fewer operations to perform. A compact model may also run on smaller hardware and serve more requests within a given memory budget.

That is why model size is a useful environmental lever. It is not a power meter. Numerical precision, sequence length, batching, accelerator generation and software optimisation all change energy per result. A poorly utilised small model can waste capacity, while a larger model that succeeds once may beat repeated failed calls to a weaker one.

Our article on how to reduce your AI carbon footprint uses the practical rule: choose the smallest model that reliably completes the whole task. Count retries and corrections rather than comparing one isolated token.

Input length can narrow the advantage. Processing a huge document on a compact model is still substantial work, so context discipline remains useful at every model size.

How small models are made capable

Model builders do more than remove layers. They curate training data so limited capacity is spent on useful patterns, teach smaller models from the outputs or representations of larger teachers, and fine-tune them for particular domains. Quantisation stores and calculates weights at lower precision, reducing memory and often accelerating inference.

Pruning removes weights or structures judged to contribute little. Retrieval can supply relevant documents at request time instead of asking the model to memorise every fact. Good application design narrows the problem with structured inputs, tools and validation.

Each technique has trade-offs. Distillation can reproduce limitations of the teacher. Aggressive quantisation may reduce quality. Narrow fine-tuning can weaken general behaviour. Claims should be tested on the actual task, language and safety requirements, not inferred from parameter count alone.

Evaluation data needs similar care. A model can look capable because its training material overlaps a benchmark, while failing on fresh examples from the application it is meant to serve.

Where small language models work well

Focused, repeated work is the natural territory. Examples include classifying support tickets, extracting fields from known document types, suggesting short replies, rewriting text to a house style and answering questions from a controlled knowledge base. Lower latency and cost can make local or edge deployment practical.

Running locally can improve privacy because data need not leave the device, although the application’s actual network behaviour still matters. It can also avoid continuous network access. Local inference is not automatically lower-carbon: an old laptop doing a slow job may use more energy than an efficient shared server, and the device’s embodied footprint remains.

Small models are less dependable for complex multi-step reasoning, obscure knowledge, long documents and difficult code. Escalation is sensible when testing shows the task exceeds their capability. Sustainable design is not a ban on large models; it is deliberate routing.

The boundary can be made visible to users. A service may explain that routine work uses its fast model and offer an intentional upgrade for cases where greater reasoning is worth the delay and resources.

Small, sparse and specialised are not synonyms

A small language model has a relatively modest total footprint in parameters and memory. A specialised model is trained or tuned for a narrow domain, whatever its size. A sparse model contains many parameters but activates only some of them for a given input.

That last category includes mixture-of-experts models. They can have a large total parameter count while using a smaller active subset for each token. Compute may resemble a smaller dense model, but all experts still need to be stored and distributed, so memory and communication do not vanish.

When a provider promotes a “small” or “efficient” model, ask for total and active parameters, hardware requirements, tokens per second, energy boundary and task quality. No one label answers all of those questions.

Also distinguish a downloadable model file from the memory required while it runs. Working context, caches and application processes can make deployment larger than the weights alone suggest.

How to choose one responsibly

Build a representative evaluation set, including difficult and safety-sensitive examples. Test the smallest candidate, record success, latency and estimated or measured energy, then compare the next model using the same conditions. Include retrieval, tool calls, retries and human correction.

For organisations, monitor the distribution after launch. A router that sends most routine traffic to a small model and genuine exceptions to a larger one can preserve quality without defaulting every request to maximum compute. A practical guide to sustainable AI covers the measurement and governance around that pattern.

Treechat uses smaller models for everyday questions and offers larger capability when needed. Its per-answer receipt estimates operational energy from model and token use, adds a published data-centre overhead and applies a stated grid factor. It is not direct hardware measurement. The current factors and limitations are at /methodology.

Small language models matter because much everyday language work does not require the largest available system. Used within their limits, they can make AI faster, cheaper and less energy-intensive. Used as a marketing label without task testing or measurement, “small” says very little.