All posts

9 min read Fabio Cingottini

Small Language Models: Are We There Yet?

Small models now run on a laptop, but the general-purpose ones still trail the giants. Where they win is one narrow job, done in milliseconds for a fraction of a cent.

A huge whale glides through a deep blue sea while a school of tiny fish swims beside it, each carrying a pebble, a shell or a strand of seaweed.

A bold claim from an unlikely source

In June 2025, a team at NVIDIA Research and Georgia Tech published a position paper with a title that reads like a dare: Small Language Models are the Future of Agentic AI.

It’s a funny thing to hear from NVIDIA. The company sells the GPUs that giant models run on, and at Computex 2026 its CEO presented AI agents as “the new computing pattern”. But the researchers’ argument is hard to dismiss: most of what an AI agent does all day is narrow and repetitive, and for that kind of work a small model is powerful enough, better suited, and a lot cheaper.

Almost a year and a half later, I wanted to check. Are we there yet?

So what is a small language model?

A language model is, at its core, a very large file of numbers called parameters. Think of them as the knobs the model adjusted while it learned to predict the next word. More knobs, more room for knowledge and nuance.

The giants behind the chat apps we use every day have hundreds of billions of parameters, some more than a trillion. A small language model, an SLM, has a few billion or less.

There’s no official cutoff. NVIDIA’s paper defines an SLM by what it can do rather than by a number: run on an ordinary consumer device, fast enough to serve one person. In practice, that means models under roughly 10 billion parameters. And an LLM? With a straight face, the paper defines it as any language model that isn’t small.

The trade-off is simple. A bigger model knows more and reasons better. A smaller one knows less, but costs less, answers faster, and can run on your own machine, without your data ever leaving it.

How much does a model weigh?

Parameters are numbers, and numbers take up space. The back-of-the-envelope math is easy: at full precision each parameter takes 2 bytes. Compressed to 4 bits, a common trick called quantization, it takes about half a byte, with a small loss in quality.

ModelParametersFull precision4-bit
Mistral Large 3675 billion~1.35 TB~340 GB
A typical 4B small model4 billion~8 GB~2 GB
Qwen3-0.6B0.6 billion~1.2 GB~0.3 GB

The first one needs a server full of data-center GPUs. The last one fits on a phone. Real downloads run a bit heavier than the math, because of the vocabulary and the packaging around the weights: the official Qwen3-0.6B files on Hugging Face weigh about 1.5 GB.

A side note on naming, because this industry has a sense of humor: Mistral’s model called Small 4 has 119 billion parameters. Small, like a small elephant.

If you want to feel the difference, install Ollama and run this:

ollama run qwen3:0.6b

It downloads about half a gigabyte, and a minute later you are chatting with a language model running entirely on your laptop. No account, no API key, no internet once it’s downloaded. The first time it’s a little magical.

Small generalists: good, but a step behind

The models you can run at home are mostly small generalists: shrunken versions of the chat models we already know. Google has Gemma, Alibaba has the small Qwen models, Microsoft has Phi, Hugging Face has SmolLM3, and the Chinese startup ModelBest has MiniCPM, which Samsung is reportedly putting on its flagship phones.

They are genuinely good. They chat, rewrite, translate, answer simple questions. ModelBest’s MiniCPM 3.0, with just 4 billion parameters, matches GPT-3.5 on several benchmarks.

Read that sentence again, though. A small model today is about as good as a big model was two or three years ago. Impressive, and also a long time in this field. Ask them something hard, a tricky bit of reasoning or a fact from the edges of their knowledge, and the gap shows. They know less of the world, so they make up more of it: the Pleias team points out that models under 4 billion parameters often hallucinate more and remember less.

So if you want a smaller ChatGPT, you get a ChatGPT that’s a bit worse. Which raises a better question: what if we stopped asking small models to do everything?

Models that do one thing

Think about what an app or an AI agent actually does all day. Not poetry. It reads a support email and decides who should handle it. It finds the order number in a message. It shrinks a 20-page document down to the paragraph that matters. Sorting, extracting, summarizing, compressing: the same few chores, thousands of times a day.

For these chores you don’t need a model that can write anything. You need one that can only give the answers you allow.

That’s the idea behind a new wave of task-specific models. A general model writes its answer word by word, and in theory it could write anything, including a wrong answer in the wrong format. A task-specific model picks from a menu you define, or copies pieces straight from the text it read. It can still pick the wrong option, but it can’t invent a new one. It’s the difference between asking a novelist to fill in a form and handing over a form with only checkboxes.

Here’s what that looks like on a real support message, using examples from LangChain:

Order A-1042 failed with ERR_PAYMENT_DECLINED. I already tried a different card.

  • Classify it as billing, technical or account → billing.
  • Extract the details → order A-1042, error ERR_PAYMENT_DECLINED, email not mentioned yet.
  • Compress the 18,402-character refund policy page the agent fetched → the 3,110 characters about this kind of case.

Who makes them

ModelWhat it doesRuns onPrice
ScaleDownClassifies, extracts, summarizes, compresses; up to 1M tokens, images and scanned PDFs tooTheir API~$0.04–0.05 per million input tokens, output free (per third-party price trackers)
Jev, by TypeSafe AIPicks an answer from your options, with a probability for eachTheir API$0.042 per million input tokens, output free
GLiNER2, by FastinoExtracts names, dates, fields; classifies textYour laptop, no GPU neededFree, open source (Apache 2.0)

To put the prices in perspective: classifying a 1,000-token support ticket (roughly 750 words) with Jev costs about $0.00004. A million tickets, about $42. TypeSafe puts general-purpose models at $0.20 to $10 per million input tokens, with output around five times more.

The speed is the other half. TypeSafe says Jev answers in 70 to 500 milliseconds. ScaleDown reports its models are 200 times faster and 467 times cheaper than frontier models on these narrow tasks. These numbers come from the companies themselves, so treat them as a promise to verify, not a fact. The early signals are good, though: according to TechCrunch, Vercel swapped Jev in for an OpenAI model on a classifier that checks commands for safety, and got results 5 to 18 times faster, with better accuracy.

What a call looks like

You send the text and your options, each with a one-line description. You get back the winner, a probability for every option, and a one-sentence reason. That’s it:

import requests

response = requests.post(
    "https://api.scaledown.xyz/classify",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "text": "Order A-1042 failed with ERR_PAYMENT_DECLINED. I already tried a different card.",
        "labels": [
            {"name": "billing", "rubric": "Is this about a charge, refund, or payment?"},
            {"name": "technical", "rubric": "Is this about a bug or something not working?"},
            {"name": "account", "rubric": "Is this about login, password, or account settings?"},
        ],
    },
)

print(response.json()["top_label"])  # billing

No long prompt, no parsing a paragraph to find the answer. The model can only say one of three words, and it tells you how sure it is.

Who’s building them

The three regions are playing the same game with very different strategies.

The US sells APIs, backed by venture money.

  • TypeSafe AI, the company behind Jev, came out of stealth in San Francisco with a $40 million seed round. Its CEO is a former OpenAI researcher.
  • Fastino, in Palo Alto, has raised about $25 million, and gives its GLiNER2 extraction models away as open source.
  • ScaleDown hasn’t disclosed its funding, but it has something money can’t easily buy: an official integration with LangChain, one of the most widely used frameworks for building agents.

Europe builds small labs and open source, on lean budgets.

  • Pleias, in Paris, makes models of 350 million and 1 billion parameters that do one thing: answer questions from your documents, quoting their sources.
  • Knowledgator, a small team mostly based in Ukraine, maintains GLiNER together with Fastino and builds GLiClass, its sibling for classification.
  • The Token Company, born in Helsinki and now in San Francisco through Y Combinator, makes a model that strips useless words from your prompts before they reach a large model.
  • Malted AI, in Scotland, distills large models into small ones for a single domain, like finance or law.

China leaves it to the giants, and gives the models away. I couldn’t find a Chinese startup selling task models through an API the way ScaleDown does. Instead, Alibaba’s Qwen team publishes free, open task models: Qwen3-Reranker-0.6B, which sorts documents by relevance, and Qwen3Guard, a 0.6-billion-parameter model that flags unsafe prompts and answers. On the generalist side, ModelBest, which puts MiniCPM on phones and in cars, reportedly raised 5 billion yuan in the first half of 2026, at a valuation above $2.8 billion.

Three strategies, one shared bet: most of the work AI does every day doesn’t need a giant.

So, are we there yet?

Yes. With a condition: you need to know exactly what you’re asking for.

Reach for a small model when:

  • the task is narrow and the possible answers are known in advance: a label, a field, a yes or no, a shorter version of a text;
  • it happens a lot, thousands of times a day, so fractions of a cent and milliseconds add up;
  • the data shouldn’t leave your machine, because a small model can run on your own laptop or server;
  • you want to check the answer, with a probability you can set a threshold on or a quote you can trace back to the source.

Stick with a large model when:

  • the job is open-ended: writing, planning, code;
  • it needs real reasoning or broad knowledge of the world;
  • it happens once in a while. If you sort ten emails a day, the savings won’t pay for the extra moving part.

A few caveats.

Most of the numbers come from the vendors. 200 times faster and 467 times cheaper are claims, not measurements. Run a test on your own data before you switch anything.

Compression can cut too much. Even the LangChain docs for ScaleDown warn against compressing short prompts or creative work, because it can drop instructions, tone or detail.

One more model is one more dependency. Another API, another bill, another thing that can go down. When Jev launched, demand briefly knocked out TypeSafe’s API.

The future NVIDIA’s researchers described isn’t small models replacing big ones. It’s a big model doing the thinking, surrounded by small ones doing the chores. LangChain sums it up nicely: cheap by default, frontier on exception.

So next time you catch an agent paying a frontier model to decide whether an email is about billing, remember: that’s a checkbox. Give it to something small.