The AI Trend I'm Watching

Distillation maybe controversial because of politics, but it is a beacon for the future of AI

The copper stills of the GlenDronach Whisky Distillery near Huntly, Aberdeenshire.

New frontier models like Claude Mythos and GPT 5.6 Sol get all the attention, even driving some pundit enthusiasts to say they should be used for all enterprise applications. I think all of us who understand the incredible inefficiencies and flaws that come with LLMs are more impressed with distilled models like Alibaba's Qwen 3.6-35B.

Distillation, training a smaller "student" model on a larger "teacher" model's outputs, has been standard AI engineering since Geoffrey Hinton and his colleagues first published the technique in 2015. Alibaba's Qwen 3.6-35B is a good current example. It's a mixture-of-experts model with about 35 billion total parameters but only around 3 billion active per token, distilled down from Alibaba's much larger Qwen 3.6 flagship.

Qwen 3.6-35B benchmarks close to that flagship on coding tasks while running on a single high-end consumer GPU or a well-configured Mac. That's the whole promise of distillation in one model: Most of the capability, a fraction of the footprint, and something you can actually run yourself instead of renting by the token.

There is an industry-wide shift toward smaller, cheaper, task-specific models that sit beneath frontier models in production systems. The reasons why should be obvious. Frontier models are costly and error-prone. From the high cost of tokens to the wildly inefficient results of using a model literally trained on every dataset parent companies find on the web, businesses want better results.

In short, a more powerful frontier model is still an LLM with all of the financial, accuracy, environmental, and political challenges. A distilled student model is trained on specific pieces of data and runs on roughly 1/10th the GPU-hours that equivalent tasks cost with an LLM.

Unfortunately, the AI is still trained by the LLM, so there are still accuracy and logic challenges. But it costs less, is smaller, and is trained on domain-specific data only. Google’s head of AI Jeff Dean said the company adopted distillation years ago specifically to shrink image-recognition models without retraining from scratch, framing it as a mainstream efficiency technique rather than anything novel.

Smaller and More Efficient Isn’t As Sexy, But…

We are likely approaching of a massive AI hype wave that will finally “crest” with the Anthropic and OpenAI IPOs later this year. As the financial investment zeal wanes and actual performance becomes most important, AI companies will find themselves in the crosshairs.

Adoption will continue, but more and more pressure will be applied to software companies for better solutions than the current LLM approach of resolving problems and generating content. There are two primary areas where this points to new or more efficient derivative models.

The most obvious one is cost. Distilled models help quite a bit here. Regardless of how this evolves, running frontier models on everything is unlikely. They are widely inefficient, especially given the use cases tend to be niche-oriented, and the costs are just too high.

The second is the bigger issue of quality. Even a derivative model suffers from being trained on a parent frontier model. Hallucinations and low-quality training data (sorry, Reddit, X, and content creators across the Internet) continue to plague LLMs almost four years after ChatGPT took the industry by storm.

Sooner or later, the economics win out. Technology companies and enterprises will realize that using a frontier model is equivalent to buying their supplies from a department store. The department store offers everything and does none of it as well as a niche competitor. At some point, customers will seek out cheaper, higher-quality suppliers that outperform the department store in every way.

Longer term, while LLM company CEOs would like you to think we are at the singularity, the cusp of artificial general intelligence, the model type isn’t fitting the moment. Even distilled models, while a good step in the right direction, will not be enough to overcome the flaws of probabilistic decision-making.

I expect we’ll see increasing focus on designing AI models that solve specific industry or even enterprise problems. Smaller, faster, and more cost-effective customization will make it easier in this new vibecoding world.

Like most things in the AI era, I suspect we will see it in the form of a new solution somehow, some way, and just be amazed at how good it is. Yes, we will have to endure more LLM hype and attempts to fuel the craze. But if we see it as a distraction — noise, if you would — and focus on where efficiencies and quality improvements can be made, then the path forward likely features smaller, less costly AI.

P.S. Em dashes inserted by the human here.