Developer Everett, who embedded the 26B MoE model into a local classification app that needs to run 24/7, ran into an unexpected obstacle. The model itself was smart enough, but its resident memory footprint increased the laptop's swap usage, threatening the SSD's lifespan. Trying to keep a large model resident on a small device for extended periods ultimately overwhelmed the storage subsystem. The need was clear: they required a model that does just one thing well.
Everett, the author of the Tolby app, found his answer in 'pruning.' This method significantly reduces the number of experts, the components inside the model that perform calculations, and restructures the remainder to create a single dense model. According to his report, this approach, combining pruning, healing, and pinning, achieved 92% agreement with the original on a single classification task while halving memory usage and doubling speed compared to a 12B model. However, the more critical point to examine in this piece is not the result but the process. The core of this experiment lies in why the intuitive assumption that retaining only the most frequently used experts would work failed completely, and what criteria should actually be used for selection.
MoE models are heavier than their compute components
To understand this experiment, we must first look at the structure of Gemma 4 26B-A4B. According to the official Ollama model page, this is an MoE model with 25.2B total parameters but only 3.8B active parameters used for actual computation. Mixture of Experts (MoE) splits the model's weights into multiple experts, and a selector called a router activates only 8 out of 128 experts for each token to perform the computation 1.
Here, intuition and reality diverge. If only 8 experts are used at a time, why must all 25.2B parameters be stored? Because the router switches experts for every token, the entire model must remain alive in memory so that any expert can be activated at any moment 2. In other words, MoE reduces computational load by decoupling memory usage from compute intensity. Everett's problem stemmed precisely from this separation of compute and memory. Even if computation was light, the sheer capacity required to load the entire model suffocated the small device.
So he thought in reverse. What if we fixed the 8 experts the router selects to just those 8 permanently? If only 8 experts are retained, the selector has only 8 candidates to choose from, effectively eliminating the router. This aligns the size of the model with the components used for computation, essentially converting the MoE into a dense model 3. The final result was a dense model with 3.81B parameters. The only issue was which 8 to keep.
The collapse of the "keep the frequently used" intuition
The first criterion that comes to mind is usage frequency. Looking at the expert usage visualization data Everett found, a heavy-tailed distribution emerged during the decoding phase, where most experts were rarely activated, with only a few flashing brightly 2. The speculation that retaining only the most frequently used components and discarding the rest seemed extremely reasonable. He attempted 'popularity-based pruning,' selecting only the top experts. The result was a complete collapse. Models like Claude warned of failure, but he assumed the selection was simply wrong and tried again 2.
To understand why it failed, one must distinguish between the two phases of how a model processes prompts. Prefill is the phase where the entire prompt is processed at once, while decode is the phase where the answer is generated token by token. Everett separated these two phases in his experiment for this very reason 2. Surprisingly, when experts were reduced to 16 or fewer, it was not the decode phase but the prefill phase that collapsed. The decode phase survived the pruning, but the prefill phase broke down beyond recovery 2.
This leads to a key lesson. In MoE, frequent usage does not equate to importance. Everett hypothesized that even if an expert is activated only once for a specific token, it might have a disproportionately large impact on the overall representation 2. According to interpretability research trends he referenced, boundary tokens—where the model makes crucial decisions while preparing the answer—are reported to be significant 2. If a rarely used expert fires at that exact moment and changes the outcome, frequency fails to reflect importance entirely. Ultimately, he shifted to selecting experts based on their contribution to accuracy, rather than frequency.
How to 'heal' at block boundaries
Even after deciding which experts to keep, the holes created by the cuts must be filled. Everett called this process 'healing.' Metaphorically, this stage is likened to a 'fire extinguisher,' suggesting that it calms damaged internal representations back to their original state, much like extinguishing a spreading fire.
During pruning, he observed an interesting phenomenon. Errors that grew as data passed through sliding window layers decreased when passing through the global attention layer at the end of a block 2. Because the global attention layer sees the entire context, it appeared to repair the prior damage. However, Everett added a caveat, noting he was "attaching plausible reasons rather than asserting this explanation is correct" 2. Regardless, he concluded that by aligning the output values of the global attention layer as closely as possible to the original model, the state could be restored at block boundaries, no matter how much the interior was rearranged.
The tool he used was pinning. Instead of merely getting the final answer correct at the model's end, this method trains the model to match the intermediate representation (hidden state) at the block end directly to the original model's representation 2. He summarized this difference in one sentence: "It's not about getting the answer right, but making it think the same thoughts" 3. This is the essence of block-local training. In fact, during the first healing run, the error rate on zero-shot prompts dropped from 0.69 to 0.42, a reduction of about 40% 2. However, Everett honestly recorded that it did not "perfectly match the original," noting it was only about half recovered.
Good at one thing, lost in others
We must separate the results and limitations of this process. On the classification task, the compressed model showed 92% agreement with the original 26B model. For reference, since the 12B model shows 90% agreement with the 26B on the same metric, the 3.81B compressed model demonstrated comparable or better consistency 32, although it fell short of Everett's original target of 98% 2. He reported that it was superior to the Gemma 4 E2B (approx. 5B), which is in a similar parameter class 2.
The initial worry about SSD lifespan due to swap increases was the actual driver of this experiment 2.
However, what was lost is also clear. Everett stated firmly that this model is not for general tasks and that knowledge-intensive tasks are significantly degraded. The explanation given is that the knowledge was pruned away 3. While block-local training aligned the reasoning process rather than just the answers, it did not become identical to the original 32. The efficiency of his proposed method is still being verified. He suggests that training can be done on user devices with about 5GB of memory using one's own data, but explicitly noted, "we have not yet tested this on actual user devices" 3. He also defines this experiment himself as an engineering side project rather than research 2.
Impact, not frequency, selects the experts
Ultimately, this experiment answers one question. Can you still do one thing well after shedding 85% of the MoE weights? The answer is "Yes, for one thing, but the criteria for what to keep determine success or failure." The intuitive method of keeping the most frequently used items among the 8 selected from 128 completely broke the model during the prefill stage 2, and the path to healing by aligning block boundaries with the original opened only after selecting experts that actually contributed to preserving the correct answer 2. The result was 92% agreement on one classification task, with half the memory usage and double the speed 3, but general knowledge was pruned away along with the rest 3.
The practical conclusions for the reader narrow down to two points. First, to reduce the gap between the number of active parameters selected by the router and the total model size in MoE lightweighting, one must fix the router and change the criteria for selecting experts from usage frequency to impact. Second, these 'fire extinguisher' models are, at best, attempts to transplant specific representations from the original into a small dense model; they are not small models that do everything well. The moment you set aside the assumption of keeping what is frequently used becomes the starting point for injecting one function of a large model into a small device.
