There are situations in which the only difference between “unable to run” and “usable” is file size, despite identical computing environments. Project llama-modchip v2.4.0, published by GitHub user synchopate, concretely demonstrated this boundary. When the DeepSeek-V4-Flash model, with a total of 284.33B (approximately 284.3 billion) parameters, is loaded in its standard quantized file (76.87GiB) on a Linux machine equipped with an AMD Ryzen 9 9955HX and 64GB RAM, the operating system throws an Out of Memory (OOM) error and terminates the process.

However, this project provided a file that reassembled the weights of the same model at 56.56GiB. Running this file with the dedicated binary keeps resident memory (VmRSS) at around 57.61~58.9GiB and generates 9~15 tokens per second (the basic unit for AI sentence processing, corresponding to a word or part of a word) without NVMe swap12. A difference of about 20GiB resolved the issue of loadability. Below, based on figures released by the project, we explain how this technology surpassed the physical limit of 64GB RAM and the level of performance felt by general users.

Physical Basis for How MoE Structure Surpassed the Wall of 64GB RAM

When running large language models (LLMs) on a CPU, the bottleneck is not computation speed but memory bandwidth. It takes much longer to retrieve data from RAM than for the CPU to compute it. llama-modchip’s technical documentation (ARCHITECTURE.md) explains this phenomenon with concrete figures.

If a typical 'dense' 30B model is loaded with Q4_0 (4-bit class) quantization, it must read about 17GB of total weights from RAM for each generated token. Since the effective read speed of dual-channel DDR5 is around 75GB per second, 75 (bandwidth) ÷ 17 (data volume) = 4.4 tokens per second is the physical upper limit. It is difficult to overcome this limit regardless of CPU performance3.

This is where the Mixture of Experts (MoE) structure provides the answer. DeepSeek-V4-Flash is an MoE model consisting of 43 layers; when processing a single token, only 6 experts (routing experts) and 1 expert (shared expert) are activated out of the total. The parameters actually read are only 13.17B (4.6% of the total)4. If the data volume to be read is reduced, the bandwidth bottleneck is alleviated accordingly. For the Nemotron-3.5 model, which has 3B active parameters, only about 1.8GB needs to be read per token, making theoretically up to 41.6 tokens per second possible3.

However, 'reading less' is different from 'storing less.' Even if the amount read is reduced, loading the file into RAM requires the entire storage capacity at that moment. For a 76.87GiB file, the OOM killer terminates the process before the data can be properly read. Therefore, llama-modchip had to solve two problems simultaneously: ① compressing the saved weight blocks themselves to under 64GB, and ② maximizing the speed at which the CPU reads data.

Physical Basis for How MoE Structure Surpassed the Wall of 64GB RAM

Compression Math that Reduced Weight Size by 20GiB

To reduce the file size from 76.87GiB to 56.56GiB, llama-modchip combined two proprietary conversion techniques12.

1. Hessian-based Compression (SwiGLU-H2) MoE expert weights are compressed using an 'analytical 2nd-order Hessian' method. The Hessian matrix represents 2nd-derivative information indicating how sensitive the model is to input changes. According to project documentation, analyzing the Attention mechanism matrix wo_b × wo_a using Singular Value Decomposition (SVD) revealed that cutting 25% of the total rank maintained 99.82% cosine similarity with the original output direction4. This secured computational evidence that the discarded values have almost no effect on the actual output, allowing the selection of which components could be discarded, leading to compression of MoE expert weights to IQ1_S level.

2. Tensor Format Conversion (Fractal-64) Multi-head Latent Attention (MLA) tensors are converted to the 'Gray-code-based Q2_0' format (averaging 3.75 bits per weight, bpw)124. This is a dedicated format using a block size of 64 and BF16 scales. These two processes result in a total reduction of 20.31GiB, bringing the final file size down to under 64GB and enabling loading.

A 4-stage tiling technique utilizing a 4-level memory hierarchy including AVX-512BW ZMM registers (ZMM registers → L1D 32KB → L2 1MB → L3 64MB) was also applied to ensure compressed data is processed quickly within the CPU. This optimization improved prefill (input processing) speed for processing 512 tokens at once from 25.0 tokens/second to 44.36 tokens/second, an approximately 77% improvement12.

Response Speed Perception: How Much is 9 Tokens per Second?

Response Speed Perception: How Much is 9 Tokens per Second?

Converting the numbers to actual user experience:

  • **Top-5 Pruning Mode: Pure 9.68~9.87 tokens (**reaching 10.15 tokens when combined with Gate-First SwiGLU) Applying Top-5 L₂ pruning to the third and subsequent layers increases the perplexity (PPL), a simulated quality evaluation metric, by 0.42%, but the top-1 token match rate is high at 95.8%124.

For reference, the Nemotron model with 3B active parameters was recorded to have a base generation speed of about 25 tokens per second on the same CPU, averaging 48 tokens per second (maximum 60) with speculative decoding applied, enabling practical real-time conversation1. DeepSeek-V4-Flash-class models are the result of a compromise between 'the intelligence of large models' and 'the limits of 64GB RAM.'

Limitations: Requirement for Dedicated Binary and Quality Debate

Limitations: Requirement for Dedicated Binary and Quality Debate

Using this technology requires considering several limitations.

  1. Compatibility Issue: Fractal-64 uses a dedicated tensor format (Type ID 40). Therefore, unmodified standard executors (llama.cpp, Ollama, LM Studio, etc.) will produce an unknown tensor type 40 error and fail to open the file. It must be run with the single binary provided by llama-modchip (Linux x86_64)12.
  2. Quality Debate: Actual quality regarding compression rates (3.75bpw) is debated in the community. Some point out that performance drops sharply with quantization below 4 bits (according to certain papers), while other users report, "2-bit DeepSeek was practical enough in my environment"5. It is necessary to distinguish that the KL divergence (distribution similarity) and cosine similarity figures released by the project indicate 'mathematical proximity to the original' and do not guarantee perceived conversation quality for users45.

Conclusion: The Result of Combining MoE Structure and Compression Technology

The case of llama-modchip demonstrates that a specific pathway exists for running 284B-class large models on 64GB RAM. The core lies in the calculation combining **reduction in read volume per token due to MoE structure (**13.17B, 4.6% of total) and compression of storage capacity (76.87GiB → 56.56GiB).

Only when these two conditions (reduction in read volume per token and compression of storage capacity) are met are 'loading success' and '9~15 tokens per second response' possible simultaneously. However, this technology requires a dedicated executable file (unknown tensor type 40 error in unpatched llama.cpp, Ollama, LM Studio)12, and quality depends on settings. The default Top-6 Lossless mode records 9.06~9.95 tokens/sec (11.5~14.7 tokens/sec with PVS applied) while maintaining KL divergence of 0 and '0.00% Bit-Exact' compared to the original file12, while the Top-5 pruning mode increases perplexity by about 0.42%, and the Top-4 mode, which reduces experts to 4, shows a larger increase of about 2.85%14. If you are in a situation where you must run a giant AI without a graphics card, the combination of MoE models and optimization tools can be a practical alternative.

References

  1. ↩
  2. ↩
  3. ↩
  4. ↩
  5. ↩