Lessons from a 1.35B Korean Model Trained for $240: VRAM Pitfalls and the Betrayal of Evaluation Metrics

A project demonstrating training a 1.35B parameter model from scratch, primarily focused on Korean, using only personal GPUs has been released. The scarlet-alpha-1.3b-preview project completed approximately 22B tokens of pretraining for a total cost of $240, lowering technical barriers to entry while simultaneously revealing the clear hardware and data limitations faced by small-scale models ¹. This article summarizes essential insights and pitfalls for practitioners designing small Korean language models, ranging from lessons learned regarding precision selection due to VRAM constraints during training, the issue of data contamination inflating benchmark scores, to sentence termination errors caused by special token initialization failures.

The Reality of Hardware Constraints When Processing 22B Tokens on a $240 Budget

The first reality encountered when training a 1.35B parameter model on personal GPUs is the physical limitation of VRAM. The pretraining of scarlet-alpha-1.3b-preview was conducted at approximately a 22B token scale using four to eight RTX 5090 GPUs, with total costs amounting to about $240 ¹. Even for a seemingly small 1.35B model, actual VRAM usage exceeds 25GB when including optimizer states, making it impossible to load the model on a single RTX 5090 environment ¹. Consequently, model parallelism or multi-GPU distributed training becomes a mandatory prerequisite.

Precision selection can also yield results contrary to intuitive expectations. While FP4 precision, a recent technology, might be considered for memory savings, experimental results showed that although FP4 matrix multiplications were fast, the additional overhead attached to each layer caused the GPU to wait for the CPU, and its memory efficiency was lower than FP8, resulting in slower training speeds ¹. Therefore, FP8 mixed-precision training remains the realistic standard even for small models, and it should be noted that indiscriminate precision reduction can harm both performance and memory efficiency.

Network and Storage Failures Occurring with Cloud GPU Rentals

To supplement the limitations of personal environments, utilizing cloud GPU rental platforms such as vast.ai introduces significant variables due to the large deviation in host-to-host network bandwidth. When using vast.ai, three out of four hosts without data center certification failed to even start training due to network issues ¹.

Furthermore, even when training proceeds normally, unstable storage connections can cause checkpoint saving delays, leading to reported cases of losing parts of the accumulated training progress ¹. Securing GPU computation alone does not guarantee successful training; infrastructural management capabilities are required to pre-validate and block issues related to network bandwidth and storage stability used for training data loading and model saving paths.

Korean Data Contamination Inflating Benchmark Scores

The source of training data during the pretraining phase is a core factor determining the reliability of evaluation metrics. scarlet-alpha-1.3b-preview recorded significantly higher scores than gemma-3-1b-pt on various Korean benchmarks such as KoBEST COPA based on pretraining checkpoints; however, data contamination was involved in this process ¹.

When Wikipedia is used intensively as training data, the overlap with benchmark test sets increases sharply, resulting in inflated scores compared to actual literacy capabilities ¹. Specifically, for KoBEST BoolQ, 1,025 out of 1,404 questions overlapped significantly with the training data and were excluded from evaluation; when the overlapping questions were removed, the difference between the compared models disappeared ¹. Additionally, while computational problem accuracy rose from 101 (out of 224) to 170 after a short fine-tuning period, 96 out of 192 test sentences were already included in the training data, suggesting simple sentence memorization ¹. To avoid being misled by absolute benchmark numbers when developing Korean models, it is essential to pre-filter and refine the intersection between test sets and training corpora.

Missing Special Token Initialization in SFT During Conversion to Conversational Models

During the Supervised Fine-Tuning (SFT) stage intended to convert pretrained models into conversational ones, the handling of special tokens in conversation templates reflecting Korean morphological characteristics emerged as a significant pitfall. While training was conducted on pretraining data that did not include special tokens, entering the SFT process caused an issue where, although special tokens for dialogue were defined in the model's tokenizer, their corresponding vectors (Embeddings) were not properly initialized ¹.

If special token vectors remain untrained, the model exhibits an error where it fails to terminate sentences while generating responses. Experimental results showed that when SFT was performed without initialization, the model terminated answers correctly in 0 out of 50 cases. However, when those special token vectors were separately initialized properly and fed into the learning process, correct termination occurred in 49 out of 50 cases ¹. Due to the complex way tokenizers split tokens based on Korean sentence-ending particles and endings, checks for token vector consistency must be performed during the early stages of SFT.

The Light and Shadow of Directional Dependency and Reasoning Ability

Limitations regarding the small model's Korean generation ability and computational ability were also specifically confirmed. scarlet-alpha-1.3b-preview showed superiority over Gemma-3-1b-pt in Korean-specific benchmark scores but failed to show meaningful differences compared to models of the same size on sentence-based reasoning problems such as GSM8K-ko ¹. Based on GSM8K-ko accuracy with greedy decoding, it recorded 0.198, which was not different from the comparison model (gemma-3-1b-it) at 0.214, and showed slightly lower performance on IFEval-Ko (0.284 vs. 0.342) ¹.

The most notable phenomenon was the high variance in computational ability depending on the form for smaller models. The model showed high prediction probability for forward-direction facts like "The capital of A is B," with an average correct log-probability of -1.42. However, when asked in the reverse direction ("The country whose capital is B"), the average correct log-probability dropped to -3.23, resulting in a sixfold difference in correct answer probability ¹. This directional bias suggests that facts must be learned separately depending on their direction, pointing to it as a cause of errors during tracking intermediate computational steps or logical connections in sentence-based calculations.

Failure of Distillation Loss Functions and Tips for Output Control Tuning

Loss functions or tuning techniques introduced for additional performance improvement sometimes proved detrimental. When a distillation loss, which aligns outputs with those of an existing model (Gemma 4 E2B), was concurrently applied to pretraining to distill knowledge from the model, performance declined across all domains ¹. Consequently, the distillation loss function was excluded from the final training process.

On the other hand, the repetition phenomenon, where specific words are repeated as sentence generation becomes longer, was not easily resolved by data adjustment alone and required control via the Repetition Penalty parameter at the decoding stage ¹. While applying a repetition penalty resolved the issue, the exact cause of the repetition was not identified ¹. This serves as a case showing that adjusting parameters at the inference stage can be a more practical and faster solution than modifying the model's weights.