How the Morning Menu Recommendation Works Using Diabetes Conversation Memory

A user had a total of 107 conversations with an AI assistant, building a history that amounted to approximately 32,000 tokens. Among these conversations, the user asked only once, "Does diabetes on the mother’s side increase the risk more than diabetes on the father’s side?" The user never explicitly stated that they have diabetes themselves. However, much later, when the user asked, "What should I eat for breakfast?", a standard language model without memory functionality provides a generic breakfast list unrelated to the context of diabetes. In contrast, Memorilla, released by researchers from Stanford, MIT, University of Pennsylvania, and the University of Toronto, generates an answer to that question by connecting blood-sugar-friendly breakfast options with the mention of diabetes in past conversations.

Memorilla is a memory system for Large Language Models (LLMs) that compresses necessary information into 16 'memory tokens' (vectors) by linking the question with the relevant information, rather than forcing the model to read entire documents or conversation histories in full. These 16 vectors are not shortened summaries of specific sentences but results of the learning module extracting semantic relationships of information scattered across multiple documents using cross-attention, based on the question.

The Decisive Difference Between RAG and Memorilla

The most representative method for AI assistants to find clues in history from the past has been RAG (Retrieval-Augmented Generation). RAG selects a few documents with the highest word similarity to the current question and passes them directly to the LLM in their original form. However, the question "breakfast" and the past conversation about "diabetes on the mother’s side" share absolutely no overlapping words on the surface. According to examples released by the research team, when asked about 'energy-rich foods,' RAG stops at retrieving sentences containing 'brown rice,' 'quinoa,' or 'oats' based on word similarity. Core context from past conversations, such as 'iron deficiency' or 'half-marathon training,' is missed because there is no word overlap.

Memorilla changes this structure. A separate learning module developed by the researchers, containing approximately 194 million parameters, simultaneously reads the embedding (a numerical representation of text) of the current question and the embeddings of the entire stored conversation history. After analyzing information scattered across multiple documents using cross-attention (a technique that connects related parts), it compresses core information into 16 vectors and injects them directly into the language model's input space ¹. This is a method that reads the meaning of the entire repository aligned with the intent of the question, rather than relying on word-similarity search.

The Principle of 'Question-Dependent' Reading, Not Summarization

The point where Memorilla differs from existing AI summarization technologies is the 'timing of compression.' Traditional methods that summarize immediately upon reading a document do not consider what questions will come later. Because a single summary must be applied to all questions, the relevance to specific questions may diminish.

In contrast, Memorilla does not compress until a question arrives. First, it converts the document history into numerical representations once using the 'Qwen3-Embedding-4B' encoder and stores them. Only when an actual question arrives do the 16 learned memory slots scan the entire repository to extract the context, such as 'to answer this question, connect this information with that information' ¹. If the question changes, the content of the compressed information changes as well.

In this process, the LLM writing the answer (Qwen3-8B) and the encoder remain frozen (unchanged) during training. Rather than training the language model itself, only the 194M-sized intermediate module, which shifts document embeddings into the LLM's input space, is trained ²³. This effectively places 'vectors concentrated with meaning' directly before the LLM's input embeddings, instead of words (tokens). Due to these structural characteristics, Memorilla cannot be used via standard API methods that only exchange text and requires control over the LLM's internals to be applied ³.

Quantitative Performance and Speed Achieved by 16 Vectors

Fixing the number of memory tokens at 16 offers significant efficiency in computational load and processing speed. According to data from the research team, Memorilla maintained a 'constant latency' of 0.74 seconds for response generation, even as the document repository grew from 1 to 512 documents (approximately 128K tokens). Compared to the method of inputting the entire document into the LLM, where latency increased to 19.9 seconds as documents accumulated, this is approximately 27 times faster ¹².

It is not just about speed. For complex questions requiring answers to be synthesized from multiple facts or clues, performance sometimes appears higher than reading the entire document. On the FactKG benchmark, which validates multiple factual knowledge graphs, Memorilla recorded an accuracy of 84.33%. This accuracy was higher than the method of reading the full context (80.43%), while the number of input document tokens was merely 1/108th ⁴¹. Given that the full context lagged behind despite reading all facts, Memorilla's superiority does not stem from simple information volume.

The difference is particularly pronounced when four or more complex facts must be combined for a single correct answer. The RAG method, which retrieves 50 documents, saw its performance drop sharply from 78.7% to 51.8%. Conversely, Memorilla maintained accuracy in the 80% range because it compresses the relationships of facts scattered across multiple documents all at once ¹³. On the PersonaMem-v2 benchmark, which utilizes personalized conversational data, Memorilla achieved 44.6% accuracy, significantly outperforming the strongest RAG method, xRAG (30.9%), and the full-context method (27.9%) ¹.

Why the Question 'Breakfast Menu' Connects to 'Diabetes'

The operational mechanism of Memorilla can be clearly confirmed through the failure cases mentioned earlier. The clue 'risk of diabetes on the mother’s side' in the user history and the question 'what should I eat for breakfast?' have no word overlap.

A standard LLM without memory provides a generic breakfast list unrelated to the diabetes context when asked, "What should I eat for breakfast?" Conversely, Memorilla identifies weak semantic connections between the question ("Question: Breakfast") and the repository ("Repository: Past mention of family history of diabetes") using cross-attention, converts them into 16 vectors, and places them before the LLM's input embeddings. As a result, the LLM does not produce a simple breakfast menu list but generates a personalized answer, such as scrambled eggs or Greek yogurt, which considers both blood sugar control and satiety based on the family history of diabetes that appeared only once in past conversations ².

Where RAG Remains Strong: 'Exact Names' and Detailed Information

Memorilla's meaning compression technology is powerful for complex relational inference but is not perfect in all areas. Because only compressed vectors, rather than original text, are injected, accurate strings written in the document or fine-grained details may be lost. The research team estimates that compression will preserve the structure and meaning of documents but explicitly acknowledges the limitation that accurate names or specific detailed facts may be lost ¹³.

These limitations are confirmed by figures. On the PubMedQA benchmark, which requires finding clear specific details in short medical paper abstracts, Memorilla recorded 59%, performing lower than LLMLingua-2 (69%), which compresses or retrieves original text ¹³. On the NarrativeQA benchmark, which involves asking questions after reading long novels, results varied by type. For questions asking 'why' and 'how,' Memorilla showed ROUGE-L scores 5–7 percentage points higher than RAG, but for 'who' and 'which' questions asking for exact names or targets, RAG, which retrieves original text directly, scored about 3 percentage points higher ¹³.

Memorilla's performance figures were measured in a benchmark environment based on a specific combination prepared by the research team (a single Qwen3-8B model) and their own training split ³. Since independent reproduction validation has not yet occurred, it is difficult to definitively regard these as universal benchmarks that apply identically to other models or general text API environments.

The reason Memorilla could retrieve the old mention of diabetes is that it scanned the entire document based on the question and reassembled the necessary causal relationships into 16 vectors. This is an attempt to expand AI memory from simple text accumulation or summarization to 'question-dependent semantic unit compression.' However, when accurate names of specific individuals or short specific details need to be asked, search methods that locate the original text remain powerful, and combining both approaches appears to be the future direction ³.