Why Rebuilding Our Ticket Classification AI Cut Costs from $22,200 to $500 Per Million Tickets

Amazon reduced the AI cost of its large-scale internal ticket classification system by approximately 98%. This was not achieved by choosing a less expensive large language model. Instead, the approach shifted from having the AI write a 'long explanation (composition)' for the same task to having it simply select the 'single correct answer (choice).'

Changing the output format reduced the average processing time from 4 seconds to under 1 second, and the cost per million items dropped from approximately $22,200 to$500. In this article, we will examine how dramatically costs and speed can improve simply by reducing output length, based on a case study shared by Amazon’s internal ticket system team (Carr·Mehul). ¹, ²

Why were 'long analytical texts' necessary for ticket classification?

Amazon’s internal ticket system handles large-scale classification tasks, categorizing customer escalations, server capacity requests, automated alerts, and security findings into one of 20 fixed categories. ²

Previously, a large language model (LLM) named 'Claude Sonnet 4.6' was responsible for this task. The AI generated complex JSON-formatted text that included not only the ticket analysis but also the reasoning behind the chosen label. On average, the amount of text (tokens) generated by the AI per ticket reached 170. ²

However, if you break down the essence of the classification task, it is simply 'selecting one option from a given list of 20.' In effect, excessive 'composition' was being performed for a task that did not require long-form writing. Large language models are typically billed proportionally to the number of tokens used for both input and output. Consequently, the longer the unnecessary explanations generated by the AI, the more rapidly costs increased. ²

A 'decision model' that removed the text generation mechanism

Amazon’s internal engineering team, which shared the system architecture via the strandsagents.com blog, replaced the system with a decision model called 'Strands Decider.' This model differs from standard generative AI in its underlying architecture. ², ³

General AI possesses a text generation mechanism called an 'LM head,' which creates sentences by selecting the next character. In contrast, Strands Decider removed this LM head from the trunk of a pretrained language model. ³

Instead, a small structure called a 'pointer head' was attached to score each candidate option. Rather than commanding the AI to write out an answer, the system was designed to simply ask, "Which option from A to T is most appropriate, and what is the probability?" and calculate only the probability and confidence score. Because the model cannot generate text, it cannot produce nonsensical answers outside the list, and the process of writing one character at a time is eliminated, resulting in significantly faster processing speeds. ³

The Amazon team initially used a 2B-scale model but transitioned to the 'E4B' model based on Gemma 4 after it demonstrated better performance in tests. ² For reference, the E4B model achieved 81.0% accuracy in evaluations across 231 public tasks on JevBench v1. ⁴

Compressing input data and controlling variables related to order

The change was not limited to the output format. Input data was also efficiently compressed to fit within the model’s processing limits.

The detailed descriptions of the 20 labels in the existing production environment already exceeded the decision model’s maximum processing capacity of 4,096 tokens. Therefore, they were rewritten as short 'criteria for judgment' one or two sentences long. Additionally, a 'none' option was added to handle tickets that did not fit precisely into any list. When ticket content was lengthy, the system truncated the oldest comments first to meet the length requirements. ²

The team also discovered that the order of data presentation altered the results. Even for conceptually identical ticket data, merely changing the alignment order of JSON keys read by the AI caused the probability assigned to the answer to fluctuate by up to 0.19. To resolve this, the Amazon team adopted a method of pre-sorting all request data keys in alphabetical order before transmission. ²

Cost calculations showing a 98% reduction by shifting from pay-as-you-go to a fixed-cost structure

These adjustments significantly reduced the cost incurred per ticket classification. The core reason for the drastic drop in costs lies in the 'change in billing structure.'

Source: Amazon internal ticket classification system comparison data ²

APIs for large language models like Claude Sonnet 4.6 operate on a 'pay-as-you-go' basis, charging $3 per million input tokens and$15 per million output tokens. Therefore, it is natural that costs decreased as output volume reduced and input data was compressed. ²

In contrast, the method for running Strands Decider E4B is a 'fixed-cost' structure, where cloud server instances (ml.g5.xlarge) are operated directly at an hourly rate. The hourly cost for this server instance is $1.408. ¹, ²

Since the processing time per item is only about 640 ms (0.64 seconds), a fully utilized single server can process approximately 5,600 tickets per hour. At this level, the cost per million items drops to around $250. However, in actual operations, surplus capacity must be maintained to handle traffic spikes, so the actual calculated cost per million items, including these factors, was approximately$500. ²

Shifting from a 'character-count-based pricing' model, calculated based on how long the AI writes, to a 'fixed-cost' system proportional to server uptime is expected to lead to large-scale cost reductions potentially reaching millions of dollars annually. ¹, ²

Validation process using redundancy in actual traffic

Applying a new model suddenly to a running system, even with good performance metrics, can cause unexpected errors. To transition the system safely, the Amazon team introduced a redundancy structure called 'DUAL mode.' ²

In DUAL mode, actual customer data is sent to both the existing Sonnet model and the new Decider model simultaneously. However, the classification result ultimately applied to the system remained that of the existing Sonnet model. Differences in results between the two models were recorded separately for precise comparison and analysis of classification discrepancies. The Amazon engineering team noted that this process provided more operational insights than offline evaluations. They also implemented a safety net allowing immediate rollback to the existing system within 45 seconds via configuration changes if issues arose with the new model. ¹, ²

Meanwhile, after adjusting confidence thresholds, evaluation of both models on a separate set of actual tickets showed that the Decider model delivered better classification results than the Sonnet model in operation. ¹, ²

What tasks can benefit from this approach?

Replacing systems with decision models based on this approach offers clear efficiency under specific conditions. According to data shared by the Amazon team, this method is particularly effective in environments involving 'closed-end tasks that involve selecting one option from a predefined list,' 'huge throughput volumes,' and 'latency-sensitive operations.' ¹, ²

Conversely, due to the nature of a model that does not generate text, it is unsuitable for generating detailed analytical sentences. It may be difficult to replace large language models for tasks where long explanatory texts for classification labels are essential, where lists change weekly, or where useful context required for classification exceeds the 4,096-token processing limit. ¹, ²

For closed-end tasks that require only selecting one option from a defined list without needing complex reasoning or long-form writing from the AI, combined with high throughput and significant latency concerns in existing paths, reducing the burden of long-text generation creates substantial advantages in both cost and speed. ¹, ²