Booking.com and AWS Win ACL Best Paper for Running a 70B Model 20 Times Faster

A joint team from Amazon Web Services and Booking.com has won the Best Paper Award at the ACL 2025 Industry Track for a framework that compresses a 70-billion-parameter language model into a 1.1-billion-parameter system running 10 to 20 times faster while recovering more than 90 percent of the original model's quality. The paper, "Speed Without Sacrifice: Fine-Tuning Language Models with Medusa and Knowledge Distillation in Travel Applications," was selected from 421 competing submissions evaluated by 453 reviewers. Amazon Science published the full production case study on October 29, describing A/B-tested results across three live Booking.com features. The decisive contribution is not that each technique works individually — both knowledge distillation and Medusa speculative decoding were established before this work — but that stacking them produces compounding, multiplicative speedup validated on real consumer traffic.

How Knowledge Distillation Closes the Gap Between a 70B Teacher and a 1.1B Student

Knowledge distillation, introduced by Hinton et al. in 2015, trains a smaller "student" model on the probability distributions a larger "teacher" model produces over every token, rather than on hard correct-answer labels alone. Those distributions encode the teacher's uncertainty across the full vocabulary — which alternatives were plausible and which were far off — providing a richer training signal than direct supervision.

The teacher is Llama-3.1-70B; the student is TinyLlama 1.1B, an Apache 2.0-licensed decoder-only Transformer built on the Llama 2 architecture and pretrained on approximately one trillion tokens. Their shared tokenizer lets the teacher's token-level outputs map directly onto the student's vocabulary. The paper demonstrates the method through ablation: TinyLlama fine-tuned on Booking.com data alone beats few-shot GPT-4o on travel tasks but falls short of the 70B teacher; adding knowledge distillation over unlabeled travel-domain text closes more than 90 percent of that gap while delivering roughly six times the raw throughput of the teacher. The full paper provides the complete ablation results.

Medusa Speculative Decoding Adds a Further 2 to 3.6 Times on Top

Language model generation is inherently sequential — each token is predicted one at a time. Medusa breaks that bottleneck by attaching multiple lightweight feed-forward "heads" to the base model's last hidden state, each predicting a different future token position in parallel. A single forward pass with tree-structured attention verifies all predicted continuations simultaneously and commits the longest valid prefix, eliminating the need for a separate draft model.

Booking.com chose Medusa-2, which trains the prediction heads jointly with the backbone in a single stage rather than freezing backbone weights (Medusa-1), achieving higher throughput with a simpler training pipeline. The measured additional speedup — 2 to 3.6 times on top of the compressed student — rises with domain constraint: travel descriptions and trip-planning responses use predictable phrasing, raising the fraction of predicted tokens that prove correct. Open-ended or highly creative tasks would see lower acceptance rates and smaller multipliers. Combined with the roughly six-times gain from knowledge distillation, the two methods compound multiplicatively to reach the reported 10-to-20-times total speedup against the 70B baseline.

Three Booking.com Features Validated in Live A/B Tests at Less Than One-Third of GPT-4 Cost

The paper reports A/B outcomes on real user traffic rather than offline metrics alone. Smart Filters, a natural-language property-search feature, saw 15-times P99 latency improvement. AI Trip Planner showed a 2.9-percent engagement lift versus a GPT-3.5 baseline. Accommodation description generation delivered a 1.4-percentage-point improvement in "helpful" ratings alongside a cost reduction to less than one-third of GPT-4 pricing at equivalent quality. The Amazon Science blog provides full details on all three production deployments. At Booking.com's query volumes across its global user base, the unit-economics gap between frontier API calls and a self-hosted 1.1B model compounds significantly — and routing queries through an owned model rather than a third-party API removes an external data dependency with implications for GDPR compliance in Booking.com's European markets.

Limitations and Outstanding Reproducibility

The pipeline's gains are domain-specific. Medusa's throughput multiplier depends on high token acceptance, which depends on predictable output phrasing. The paper does not characterize where the residual quality loss — the roughly ten percent not recovered by distillation — is concentrated, nor does it report hallucination rates. No standard academic benchmarks appear, so general capability comparisons with other published systems are not possible. The training recipe is not released as open-source code, requiring practitioners to reconstruct the pipeline from the paper and upstream repositories.

A Production-Validated Template for Replacing Frontier API Costs

The ACL Best Paper designation from 421 competing submissions establishes the stacked distill-then-decode pipeline as a peer-validated engineering result, not merely a plausible combination of known techniques. Enterprise teams paying frontier API rates for constrained-domain NLP — automated content generation, structured classification, domain-specific search — now have the first A/B-validated evidence that a compressed, speculative-decoding-enhanced self-hosted model can meet or beat frontier quality at a fraction of the cost. The open question is how broadly the approach generalizes: how many production domains offer the vocabulary constraint that makes Medusa's acceptance rate high enough to justify the training investment, and whether the template holds as base models and distillation targets continue to evolve.