The BLEU score (Bilingual Evaluation Understudy) is an automatic metric that rates machine translation by comparing it with one or more human reference translations. It counts matching word sequences, applies a penalty for outputs that are too short, and returns a number from 0 to 100. A higher BLEU score means the output is closer to the reference, but it does not prove that the translation is correct.
This guide explains how the BLEU score works, how to read it, where it fails, and which metrics to use next to it when you evaluate machine translation for real localization projects.
What Is the BLEU Score?
BLEU was introduced in 2002 by IBM researchers in the paper BLEU: a Method for Automatic Evaluation of Machine Translation. The idea was simple. Human evaluation is accurate but slow and expensive, so researchers needed a fast, repeatable number that could be recalculated every time an engine changed. More than twenty years later, BLEU is still the most quoted score in machine translation papers, vendor benchmarks and engine comparisons.
For localization teams, the BLEU score answers a narrow question: how similar is this machine output to a translation a human already approved? It is a useful signal for comparing engines or tracking whether a custom model improved. It is not a quality guarantee for a single sentence going to a customer.
How the BLEU Score Is Calculated
The calculation has three parts. First, BLEU measures modified n-gram precision. It checks how many one-word, two-word, three-word and four-word sequences in the machine output also appear in the reference, and each sequence is only counted as many times as it occurs in the reference. Second, it combines the four precision values with a geometric mean. Third, it multiplies the result by a brevity penalty, which lowers the score when the output is shorter than the reference, so a system cannot win by translating only the easy words.
Here is how three candidate translations compare with the reference sentence "The contract must be signed by both parties."
| Candidate | Output | What happens to the BLEU score |
|---|---|---|
| A | The contract must be signed by both parties. | Exact match with the reference, so the score is at its maximum. |
| B | The agreement has to be signed by both sides. | Meaning is correct, but synonyms break many n-grams, so the score drops sharply. |
| C | Contract signed both parties. | Missing words reduce precision and the brevity penalty lowers the score again. |
Candidate B shows the central weakness of the metric. A professional translator could accept it, yet BLEU treats it as a poor match because it only sees surface words.
How to Read BLEU Scores
There is no universal threshold, because scores depend on the language pair, the domain and the number of references. As a rough guide, the Google Cloud documentation on BLEU interprets the range like this:
| BLEU score | Rough interpretation |
|---|---|
| Below 10 | Almost useless |
| 10 to 19 | Hard to get the gist |
| 20 to 29 | The gist is clear, but with significant grammar errors |
| 30 to 40 | Understandable to good translations |
| 40 to 50 | High quality translations |
| 50 to 60 | Very high quality, adequate and fluent |
| Above 60 | Often better than human translation |
Treat these bands as direction, not as a verdict. A score of 45 on legal text with a strict glossary and a score of 45 on marketing copy are not the same quality level.
Limits of the BLEU Score
- It ignores meaning. Synonyms, paraphrases and correct reordering are punished.
- It depends on the reference. A weak reference gives a misleading score, and several references usually give a fairer one.
- Tokenization changes the number. Two tools can report different scores for the same output, so use a standard implementation such as SacreBLEU and always report its settings.
- It struggles with some languages. Languages without spaces, such as Chinese and Japanese, and morphologically rich languages need special handling.
- It is weak at sentence level. BLEU is designed for large test sets, not for judging one segment.
BLEU vs Other Translation Quality Metrics
| Metric | What it measures | Best used for |
|---|---|---|
| BLEU | Word n-gram overlap with a reference | Quick engine comparison on a large test set |
| chrF | Character n-gram overlap | Morphologically rich languages |
| TER | Edits needed to reach the reference | Estimating post-editing effort |
| METEOR | Matches including stems and synonyms | Closer agreement with human judgment than BLEU |
| COMET | Neural score trained on human ratings | Modern engine evaluation, including reference-free variants |
| Human LQA / MQM | Error types and severity marked by linguists | Final quality sign-off |
Using BLEU in a Real Localization Workflow
In practice, run BLEU on a representative sample of approved translations from your own domain, not on a public benchmark. Compare two or three engines, keep the one that wins on your content, then confirm with a human review. If you plan to use raw output at scale, combine the score with machine translation post-editing and a documented translation quality assurance checklist. Our guide on how to choose a machine translation tool explains the other criteria that matter, such as data privacy and glossary support.
The best engine often changes by language pair and content type, which is why AI machine translation with several engines in one workspace gives better results than committing to a single vendor.
Frequently Asked Questions
What is a good BLEU score?
For most language pairs, a score above 30 usually means understandable output and above 40 means high quality, but the number is only comparable on the same test set, language pair and tokenization.
Can BLEU evaluate LLM translations?
Yes, but LLMs often paraphrase, which lowers BLEU even when the meaning is right. Pair BLEU with COMET and a human spot check.
Is a higher BLEU score always better?
No. A higher score only means closer similarity to the reference. A fluent translation that differs from the reference can be correct and still score low.
Compare Engines Segment by Segment With WordBeam
Instead of trusting one number, WordBeam lets you run GPT, Claude, Grok and DeepL side by side, pick the best result per segment, and catch errors with real-time QA. See the AI translation engines in action, or start a free trial with 14 days and 10,000 words included.