An annotation guideline is the instruction document that tells annotators what each label means and how to apply it. When annotators work in different languages, each team needs the same rules in a language it reads natively, or the labels start to drift. Inconsistent labels train inconsistent models.
I believe translating annotation guidelines belongs to annotation quality control, and it deserves the same review as the annotation itself. A guideline that means slightly different things in two languages produces the inconsistency a quality process exists to prevent.
In July 2026, Tomedes translated one assessment with a built-in annotation key into 20 locales, including Arabic for Egypt and Saudi Arabia, Hindi, Tamil, Portuguese for Brazil and Portugal, Japanese, and Korean. This post covers what gets translated, how the review works, and when the step is not needed.
An annotation guideline is the instruction document that tells annotators what each label means and how to apply it, usually with examples of correct and incorrect labels. It needs translating when annotators work in different languages, because every team has to apply the same rules in a language it reads natively. A labeling key is the list of labels and their definitions inside the guideline.
Data annotation adds labels a machine learning model can learn from, such as a transcript for a recording or a tag on a phrase. AWS groups data labeling into computer vision, natural language processing, and audio processing, and Tomedes works on the two language-led types, speech and text.
As far as I understand, many guideline problems in multilingual projects start with a rule that reads clearly in the source language and ambiguously in another.
Tomedes translates the annotation guidelines, the labeling key, and the examples for each project language, so every team follows the same rules. The annotation itself is done by native speakers of the exact locale. Tomedes works in 270+ languages, so each team can read the guideline in its own language.
| Item | What it contains | Why it is translated |
|---|---|---|
| Guidelines | The rules for applying each label | So annotators in every language follow the same rules |
| Labeling key | The list of labels and what each one means | So each label keeps one meaning across languages |
| Examples | Sample items with the correct label | So annotators see the rule applied in their own language |
I think the examples need the most care, because an example that works in the source language may make no sense in another. A glossary of label names helps keep fixed terms fixed, and our guide to building a multilingual glossary covers how to build one.
No. The test is whether annotators in each language make the same labeling decision on the same item, and a literal translation can fail that test. A guideline is a set of decisions written as sentences, so the translation has to carry the decision as well as the words.
Take a guideline that tells annotators to mark hesitations, unintelligible speech, and non-speech sounds, the label types Tomedes lists for verbatim transcription. The sounds people make when they hesitate, and the way each language writes them, differ by language, so the examples have to be rewritten for each locale. In my view, that adaptation is part of the translation job.
In July 2026 a new customer sent Tomedes an assessment with a built-in annotation key to be translated into 20 locales, each of about 1,000 words. Tomedes assigned dedicated linguists and project managers, because the translations would be used in testing and had to follow the same standard as the original assessment. The deadline was a few days after the order.
The locales included two versions of Arabic and two of Portuguese, so one language sometimes needed more than one locale. A typical multilingual data annotation project starts with scope, then a pilot batch that tests the guidelines, then production, review, and delivery.
My read is that two locales of one language is a good reminder that the locale, not the language, is the unit of work.
Native-speaker linguists of the exact locale do the work, a second reviewer checks it, and one dedicated project manager coordinates the batches and reports progress. In the 20-locale project, Tomedes assigned dedicated linguists and project managers because the translations would be used in testing.
Tomedes confirms native-speaker availability before a project starts, which matters when a model needs speakers of languages such as Tagalog, Amharic, or Yoruba. My preference is to confirm who will review each locale before the guideline is final, because a reviewer who reads the guideline early can flag unclear rules.
Match each task to a native speaker of the exact locale, add a second reviewer, and measure both against the written guideline. A pilot batch before full production shows where a guideline reads differently across languages. Tomedes is ISO 17100:2015 certified for its language services, and the standard includes an independent review step.
I would add one check of my own, which is to compare how annotators in different languages label the same pilot items and ask why they differ. Disagreement on a pilot item often points to a guideline that reads differently across languages.
Article 10 of the EU AI Act lists annotation and labelling among the data-preparation steps covered by data governance for high-risk AI systems. In my opinion, a translated guideline for each language is part of the documentation a reviewer would want to see, and this is not legal advice.
It is not needed when every annotator reads the source-language guideline natively, because a translation then adds cost and a second version to maintain. It is needed when annotators work in different languages, or when the guideline is long, technical, or full of examples that depend on the language.
I could be wrong on this, but a short guideline with few examples may be easier to keep in one language with a glossary of terms, and a long one is where translation pays off.
When every language team starts from the same rules, the labels that come back are easier to compare.
Planning annotation in several languages? Send the guidelines and a sample task when you request a quote from Tomedes.
Q: What is an annotation guideline?
A: An annotation guideline is the instruction document that tells annotators what each label means and how to apply it, usually with examples of correct and incorrect labels. A labeling key is the list of labels and their definitions inside it. Clear guidelines are how a team of annotators reaches consistent labels across thousands of items.
Q: Can annotation guidelines be translated?
A: Yes, and for multilingual teams they should be, because annotators need the rules in a language they read natively. Tomedes translates and adapts annotation guidelines, labeling keys, and examples for each target language so labels stay consistent across teams. Examples usually need adapting, and a literal translation can miss that.
Q: How do you keep annotation labels consistent across languages?
A: Give every language team the same written guideline in its own language, match each task to a native speaker of the exact locale, and add a second reviewer who checks completed work against the guideline. A pilot batch before full production shows where a guideline reads differently across languages.

Rachelle leads product and AI at Tomedes, where she runs the experiments that turn internal data into better translation experiences. She writes about what actually happens when you build AI products such as MachineTranslation.com — the numbers, the surprises, and the parts that don't go to plan.
Share:
Post your Comment