The government community is realizing the importance of incorporating technology into the translation workflow for mission areas that deal with foreign languages. In this era of big data, human linguists can no longer handle the workload alone. Therefore, there is growing interest in how to incorporate the best attributes of human translators with those of translation technologies into workflows that make sense for government entities. This session will discuss a variety of translation technologies (translator skill assessment, computer-assisted translation, machine translation, and translation management) and how they are used in government environments today. Specific topics to be discussed will include: an introduction to government language standards and policies, an innovative use of technology for translation testing, differences between the commercial and government settings, and a case study discussing the operational use of translation technology.

1. Translation Testing at the
Federal Bureau of
Investigation
Amanda Curry
Language Testing and Assessment Unit
Federal Bureau of Investigation
Oct. 6, 2015

2. Interagency Language
Roundtable (ILR)
 Organization of representatives from
different government agencies, academia,
and industry
 Goals
 Standardization of assessment criteria
 Collaboration of training, testing, and
translation methodology between agencies
 Dissemination of new language information
and practices from both inside and outside the
government community

3. ILR Skill Level Descriptions (SLDs)
 Language tasks in a skill are described and
categorized by levels of progression
 Ordinal scale ranges from 0 to 5, with plus levels
 Skills measured:
 Proficiency: Listening, Reading, Speaking, Writing
(1985)
 Performance: Translation (2005), Interpretation
(2007), Audio Translation (2011)
 Competence: Intercultural Communication (2011)
 Adopted as standard for all US Government
agencies
 Located at: www.govtilr.org

4. ILR-Based Tests
 SLDs prescribe tasks appropriate for each
level.
 ILR-based tests present tasks as prompts
for examinee production.
 An ILR score on the 0 to 5 ILR scale is
assigned according to whether examinee
has handled a task successfully and
appropriately at a given level.

5. ILR Skill Level Descriptions
for Translation Performance
 At each level, the ILR SLDs for Translation
Performance refer to elements that
contribute to the difficulty of the
translation task.
 The ILR SLDs for Translation Performance
provide the basis for ranking texts
according to their level of difficulty for
translation purposes.

6. Text Level Difficulty for
Translation
 Translation text difficulty is not defined in
terms of obscure vocabulary.
 Difficulty is determined instead by the
extent of congruity judgment required to
choose equivalent expressions in the TL
that fully convey and best match the
meaning intended in the SL.

7. Translation Text Level Difficulty
LEVEL 2
LEVEL 3
LEVEL 4
Limited
Performance
Professional
Performance
Advanced Professional
Performance

8. Level 2 Characteristics
 CONTENT: straightforward narration,
description, directions, instructions, simple
explanation
 PURPOSE: to convey concrete information
 LEXICON: everyday usage; a minimum of
words with different meanings; technical terms,
if any, that do not demand special knowledge or
expertise
 STRUCTURES: basic, high-frequency complex,
with few modifiers, without unusual stylistic
devices

9. Sample Level 2 Text
Evidence containing blood should be completely
dried before it is packaged and shipped to the
laboratory for analysis. To avoid direct contact
and exposure to these materials in the courtroom,
evidence should be placed in a sealed, transparent
package and labeled appropriately.

10. Level 2+ Characteristics
 CONTENT: more than direct exposition,
including judgments and opinion, but no
complicated concepts
 PURPOSE: to convey information
 LEXICON: no figurative language, but
some terms may reflect source language
conventions
 STRUCTURES: adjectives and adverbs,
some embedding, but syntax is generally
uncomplicated

11. Sample Level 2+ Text
The National Center for Missing and
Exploited Children is a new hi-tech
facility. It contains a law-enforcement
training center; a “living laboratory” with
computer mapping software; and an age-
progression unit. Created under a
Congressional mandate, it assists in
locating and recovering missing children
and attempts to raise public awareness
about ways to prevent child abduction,
molestation, and sexual exploitation.

12. Level 3 Characteristics
 CONTENT: not only facts but abstract
concepts and language
 PURPOSE: varied, including discussion,
argumentation, evaluation of options,
value judgments of an institutional kind
 LEXICON: professional terminology,
some culture-specific terms
 STRUCTURES: complex syntax, stylistic
devices, nuanced expressions

13. Sample Level 3 Text
Any such prosecution that is not time-barred by
the applicable statute of limitations on the date
of the signing of the agreement may be
commenced against the Defendant,
notwithstanding the expiration of the statute of
limitations between the signing of this
agreement and the commencement of such
prosecutions. It is the intent of this agreement to
waive all defenses based on the statute of
limitations with respect to any prosecution that
is not time-barred on the date that this
agreement is signed.

14. Sample Level 3 Text
It is misleading to suggest that affirmative action
as practiced in the American university
admissions process is a subtle thumb on the
scale to help those whose circumstances have
given them few advantages. In fact, the
preferences typically accorded to affirmative
action beneficiaries in university admissions at
elite schools are hardly a subtle thumb. A heavy
hand would a more accurate description.

15. Level 4 Characteristics
 CONTENT: highly abstract and idiomatic
texts, including formal and informal writing
 PURPOSE: varied, such as argumentation and
commentary reflecting a specific culture
 LEXICON: professional, highly-technical,
colloquial, subtle, nuanced
 STRUCTURES: less frequent and highly
complex, requiring decoding, item analysis

16. Sample Level 4 Text
Notwithstanding any other provision of law to the
contrary, any bank holding securities as a fiduciary or
a custodian or agent of another fiduciary is authorized,
in the absence of specific provision to the contrary in
the governing instrument or court order under which
the bank is acting, to hold such securities in a manner
such that, with certification as to the ownership
attached, certificates representing securities of the
same class of the same issuer and from time to time
constituting assets of different fiduciary accounts are
held in bulk, including, to the extent feasible, the
merging of certificates of smaller denominations into
one or more certificates of larger denominations.

17. Translation Exams (TEs)
 TE Description
 4 passages/4 topics
 All passages are authentic
 Passage texts correspond to text
description for each level in SLDs.
 Difficulty levels: 2, 2+, 3, 4
 Passages must be translated in order
 2 parallel versions

18. TE Length: One Page
Level 2
Level 2+
Level 4
Level 3

19. Rating TEs
 Based on the check/probe process.
 Score reflects highest level attained.
 Mistranslations noted, to assist in
determining whether accuracy is at level.
 Grammar errors noted, stylistic devices
checked, to assist in determining whether
expression is at level.

20. Rating (cont’d)
 Each test is rated by two independent
raters and one certifier
 Approach is holistic, not discrete point
 Each passage is rated separately according
to how well it matches SL content:
SM, MM, IM, RM, or NT
 The examinee’s profile (e.g. SM/ IM/
IM/NT) in conjunction with ILR SLDs are
used to determine final rating

21. The next challenge:
Can we automate translation
testing?

22. Translation Online Testing
System (TOTS)
22
Research on meaning
equivalents for MT
evaluation
DARPA
Proof-of-concept
TSWG
Development
and Operationalization
FBI & TSWG
OUTCOME: TOTS
* Automated, web-
based translation
testing
* Lower costs
* Quicker turn-around

23. Hybrid Translation Edit Rate
(HyTER)
23
 Concept: If the system has a large set of
meaning-equivalent translations for a
sentence, we can automate evaluation of
translation.
 Approach: HyTER is a specialized
annotation tool for building mean-
equivalent translations starting with a
basic unit and building up

24. HyTER
 Annotators build networks and the
system generates alternative
translations
 The tool is web-based to enable off-
site annotation and global sharing.

25. HyTER Proof of Concept
25
This results in over 17 Million sentence alternatives:

26. 26
Five Components & Modules of TOTS
1. Hybrid Translation Edit Rate (HyTER) - Network
Building Tool
2. Admin Module
3. Exam Taking Module
4. Rater and Certifier Module
5. Translation Exam Development Module

27. Coming in 2015 – 2016:
27
 FBI Integration
 Testing of system by LTAU
 Piloting in FBI Field Offices
 DOJ Justice Cloud Integration
 Operability with existing workflow systems
e.g. LSMS
 HyTER-izing new Translation Exams

28. Contact Information:
K. Amanda Curry
K.Curry@ic.fbi.gov
202-436-7984