Data-driven Evaluation of Computer-assisted Translation Systems for Pedagogocial Purposes

Abstract

Human translators increasingly work with computer-assisted translation (CAT) systems to help them translate better and faster. Several courses at our department are devoted to training students to use these systems. Besides acquiring proficiency with these systems, students also need to be cognizant of the strengths and limitations of CAT, and the types of translation work for which CAT is appropriate. These topics are best taught not by abstract discussion, but through first-hand experiments to evaluate CAT performance under different conditions. Unfortunately, none of the existing systems support this type of evaluation. We propose to develop a CAT evaluation system that enables students to vigorously and quantitatively investigate these topics. For example, students will compare different algorithms for sentence matching and for mining translation examples, and the amount of effort needed for post-editing system output; as a result, they will be able to analyze whether, and how much, CAT outperforms other translation methods. Since many of these topics are open questions, statistics collected by the proposed evaluation system are also expected to contribute to CAT research.