Views on machine translation: An analysis on the performance of DeepL and Google Translate

Authors

Keywords:

Machine translation output error typology, Neural machine translation, Machine translation evaluation

Abstract

This paper comparatively evaluates the performance of two machine translation systems, namely DeepL and Google Translator. It starts with a brief overview of machine translation using neural networks, followed by a reflection concerning the performance of machine translation output using human and automatic evaluation. After that, this study addresses the difference in performance between both systems and identifies possible linguistic problems, possibly generated from these machine translation output. To carry on such comparison, this study employs an excerpt from Machado de Assis's Dom Casmurro, taken form the chapter "Olhos de Ressaca". This investigation uses the automatically generated translation outputs to critically analyze which system conveys more quality in their translation output and to comparatively examine linguistic errors generated by DeepL e Google Translate results. The evaluation criteria include the accuracy of the translation, the ability to maintain the meaning and structure of the original sentence, as well as fluency and adequacy of the resulting translation.

Downloads

Download data is not yet available.

Author Biographies

Renata Ribeiro da Silva, Universidade de Brasília

Bacharel em Línguas Estrangeiras Aplicadas ao Multilinguismo e à Sociedade da Informação (LEA-MSI) pela Universidade de Brasília (UnB).

Thiago Blanch Pires, Universidade de Brasília

Professor adjunto vinculado ao Departamento de Línguas Estrangeiras e Tradução (LET) e ao Programa de Pós-Graduação em Linguística (PPGL) da Universidade de Brasília (UnB). Doutor em Ciência da Informação pela UnB e mestre em Letras-Inglês pela UFSC.

References

ASSIS, M. de. Dom Casmurro. Rio de Janeiro: Edições Câmara, 2019.

BANITZ, B. Machine translation: a critical look at the performance of rule-based and statistical machine translation. Cadernos de Tradução, 40(1), 2020. Disponível em:

https://doi.org/10.5007/2175-7968.2020v40n1p54 . Acesso em 17 de novembro de 2023.

BROWNLEE, J. A Gentle Introduction to Calculating the BLEU Score for Text in Python. 2019. https://machinelearningmastery.com/calculate-bleu-score-for-text-python/ > Acesso em 17 de novembro de 2023.

CALLISON-BURCH, C., OSBORNE, M., & KOEHN, P. Re-evaluating the Role of BLEU in Machine Translation Research, 2006. https://aclanthology.org/E06-1032.pdf . Acesso em 17 de novembro de 2023.

DEEPL PRESS INFORMATION. (n.d.). https://www.deepl.com/en/press.html. Acesso em 17 de novembro de 2023.

DENSMER, L.. Interview with an Expert: How Do You Measure MT? RWS. 2019. https://www.rws.com/blog/interview-with-an-expert-how-do-you-measure-mt/ . Acesso em 17 de novembro de 2023

FORCADA, M. Machine translation today. In Handbook of Translation Studies (pp. 215–223). John Benjamins Publishing Company, 2010.

GOOGLE CLOUD. Como avaliar modelos. 2022. https://cloud.google.com/translate/automl/docs/evaluate?hl=pt-br. Acesso em 17 de novembro de 2023.

HUTCHINS, W. J., & SOMERS, H. L. An introduction to machine translation (pp. 215-223). Academic Press, 1992.

ISABELLE, P., CHERRY, C., & FOSTER, G. A challenge set approach to evaluating machine translation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2486–2496, 2017. https://aclanthology.org/D17-1263.pdf . Acesso em 17 de novembro de 2023.

ISABELLE, P., & KUHN, R. A Challenge Set for French -> English Machine Translation. 2018. https://arxiv.org/pdf/1806.02725.pdf. Acesso em 17 de novembro de 2023.

MATTHES, E. Curso Intensivo de Python. Novatec Editora. 2016.

PLANETCALC. Levenshtein distance.2022. http://planetcalc.com/1721. Acesso em 17 de novembro de 2023.

SCHUSTER, M., JOHNSON, M., & THORAT, N. Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System. 2016.

https://ai.googleblog.com/2016/11/zero-shot-translation-with-googles.html . Acesso em 17 de novembro de 2023.

TAN, Z., WANG, S., YANG, Z., CHEN, G., HUANG, X., SUN, M., & LIU, Y. Neural machine translation: a review of methods, resources, and tools. AI Open, 5–21, 2020.

Published

2024-11-26

How to Cite

SILVA, R. R. da; PIRES, T. B. Views on machine translation: An analysis on the performance of DeepL and Google Translate. Babel: Revista Eletrônica de Línguas e Literaturas Estrangeiras, Alagoinhas, BA, v. 14, p. e20368, 2024. Disponível em: https://revistas.uneb.br/index.php/babel/article/view/20368. Acesso em: 4 dec. 2024.