Vocabulary learning via optimal transport for neural machine translation

Zhong Ruiyu

The choice of token vocabulary affects the performance of machine translation. This paper aims to figure out what is a good vocabulary and whether one can find the optimal vocabulary without trial training. To answer these questions, we first provide an alternative understanding of the role of vocabulary from the perspective of information theory. Motivated by this, we formulate the quest of vocabularization – finding the best token dictionary with a proper size – as an optimal transport (OT) problem. We propose VNMT, a simple and efficient solution without trial training. Empirical results show that VNMT outperforms widely-used vocabularies in diverse scenarios, including WMT-14 English-German and TED multilingual translation. For example, VNMT achieves almost 70% vocabulary size reduction and 0.5 BLEU gain on English-German translation.

Keywords:natural language processing, machine translation, vocabulary, optimal transport, machine learning, multilingual translation.