Telephone Game with Multilingual LLMs

Abstract

This is an experiment inspired by the children’s game “telephone.” In that game, a secret message is whispered to the first person, then whispered from person to person, and then the original and final messages are compared and everyone giggles. In this experiment, the whisperers are large language models (LLMs) and the message is translated to a different human language in each step, except for the final step in which the message is translated back to the starting language. We then measure the semantic drift using a sentence embedding model. We also explore the effects of different temperature and top_p settings. This game allows us to explore the accuracy and failure modes of LLM natural language translation.

Click here for the full PDF report.