Lab: Phrase-based Statistical Machine Translation

Introduction

In this laboratory exercise, we will make use of Moses, an open-source phrase-based statistical machine translation decoder, which was developed at the 2006 JHU Summer Workshop. You will build several complete phrase-based statistical machine translation systems from very small amounts of training data, evaluate their performance, and identify ways that translation quality can be improved. Your resulting systems will be evaluated on test data (released after 1 hour after the start of the exercise). You should work in groups of 2-3.

Questions

It is not necessary to submit answers to the questions asked throughout this instruction page; however, you should take time to consider them carefully.

Preliminaries

Setting up your environment

The Moses decoder will not run on the Solaris desktops. Therefore, you will need to begin by logging into a node on the CLSP Linux cluster (x01-x63). You will then need to set up your environment to run the decoder and training scripts by issuing the following command (this will work under the bash shell, but if you are using a different shell you may have to execute a different command):

     source /export/ws06osmt/data/mtlab07/config-env.mt

You can verify this has worked properly by typing which moses and verifying its output:

[x32:~]$ which moses
/export/ws06osmt/data/mtlab07/scripts/moses

Setting up a workplace

Create a directory where you will work on this lab and copy the training and evaluation data to it:

     mkdir MT-LAB
     cd MT-LAB
     cp -r /export/ws06osmt/data/mtlab07/data/* .

Building an English-English machine translation system

We will start by building a system to translate from Early Modern English to Modern English by using two different English translations of the Bible as a resource, the King James Version (1611) and a more recent version (circa 1890).

Train the system:
lab-train train.kvj-modern
This step will take approximately 5 minutes. The training process involves using the IBM models 1, 3, and 4 to generate word alignments, from which phrases are then extracted. The phrases are gathered and statistics about the phrase translation probabilities are computed.
Translate the test set:
lab-translate train.kvj-modern
This will translate the unseen (held-out) test data in train.kvj-modern/test.foreign using the model constructed in step 1. The resulting file will be train.kvj-modern/test.foreign.trans. Does the translation sound more modern?
Evaluate the quality of the translation:
lab-eval train.kvj-modern
Results are reported in BLEU, which is a metric that ranges from 0 to 1 (commonly written as a perentage) and has been found to correlate strongly with human judgements of translation quality. BLEU is an imperfect approximation of translation quality, manual human evaluation should be attempted to whatever extent possible!

Building a Spanish-English translation system

The same steps above can be repeated with the parameter train.es-en. How does the BLEU score you obtain for English-English translation compare to the BLEU score you obtain for Spanish-English translation? What are the subjective differences? Can you see a pattern in the types of errors the translation system is making?

Comparison with an out-of-domain Spanish-English translation system

Statistical machine translation systems that often perform well in one domain may perform quite poorly in another. The directory model.project-syndicate.es-en contains a model built from an order of magnitude more training data; however, the training data consists mostly of news commentary (from Project Syndicate). You can evaluate how well this system performs on Bible translation with the following:

Translate the test set:
lab-translate model.project-syndicate.es-en
Evaluate the quality of the translation:
lab-eval model.project-syndicate.es-en
How does this compare to the in-domain system from the previous section? Is any aspect of the translations of the out-of-domain system better?

Improving the system

Now you should work to try to improve your Spanish-English translation system. This will be the system used for the competitive evaluation. Refer to the description of the lab tools and data formats for an explanation of how to make changes to the training and/or test data. Here is a list of some ideas to get you started, but be as creative as you want:

Lower-case the source and the target language text. Remember to do this to the test data, as well. The evaluation will be case-insensitive!
Use the Porter stemmer, or write your own stemming algorithm to transform the source text. If you also transform the target text, what effect will this have on human judgments?
Truncate the source language words to their first four characters.
Splitting of clitics. The training data in train.tok.es-en has had clitics split into separate words (del -> de el, llamarse -> llamar se, etc.). The evaluation data will be provided in both standard and decliticized forms.
You can change the maximum phrase length that is extracted during training (default is 7) by specifying --max-phrase-length n. Does a length of 3 change the quality? What about values greater than 7? Does decoding time/training time change?
There are a variety of other training options that you can specify as described in this documentation (these can be used with the lab-train command).
The Moses translation system has many hyperparameters you can adjust (these can be used with the lab-translate command). To see a full list, type moses with no parameters at the command line prompt. Some interesting ones to try are:

distortion penalty (-d 1)
word penalty (-w -1)
language model weight (-lm 1)

Add a Spanish-English dictionary.
Postprocess your the translation output to correct certain common mistakes.
Use data from the out-of-domain system. Note: the training data is not provided, but the phrase-tables are available (look in model.project-syndicate.es-en/model/moses.ini to find the path). How can this data be used?
Use better lambdas. The default lambdas created during the training process are quite poor (see MYTRAINDIR/model/moses.ini for the weights for the different features), and there are a variety of automated techniques for selecting better ones (Minimum Error Training, or MERT is the most widely-used one since it allows for the optimization of virtually any loss function and is guaranteed to converge.) What weights do you think make more sense? Those of you who are very ambitious can try to get mert running. You have access to the mert-moses.pl script and here's some documentation.
Find more Spanish-English parallel data on the web. All translated materials except the Bible are fair game, but be careful: training can take a long time if you have a lot of parallel data!
Basic system combination. Do you have one system that seems to perform better on some sentences and one that performs better on others? A simple classifier to select the output from either sytem might be a good approach to take.

Competitive evaluation

After one hour, you will be able to download the evaluation data here. Use your best system (as evaluated on the development data only) to translate it. Have one member of your group email me (see blackboard) with the path to your output file.

Rules

You may include extra training data in the system, but this may not be drawn from the Bible.
Google Translation, Babelfish, etc. are off limits.
The evaluation data should be treated as blind. Make decisions based on what system to use based on the dev-test data provided.
Evaluation will be conducted using case-insensitive BLEU.

Have fun!

Lab tools and data formats

This section describes the operation of the tools used in the lab and the data layout required. These tools are thin wrappers around the Moses tools, so it may be instructive to look at the sources (see use which lab-tool to find the path to the script).

Data format and layout

Each tool used in the lab takes a parameter that specifies a set of training data, the resulting model, and a set of development (dev-test) data. You are provided with four example systems (train.kvj-modern, train.es-en, train.tok.es-en, and model.project-syndicate.es-en), and you can use these as the starting point to create new systems. Each directory must minimally containing training data and evaluation data:

corpus.english - English (target) half of the training data. 1 sentence per line.
corpus.foreign - Foreign (source) half of the training data. 1 sentence per line.
test.english - reference translation for dev-test/evaluation data. (needed by lab-eval)
test.foreign - dev-test data (needed by lab-translate).

Once a system has been trained, the directory will contain a model subdirectory which contains the moses.ini configuration file, which contains references to all the models generated in the training step.

lab-train

The lab-train tool takes a single parameter that specifies a directory containing training data and uses this to build a model. If a model already exists, the command will fail. To clear an existing model, you can use the lab-clean command.

lab-translate

The lab-translate tool translates the contentes of test.foreign using the model and parameters in model/moses.ini and generates test.foreign.trans. To invoke Moses directly (to translate a different file or to translate command-line input), use the following:

      moses -f SYSTEM-DIR/model/moses.ini

Moses reads input from STDIN in the source language and writes translations to STDOUT.

lab-eval

The lab-eval computes the BLEU score of the translation output in test.foreign.trans using test.english as a reference.