6. Computing Character Error Rate

While the Character Error Rate (CER) of a model is measured on the Validation Data pages at the end of the training process, testing a model on new pages (not included in your Training or Validation Data) and computing its accuracy lets you see how it performs in real-world scenarios. You can compute its accuracy at both the page level and the document level.

Computing CER on a Single Page

Computing CER across an Entire Document

Requirements

To compute accuracy, you need to compare two distinct versions of the transcription:

1. Ground Truth (Reference): A manual transcription that is as accurate and close to the original text as possible.
2. Hypothesis: The automatic transcription generated by the Text Recognition model you want to evaluate.

Tip: A simple way to create these two versions is to run Text Recognition on your page first, manually correct the output to make it perfect, and save it as your Ground Truth version.

Computing CER on a Single Page

Evaluating accuracy at the page level gives you both a precise CER percentage and an immediate visual breakdown of where the model succeeded or failed.

1. Open the page in the Transkribus editor.
2. Click the Clock icon in the top-right corner to open the page version history.
3. Ensure your Ground Truth version is currently open:
   • If it was the last version saved, it is open by default.
   • If it is not open, click the three dots (⋮) next to the Ground Truth version and select Open this version.
4. Find the version generated by the Text Recognition model you want to evaluate, click its three dots (⋮), and select Compare with current version.
5. Click Compute CER to generate the error rate for this specific page.

Transkribus highlights the differences between the two versions directly on the text editor:

• Red: Words transcribed incorrectly by the Text Recognition model.
• Green: The correct text as written in the Ground Truth version.
• Uncolored: Passages where the model output matches the Ground Truth perfectly.

The visual comparison helps you quickly spot small errors, such as minor punctuation mismatches or capitalization discrepancies, versus major misreadings.

Computing CER across an Entire Document

To measure accuracy across multiple pages or an entire document at once:

1. Ensure all selected pages contain both a Ground Truth version and a version generated by a Text Recognition model.
2. Select the pages or the document you want to evaluate.
3. Click the downward arrow (Action menu) and select Compute CER.
4. In the configuration window:
   • Reference: Select your Ground Truth version.
   • Hypothesis: Select the model you want to evaluate.
   • (Pages that do not contain either a Ground Truth version and a model-generated version will be excluded)
5. Click Compute.

Once processing completes (which may take a few seconds to a few minutes depending on the number of pages and the number of lines per page), Transkribus displays:

• The Average CER for the selection.
• Page-by-page breakdowns for both CER (Character Error Rate) and WER (Word Error Rate).

You can export the accuracy results as a CSV file for further analysis or reporting.

By testing multiple models on the same sample pages, you can compare their accuracy against your Ground Truth. This helps you determine which model performs best (has the lowest CER) and provides a clear baseline to decide which one to use to recognise your entire collection.