Format

Both files are tab-separated. The contents of the columns is described below.

1. -annotation
Columns:
- ID: annotation ID, unique
- Source: source sentence
- Target: machine translation of the source
- Feedback_Type: type of collected feedback, either "post_edit", "marking", or "user_decide". 
- User_Choice: identical to Feedback_Type except for "user_decide", where it contains either "post_edit" or "marking" depending on the annotator's choice of feedback mode.
- User_ID: user identification
- Annotation: List of true/false for markings corresponding to white-space tokenized targets, where true indicates that an error was marked, or the full corrected translatoin if in post-editing mode.  
- Key_Strokes: number of keystrokes, excluding submit button
- Time_Started: time stamp of start
- Time_Submitted: time stamp of submission
- Time_Paused: time that was paused
- Click_Count: number of clicks counted during the annotation
- Sentence_ID: sentence identifier, e.g. 19p2.5: talk 19, part 2, 5th sentence -- every talk was split into 3 parts for the three editing modes.
- Agreement: whether it was repeated for agreement evaluation (all 0 for this table, and all 1 for the other one)
- Duration: total duration for the annotation in seconds, excluding pauses
- Reference: reference translation as provided in the TED translation corpus (IWSLT17)
- KSMR: key-stroke-mouse-action ratio
- Action_Count: absolute number of clicks and key strokes
- sBLEU: per-sentence BLEU
- TER: per-sentence TER
- CHRF: per-sentence ChrF score
- Topic: hand-chosen label of one of the talk's keywords
- Talk_ID: talk identifier
- Split_ID: split identifier, see Sentence_ID
- Sentence_Num: sentence number within talk
- Post-Edit: contains post-edits for post-edited sentences, empty otherwise
- Marking: contains markings (0.0: correct, 1.0: error/incorrect) for all words in the translation, and whether a word was post-edited (1.0) or not (0.0) in post-edit mode
- Marked_Ratio: ratio of words marked as errors over total number of words per sentence
- Source_Length: length of the source sentence in characters
- Target_Length: length of the machine translation
- Reference_Length: length of the reference translation



2. -agreement
Subset of the columns described above. The Annotated column is set to 1.
