Emotions Evoked by Common Words and Phrases:
Using Mechanical Turk to Create an Emotion Lexicon

Saif M. Mohammad (joint work with Peter D. Turney)

Even though considerable attention has been given to semantic
orientation of words and the creation of large polarity lexicons,
research in emotion analysis has had to rely on limited and small
emotion lexicons.  We show how we create a high-quality emotion
lexicon using Mechanical Turk.  We will flesh out challenges pertaining
to both emotion annotation and the use of Mechanical Turk, and
show how we address them. In addition to  questions about
emotions associated with terms, we show how the inclusion of a word
choice question can discourage malicious data entry, help identify
instances where the annotator may not be familiar with the target term
(allowing us to reject such annotations), and help obtain annotations
at sense level (rather than at word level). We perform an extensive
analysis of the annotations to better understand the distribution of
words associated with emotions. We show how without any
training or control over the educational background of the annotators,
one can still obtain high-quality and high-agreement emotion annotations.

The talk draws from work described in this paper:
Emotions Evoked by Common Words and Phrases: Using Mechanical Turk
to Create an Emotion Lexicon. Saif Mohammad and Peter Turney. Proceedings
of Workshop on Computational Approaches to Analysis and Generation of
Emotion in Text at NAACL-HLT, June 2010.


Bio:
Saif Mohammad is a Research Officer at the National Research Council
Canada (NRC). He received his Ph.D. in Computer Science from
University of Toronto in 2008. He was a Research Associate in
the Institute of Advanced Computer Studies at the University of
Maryland College Park before joining NRC in 2009.  Saif's
research interests are in Natural Language Processing, especially
Lexical Semantics, and include developing computational models for
emotion detection, semantic distance, and lexical-semantic relations
such as word-pair antonymy.