Title: What did you just call me? Computers versus Abusive Language. Abstract: The problem of abusive language and the detection of it, is still in the early phase of research with a host of open questions and community-wide agreements to be reached. Considering the increase in interest in hate speech and abusive language detection, it is vital that we consider the biases that we introduce through our annotations and the consequences that they have on the classification methods we employ, such that we can mitigate the influence of annotation errors.