Home // IMMM 2020, The Tenth International Conference on Advances in Information Mining and Management // View article

Towards Inter-Rater-Agreement-Learning

Authors:
Kai-Jannis Hanke
Andy Ludwig
Dirk Labudde
Michael Spranger

Keywords: Agreement Measures; Weighted Majority Vote; Text Labeling

Abstract:
While technological advances and improved algorithms enhance most scientific fields, there remains a simple problem in many domains. If a decision has to be made we resort to simple majority votes or utilize agreement measures to determine how unanimous a decision is. Especially in text classification, a text is usually sorted into a specific category based on how many people agreed on it. However, the problem is that in these methods the individual that made the decision is neglected. Therefore, we propose a weighted approach that includes a flexible feature space and adjustments to the weights not only according to the individual’s expertise but also to their performance on previous tasks. Preliminary experiments with a data set including short music related texts yield promising results with fewer cases for which no majority vote was achieved.

Pages: 10 to 14

Copyright: Copyright (c) IARIA, 2020

Publication date: September 27, 2020

Published in: conference

ISSN: 2326-9332

ISBN: 978-1-61208-806-8

Location: Lisbon, Portugal

Dates: from September 27, 2020 to October 1, 2020