Papers by Keyword: Arbitrariness

Abstract: In this paper, we propose a Template-Matching strategy combined with statistics in the tasks of human evaluation of NTCIR to avoid the arbitrariness of human evaluators. By practical operation and within-system checkup, the strategy helps avoid the arbitrariness of human evaluators and provides a benchmark for human evaluation of NTCIR.