Scorer for GermEval 2015: LexSub, version 2015-04-16 ==================================================== Contents -------- This archive contains scoring software for the GermEval 2015: LexSub shared task. The archive contents are as follows: README - this file README-semeval2007 - the original README file from the SemEval-2007 English Lexical Substitution task task10documentation.pdf - documentation for the SemEval-2007 English Lexical Substitution task score.pl - Perl script for scoring the "best" and "out of ten" (OOT) metrics. This is an updated version of Diana McCarthy's scorer from the English Lexical Substitution task at SemEval-2007, available at . scoreGAP.pl - Perl script for scoring the "generalized average precision" (GAP) metric trial-data - directory containing sample data for demonstrating the data format for the gold standard and "best", "OOT", and "GAP" scorers Instructions ------------ A complete description of each scorer's command-line options is printed when it is run without any command-line arguments. The following examples represent the most common use cases. To compute the "best" score for the system output against a gold standard: $ perl score.pl systemfile goldfile -t best To compute the "OOT" score for the system output against a gold standard: $ perl score.pl systemfile goldfile -t oot To compute the "GAP" score for the system output against a gold standard: $ perl scoreGAP.pl rankedfile goldfile The format of the input files for score.pl is described in the task10documentation.pdf file. The format of the input files for scoreGAP.pl is as follows: goldfile must contain one scoring item per line in the format word.pos id :: subst1 weight1;subst2 weight2; ... ;substn weightn and rankedfile must contain one item per line with a ranked list of substitutions in the format word.pos id :::: subst1;subst2; ... ;substn Tristan Miller and Chris Biemann, 2015-04-16