Python command-line evaluation scripts for TAC entity linking and related wikification, named entity disambiguation, and within- and cross-document coreference tasks.
It aims for fast and flexible coreference resolution and sophisticated named entity recognition evaluation, such as partial scores for partial overlap between gold and system mentions. CEAF, in particular, is much faster to calculate here than in the CoNLL-11/12 scorer. It boasts features such as configurable metrics; accounting for or ignoring cross-document coreference (see the evaluate –by-doc flag); plotting to compare evaluation by system, measure and corpus subset; and bootstrap-based confidence interval calculation for document-wise evaluation metrics.
- Basic Usage
- File formats
- Command-line reference
- Convenience scripts for TAC KBP evaluation
This project extends the work described in:
- Ben Hachey, Joel Nothman and Will Radford (2014), “Cheap and easy entity evaluation”. In Proceedings of ACL.
It was used as the official scorer for Entity (Discovery and) Linking in 2014–:
- Heng Ji, Joel Nothman and Ben Hachey (2014), “Overview of TAC-KBP2014 Entity Discovery and Linking Tasks”, In Proceedings of the Text Analysis Conference.
- Heng Ji, Joel Nothman, Ben Hachey and Radu Florian (2015), “Overview of TAC-KBP2015 Tri-lingual Entity Discovery and Linking Tasks”, In Proceedings of the Text Analysis Conference.
- Heng Ji and Joel Nothman (2016), “Overview of TAC-KBP2016 Tri-lingual EDL and Its Impact on End-to-End KBP”, In Proceedings of the Text Analysis Conference.