neleval list-measures
¶
List measures schemes available for evaluation
Usage summary¶
$ neleval list-measures --help
usage: neleval list-measures [-h] [-m NAME]
List measures schemes available for evaluation
optional arguments:
-h, --help show this help message and exit
-m NAME, --measure NAME
Which measures to use: specify a name (or group name)
from the list-measures command. This flag may be
repeated.
List all predefined measures¶
$ neleval list-measures
The following lists possible values for --measure (-m) in evaluate,
confidence and significance. The name from each row or the name of a
group may be used.
Name Aggregate Filter Key Fields In groups
=========================== ============ ========= ============== =======================================================
b_cubed b_cubed None span all, all-coref, luo, tac11, tac14
b_cubed_plus b_cubed None span+kbid all, all-coref, tac11, tac14
entity_ceaf entity_ceaf None span all, all-coref, luo, tmp
entity_match sets is_linked docid+kbid all, all-tagging, cornolti, hachey
mention_ceaf mention_ceaf None span all, all-coref, luo, tac14, tmp
mention_ceaf_plus mention_ceaf None span+kbid all, all-coref
muc muc None span all, all-coref, luo
pairwise pairwise None span all, all-coref, tmp
strong_all_match sets None span+kbid all, all-tagging, tac09, tac11, tac14
strong_link_match sets is_linked span+kbid all, all-tagging, cornolti, hachey, tac09, tac11, tac14
strong_linked_mention_match sets is_linked span all, all-tagging, cornolti, hachey
strong_mention_match sets None span all, all-tagging, hachey, tac14
strong_nil_match sets is_nil span all, all-tagging, tac09, tac11, tac14
strong_typed_all_match sets None span+type+kbid all, all-tagging, tac14
strong_typed_link_match sets is_linked span+type+kbid all, all-tagging
strong_typed_mention_match sets None span+type all, all-tagging, tac14
strong_typed_nil_match sets is_nil span+type all, all-tagging
typed_mention_ceaf mention_ceaf None span+type all, all-coref, tac14
typed_mention_ceaf_plus mention_ceaf None span+type+kbid all, all-coref
Default evaluation group: all
In all measures, a set of tuples corresponding to Key Fields is
produced from annotations matching Filter. Aggregation with `sets'
compares gold and predicted tuple sets directly; coreference
aggregates compare tuples clustered by their assigned entity ID.
A measure may be specified explicitly. Thus:
strong_all_match
may be entered as
sets:None:span+kbid
Available aggregates are:
- non-clustering: overlap-maxmax, overlap-maxsum, overlap-summax, overlap-sumsum, sets
- clustering: b_cubed, entity_ceaf, mention_ceaf, muc, pairwise, pairwise_negative
Available filter and key fields: candidates, docid, eid, end,
is_first, is_linked, is_nil, kbid, link, score, span, start, type.
More fields can be stored dynamically by entering a candidate's type
as a JSON key-value mapping.