Test evaluation resultsThe BioNLP-ST 2013 is completed. Thanks to all participating teams.
The final results are given in the table below. Detailed results are available in the GRN Test Results page.
The goal is to retrieve all the genic interactions of the reference network –at least one occurrence per interaction– independently of where they are mentioned in the literature.
Compared to the BI task of BioNLPST’11, the evaluation will measure the capability of the IE systems to reconstruct the reference regulatory network. The corpus is an extension of the BioNLP-ST 2011 BI corpus derived from the LLL challenge corpus. For GRN, it has been expanded in order to cover more extensively the regulation network of a specific cellular function in Bacillus subtilis: the sporulation. This phenomenon is an adaptation of the bacteria to scarce resource conditions (e.g. low nutrients), it has been thoroughly studied in the past and the regulation network is stable and suffers no controversy.
The annotation was revised and enriched by a joint effort of the Bibliome team of MIG Laboratory at the Institut National de Recherche Agronomique (INRA) and the Laboratoire d'Informatique de Paris Nord at the Université Paris 13. The annotation has been carried and validated by a senior bioinformatics/Bacillus subtilis specialist, and by a bioinformatics/NLP engineer by using AlvisAE, the Annotation Editor. The annotation guidelines will be available in English.
The following picture is the regulation network corresponding to the training data (click for larger image).
The GRN task is a relation extraction task that follows the BioNLP-ST 2013 frame of representation. The participants are provided a manually curated annotation of the training corpus including entities, events and relations, including genic interactions. For training, the participants are provided the genic regulation network that can be reconstructed with interactions mentioned in sentences of the training corpus.
The network is a directed graph where vertexes represent genes, and arcs represent interactions between genes extracted from the text. The arcs are labeled with an interaction type following two distinct axes:
When no mechanism or effect can be inferred, then the arc is labelled Regulation.
All text-bound entities are given as input in train and test phases, except for event triggers (Action) that are only given in the train phase. For genic entities, only those belonging to Bacillus subtilis are annotated; genes and proteins of other organisms are not annotated.
All event and relations are given in the train phase. In the test phase they are not given, however participants are only evaluated on the prediction of Interaction.* relations. The other events and relations are provided as a guidance during the training of the systems.
The following types of event and arguments are given, along with the valid types for each argument.
In the annotated corpus, genic entities that can potentially interact are assigned a Gene Identifier. This identifier is the name of the gene, operon or family denoted by the entity. For instance, the Gene Identifier for Protein entities is the name of the gene that encodes for the annotated protein. The provided Gene Identifier saves the participants the inconvenience of searching through nomenclatures of B. subtilis genes.
The genic regulation network corresponding to a corpus is inferred from the set of Interaction relations (manually annotated or predicted). The inferrence is done in two steps: resolution of Interaction relations, and Removal of redundant arcs. The training data is distributed with a script that automatically performs these two steps.
The Agent and the Target of an Interaction relation are not necessarily an entity with a Gene Identifier. They can be secondary events or relations (Action_Target, Transcription_by, or even another Interaction), or auxiliary entities (Promoter). The resolution of an Interaction aims to look for the entity with a Gene Identifier in order to infer the node concerned by the Interaction relation. The resolution of Interaction arguments is performed with the following rules:
These rules are applied iteratively. In other words the resolution of Interaction arguments is a traversal of the graph of annotations; event and relation arguments are walked through, and Promoter entities are walked through according to rules 4 and 5.
If the resolution of the Agent or the Target yields more than one node, then the Interaction resolves to as many arcs as the cartesian product of resolved nodes. For instance, if both the Agent and the Target resolve to two nodes, the the Interaction relation resolves into four arcs.
Moreover Interaction types are ordered according to the following hierarchy:
For a given arc, if there is another arc for the same node pair with a more specialized type, then it is removed. For instance, the arcs (A, Regulation, B) and (A, Transcription, B) are simplified into (A, Transcription, B). Indeed the former arc conveys no additional information in comparison with the latter.
Participants can submit predictions in two ways:
The predicted network is compared to the reference network using a Slot Error Rate [Makhoul et al, 1999]:
Since this measure is an error rate, the lower is the better: a SER of zero means a perfect prediction. The SER has no upper bound but a value below 1 is expected for decent predictions.
The participants are provided a script that performs Interaction resolution and evaluation against a reference.
Illustrative examples can be downloaded here: Sample Data