Bacteria Biotopes (BB)

Downloads

Training data

Development data

Test data

Validation/Evaluation tool

Test evaluation service

Test evaluation results

The BioNLP-ST 2013 is completed. Thanks to all participating teams.

The final results are given in the tables below for each sub-task. Detailed results are available in the BB Test Results page.

Task 1

Task 2

Task 3

Good luck.

Goals in IE

• 1. Promote Information Extraction on the subject of microorganisms ecology.

  2. Assess the performance of automatic categorization systems.

  3. Assess the performance of relation extraction on this subject by different methods.

Motivation in biology

There is surprisingly no comprehensive database of natural environment location of bacteria, although this is a critical information for studying the interaction mechanisms of the bacteria with its environment at a molecular level. The literature on bacteria ecology is abundant. As in 2011, the extraction of habitat mentions and their attachment to bacteria from textual data would fill this need. Moreover, once the habitats are identified, they must be normalized to be compared. Ontologies of habitats such as EnvO and OntoBiotope are available. There is a nice opportunity to develop and evaluate methods for normalizing the habitat descriptions with these ontologies. This task is more generally related to semantic annotation of entities with ontologies.

The knowledge tackled by this task is the habitats where bacteria live, and the environment properties of bacteria. This information is a particularly interesting in the fields of food processing and safety, health sciences and waste processing. There are also fundamental research that requires this knowledge like metagenomics or phylogeography / phyloecology. There is currently no database that supply the habitats of bacteria in a comprehensive way. Moreover the efforts for normalizing the habitats are just beginning. The diversity of habitats is such that several ongoing projects aim at building habitat ontologies (EnvO, OntoBiotope).

Representation and Task Setting

The BB task consists in:

• • Entity recognition of bacteria taxa and bacteria habitats.

  • Bacteria habitat categorization through the OntoBiotope-Habitat ontology.

  • Extraction of localization relations between bacteria and habitats.

We propose three sub-tasks:

• 1. entity detection and categorization

  2. localization relation extraction

  3. localization extraction without gold entities

The three sub-tasks are independant, the evaluation metrics will be distinct for each one. Thus participants may opt to submit to one, two or three sub-tasks.

Sub-tasks 1 and 2 are elementary since they have a single prediction target (categorizations and relations, respectively). Sub-task 3 requires the prediction of relation as well as the boundaries of their arguments, so they may require more complex prediction systems.

Sub-task 1: entity detection and categorization

In sub-task 1, participants must detect the boundaries of bacteria habitat entities and, for each entity, assign one or several concepts of the OntoBiotope ontology. It contains 1,700 concepts organized in a hierarchy of is-a relations. It is available for download in OBO format.

In the training phase, participants are provided with document texts along with manually annotated habitat entities and their concept attributions. In the test phase, participants are provided with document texts alone, their systems must predict habitat entities and their concept attributions.

Entity types. There is a single entity type: Habitat that denote mentions of potential bacteria habitats. These entities may be named entities, but they are mostly noun phrases, adjectives or subordinates.

Normalization types. There is a single normalization type: OntoBiotope that links Habitat entities to one or several concepts in the OntoBiotope ontology. All Habitat entities are associated to at least one concept.

Evaluation.

The evaluation is based on a 1-to-1 pairing between reference entities and predicted entities. This pairing maximizes a score S defined as similarity between a reference and a prediced entity:

S = J . W

• • J is the Jaccard index between the reference and predicted entity as defined in [Bossy et al, 2012]. J measures the boundaries accuracy of the predicted entity.

  • W is the semantic similarity between ontolgy concepts attributed to the reference entity and to the predicted entity. We use the semantic similarity described in [Wang et al, 2006]. This similarity is exclusively based on the is-a relationships between concepts, we set the w_is-a parameter to 0.65 in order to penalize favor ancestor/descendent predictions rather than sibling predictions.

Submissions will be evaluated using the Slot Error Rate (SER):