# BB Test Results

## Task 1

### Main evaluation results

Legend

*S*: substitutions; see Evaluation algorithm below*D*: deletions; there is no predicted habitat corresponding to the reference habitat (false negative)*I*: insertion; there is no reference habitat corresponding to the predicted habitat (false positive)*M*: matches; see Evaluation algorithm below*P*: predicted; number of predicted habitats*SER = (S + D + I) / N*, where*N*is the number of habitats in the reference*Recall = M / N**Precision = M / P**F1*: harmonic mean of*Precision*and*Recall*

Evaluation algorithm

The evaluation performs a pairing between each reference habitat to a predicted habitat. The pairing maximizes a score defined as:

J . W

*J*is the Jaccard index between the reference and predicted entity as defined in [Bossy et al, 2012]. J measures the boundaries accuracy of the predicted entity.*W*is the semantic similarity between ontolgy concepts attributed to the reference entity and to the predicted entity. We use the semantic similarity described in [Wang et al, 2006]. This similarity is exclusively based on the is-a relationships between concepts, we set the*w*_{is-a}parameter to 0.65 in order to penalize favor ancestor/descendent predictions rather than sibling predictions.

Habitat entities in the reference that have no corresponding entity in the prediction are *Deletions* (*D* column).

Habitat entities in the prediction that have no corresponding entity in the reference are *Insertions* (*I* column).

The sum of the scores for all successful pairings is the *Matches* (*M* column). The difference between the number of pairings and the *Matches* is the *Substitutions* (*S* column).

### Entity boundaries evaluation

In this evaluation, the *Matches* are re-defined as the sum of the *J* component of the score for each pairing. In this way the scores measure the boundaries accuracy of predicted entities, without taking into account the semantic categorization.

Note however that the pairing still maximizes *J.W*. Therefore columns *I*, *D* and *P* remain unchanged.

### Ontology categorization evaluation

In this evaluation, the *Matches* are re-defined as the sum of the *W* component of the score for each pairing. In this way the scores measure the semantic categorization accuracy of predicted entities, without taking into account the entities boundaries.

Note however that the pairing still maximizes *J.W*. Therefore columns *I*, *D* and *P* remain unchanged.

In the following evaluations, the semantic weight attributed to the is-a relation has been altered:

w = 1

With a weight of 1, the score approches a "Manhattan distance" between the reference category and the predicted category; it is nearly equivalent to step counting semantic distances. It is more forgiving if the prediction is "in the vicinity" of the references, even though it is not an ancestor or descedent. It is more severe for predictions that further from the reference.

w = 0.1

With a weight of 0.1, the score favours predictions in the "lineage" of the reference, that is to say ancestors and descendants. It severly penalizes predictions of siblings. However, since the ontology root is the ancestor of all possible concepts, this score does not penalize predictions that are too general.

w = 0.8

0.8 is the value recommended by the authors of the semantic distance. It is shown for reference and bears no particular interest for the task.

## Task 2

### Main evaluation results

### Localization

This evaluation takes into account only Localization relations.

### PartOf

This evaluation takes into account only PartOf relations.

## Task 3

### Main evaluation results

### Evaluation algorithm

The evaluation performs a pairing between each relation in the reference with a predicted relation. The pairing maximizes the following score for relations of type Localization:

*B . J*

*B*is the*Bacterium*boundaries match. It is equal to 1 if the*Bacterium*arguments of the reference and the prediction have the exact same boundaries, otherwise 0.*J*is the*Localization*boundaries match. It is the Jaccard index between the*Localization*arguments of the reference and the prediction.

For relations of type *PartOf*, the score is 1 if the *Host* arguments overlaps and if the *Part* arguments overlap, otherwise 0. Boundaries are not taken into account in *PartOf* relations in order to not penalize boundaries mismatches twice; boundaries are already factored in the score of *Localization* relations.

In the case of equivalence between entities, the pairing uses the equivalent entity that maximizes the score. If several equivalent (and redundant) relations are found then the one that has the highest score is used.

The *Recall* is the sum of the scores of reference to prediction pairing divided by the number of relations in the reference.

The *Precision* is the sum of the scores of prediction to reference pairing divided by the number of relations in the prediction.

The *F1* is the harmonic mean of *Recall* and *Precision*.

### Alternate evaluations

Description of parameters

**Localization only**:*PartOf*relations have been removed from the pairing, the evaluation only measures the accuracy of*Localization*relations.**PartOf only**:*Localization*relations have been removed from the pairing, the evaluation only measures the accuracy of*PartOf*relations.**No boundaries**: the*Recall*and the*Precision*are computed from the number of pairings, as if*J*has been removed from the score formula. Note however that the pairing still maximizes "*B . J*"; only the scores are altered.**Relaxed bacteria**:*B*has been redifined as: 1 if the*Bacterium*arguments overlap, otherwise 0. Both scores and pairing are affected. The pairing maximizes the overlap between*Bacterium*arguments.

No boundaries

Relaxed bactreria

No boundaries, relaxed bacteria

PartOf only

Localization only

Localization only, no boundaries

Localization only, relaxed bacteria

Localization, no boundaries, relaxed bacteria