GRN Test Results

Evaluation of submissions

Legend

  • S: substitutions; the predicted arc type is different from the reference arc type

  • D: deletions; there is no predicted arc corresponding to the reference arc (false negative)

  • I: insertion; there is no reference arc corresponding to the predicted arc (false positive)

  • M: matches; the predicted and reference arcs have the same type

  • SER = (S + D + I) / N, where N is the number of arcs in the reference

  • Recall = M / N

  • Precision = M / P, where P is the number of predicted arcs (P = S + I + M)

  • F1: harmonic mean of Precision and Recall

This is the main evaluation. The primary ranking criterion is the strict SER.

Evaluation algorithm

The evaluation algorithm operates pair by pair (of genes). For each pair, it tries to maximize Matches, then to maximize Substitutions. For instance, let's consider a pair of genes, where the reference says one Inhibition arc and one Transcription arc. Here is the error count for the following predictions:

  • Inhibition and Regulation: 1 Match, 1 Substitution

  • Activation and Binding: 2 Substitutions

  • Inhibition, Requirement and Binding: 1 Match, 1 Substitution, 1 Insertion

  • Regulation: 1 Substitution, 1 Deletion

Relaxed scores

The relaxed scores are computed the same way except that Substitutions are considered as Matches. This is an attempt to score the predictions regardless of the arc types. However it does not take into account the redundancy of arcs. For interpretation purposes, the following table is more accurate.

Network shape evaluation

Attention: team ranks are different

This evaluation has been done as if all arcs in the reference and in the prediction were of type Regulation. Redundant arcs were removed. Notice that, as expected, Substitutions is always equal to zero, and that relaxed scores are strictly equal to strict scores.

This evaluation gives the accuracy of the prediction regardless of the type of the arcs, more accurately than relaxed scores in the previous evaluation. A good score means that the prediction reproduces accurately the shape of the network. The gap between this evaluation and the previous one indicates the (in)accuracy of predicted arc types.

Valued network evaluation

In the same way as the previous evaluation, the arcs of type Binding and Transcription have been turned into Regulation. Again, redundant arcs have been removed. In this way the network only contains Regulation, Activation, Inhibition and Requirement arcs. In fact interactions of the mechanism axis have been removed, leaving only interactions of the effect axis. Some Systems Biology applications only need this kind of information. The gap between this evaluation and the first one indicates the (in)accuracy of predicted arcs of type Binding and Transcription.