Legend
S: substitutions; the predicted arc type is different from the reference arc type
D: deletions; there is no predicted arc corresponding to the reference arc (false negative)
I: insertion; there is no reference arc corresponding to the predicted arc (false positive)
M: matches; the predicted and reference arcs have the same type
SER = (S + D + I) / N, where N is the number of arcs in the reference
Recall = M / N
Precision = M / P, where P is the number of predicted arcs (P = S + I + M)
F1: harmonic mean of Precision and Recall
This is the main evaluation. The primary ranking criterion is the strict SER.
Evaluation algorithm
The evaluation algorithm operates pair by pair (of genes). For each pair, it tries to maximize Matches, then to maximize Substitutions. For instance, let's consider a pair of genes, where the reference says one Inhibition arc and one Transcription arc. Here is the error count for the following predictions:
Inhibition and Regulation: 1 Match, 1 Substitution
Activation and Binding: 2 Substitutions
Inhibition, Requirement and Binding: 1 Match, 1 Substitution, 1 Insertion
Regulation: 1 Substitution, 1 Deletion
Relaxed scores
The relaxed scores are computed the same way except that Substitutions are considered as Matches. This is an attempt to score the predictions regardless of the arc types. However it does not take into account the redundancy of arcs. For interpretation purposes, the following table is more accurate.
Attention: team ranks are different
This evaluation has been done as if all arcs in the reference and in the prediction were of type Regulation. Redundant arcs were removed. Notice that, as expected, Substitutions is always equal to zero, and that relaxed scores are strictly equal to strict scores.
This evaluation gives the accuracy of the prediction regardless of the type of the arcs, more accurately than relaxed scores in the previous evaluation. A good score means that the prediction reproduces accurately the shape of the network. The gap between this evaluation and the previous one indicates the (in)accuracy of predicted arc types.
In the same way as the previous evaluation, the arcs of type Binding and Transcription have been turned into Regulation. Again, redundant arcs have been removed. In this way the network only contains Regulation, Activation, Inhibition and Requirement arcs. In fact interactions of the mechanism axis have been removed, leaving only interactions of the effect axis. Some Systems Biology applications only need this kind of information. The gap between this evaluation and the first one indicates the (in)accuracy of predicted arcs of type Binding and Transcription.