Participation

Participation in the BioNLP shared task is free and open to all, academia and industry, individuals and groups. The official test evaluation period is over, but the online evaluation services remains available on each individual task pages. Registration to the shared task mailing list is requested.

Please see the main page for descriptions of the tasks and other information.

Contents

Instructions for participation

The general flow of participation is outlined in the following.

Registration

All participants should register for the shared task at this page. BioNLP-ST 2013 registration is closed

To use the online evaluation services,

  • please subscribe the shared task mailing list, bionlp-st (at) bionlp-st (dot) org,

  • by sending an e-mail to majordomo (at) bionlp-st (dot) org with "subscribe bionlp-st" in the body of the message.

  • Note that the mail has to be a plain text mail with no rich formatting.

  • The mailing list is archived at https://groups.google.com/d/forum/bionlp-st.

Sample, training and development data can be downloaded from the task page, without registration, but registration is required for use of the test server.

Registration does not require commitment to participate, and while registrants are requested to provide names and other identifying information to the organizers, this information will not be published. The BioNLP shared task allows anonymous participation: final results will be initially published without identifying information, and participants wishing to remain anonymous may withdraw from the task at this point.

Before training and development data release

Prior to the release of training and development data, small data samples for each task were available (Oct. 2012). The samples contain only a small number of abstracts each, and they are not intended for training machine learning methods or for precise testing, but intended to serve an example of the final data. The samples may allow participants to start system design and general implementation and rule writing for rule-based systems.

System training and development phase

The training and development data were available in January 2013 and the test data in April 2013. The period between these releases is the primary system training and development phase of the shared task.

During this phase, an online submission system accepting system outputs for development test data and returning evaluation results have been available. This system is identical to the submission system for final results. We strongly encouraged participants to test submitting their system outputs to the online development test set evaluation system to avoid surprises in final submission.

Test phase

Test data have been released April, 6 2013. Following test data release, participants had a period of ten days to submit their final results.

The submission system for final results have been opened after the test data were made available. To encourage early testing of submission to ensure that the format of the final submission is correct, the system accepts any number of submissions. However, to prevent fine-tuning against the test set, the final submission system does not provide immediate feedback on results. Additionally, only the last submission from each participant was considered in evaluation; possible prior submissions are simply discarded.

Final results will be announced to all participants one day after the close of the submission system.

Please note that while the shared task test data will be made available for further testing through the web interface after the shared task, the full gold annotations for this data will not be released at this time to allow the test data to continue to serve as the basis for stable evaluation after the task in a way that minimizes risk of overfit or otherwise unrealistic results. Participants wishing to perform manual error analysis are encouraged to do so on the development test data, which should have statistically identical properties to the test data.

After the evaluation, BioNLP workshop

After final results are published, the evaluation part of the BioNLP 2013 shared tasks is over.

After completion of the evaluation, all participants were encouraged to write a manuscript describing their system, analysis, and results for submission to the BioNLP 2013 shared task workshop. The manuscript submission deadline was 26 April 2013. Detailed instructions for authors were made available before the end of the evaluation phase.

The BioNLP 2013 shared task workshop held Aug 9 2013, co-hosted with the ACL/HLT BioNLP workshop in Sofia, Bulgaria.

Submission for the BioNLP workshop were handled separately and should be submitted through the submission BioNLP web page.

Guidelines for participation

Participants are encouraged to

Use any text resource

In addition to the training data for each task, participants were encouraged to make use of the development data for the task, the training and development data for any other task (or the 20011 task), any ontologies, large-scale unannotated resources (e.g. PubMed, PMC), and any other annotated corpus, in any way they see fit. The only restriction is that participants may not use human annotations for the test data (see also below).

Participants were encouraged to use the supporting resources.

Use any tool

Sentence splitters, parsers, taggers, coreference resolution tools and any other tools can be used, and both previously available tools and ones newly introduced/retrained/etc. by the participants can be used. The tools used for computing the supporting resource will be available.

Participants were also encouraged to make use of available event extraction systems, and may submit results from their own previously introduced systems. However, it is not sufficient for participation to submit the output of a system developed by another group (even if retrained), and participants making use of event extraction systems introduced by others in their own system should carefully evaluate the contribution of their proposed extensions or modifications to the performance of the base system.

We ask the participants to observe the following minimal restrictions.

One final submission per team per task

Participants may take part in any number and any combination of tasks, and there are no limits on the use of the development test evaluation system or the number of attempts to submit final results. However, only one final submission per team have been considered in each task, that is, the shared task does not allow multiple "runs" for the final test data.

No human annotation of test data

Participants were encouraged to use external tools and resources, including other manually annotated resources in addition to the shared task training data. However, participants should not perform any manual annotation of the final test data or use manual annotations created by other groups. In making use of annotated corpora, we would like to ask the participants to make sure that these do not overlap with the final test data e.g. by checking for PMID overlap.