Guidelines

Can I use extra data or pre-trained models?

The use of any external audio/video/text data and pretrained model that is not provided by organizers is allowed under the following conditions:

  • These external resource used must be referenced and freely accessible to any other research group worldwide
  • The technical report must indicate the list of external data sources used in training.
  • Participants must email the organizers about external data sources, and the list will be updated on the website accordingly. After the evaluation set is published, no further external data sources will be allowed.

Which additional meta information (excluding raw audio, video, and label) can I use?

The following additional meta information will be provided:

  • the corresponding room sizes
  • the corresponding speaker labels and information

Which information shall I not use?

The following processes are not allowed:

  • Manual data or annotation modifications (e.g., manual refinement of the utterance start and end times) are forbidden.
  • All parameters should be tuned on the training set or the development set. Modifications of the development set are allowed, provided that its size remains unchanged. These modifications do not induce the risk of inadvertently biasing the development set toward the particular speakers or acoustic conditions in the evaluation set. For instance, enhancing the signals, applying “unbiased” transformations, or automatically refining the utterance start and end times is allowed.
  • It is forbidden to augment the development set by generating simulated data, applying biased signal transformations (e.g., systematically increasing intensity/pitch), or selecting a subset of the development set.

If you have questions, ask before the submission deadline.

Which results should I submit?

The following results will be submited:

  • During the leaderboard updating phase, you should submit a text file containing the key for each test sample and its corresponding output result for the tested system.
  • After the leaderboard is frozen, you will need to submit a technical report detailing the implementation of the best-performing system.

The organizing committee will assess the authenticity and compliance of the system based on the technical report to determine the final rankings.