Evaluation set for DCDC4

This folder contains evaluation data for DBDC4 (dialogue breakdown detection challenge 4).
There are two folders "en" and "jp" containing data for English and Japanese.
The format of the files can be found here: https://dbd-challenge.github.io/dbdc3/datasets

* English data

Under "en" folder, you have 200 json files. 

The dialogue data were from the conversational intelligence challenge (ConvAI) and the IRIS system (Banchs et al.)
The prefix of the json files represent the anonymized bot names in ConvAI (Bot001-006) and IRIS.

* Japanese data

Under "jp" folder, you have a total of 223 json files. 

You have 50 dialogues each for DCM, DIT, and IRS. In addition, you have 73 dialogues for
five systems (IRS MMK MRK TRF ZNK) under "dbd_livecompe_eval".

* Note

- You can build your detectors for each dialogue system (e.g., IRIS, DCM etc.)
- You cannot use utterances after the target utterance for breakdown detection;
  this is because we want to build techniques for a system to avoid dialogue breakdowns.
- please use t=0.0 so that the gold label is determined by the simple majority of the annotators.

* Contact

If you have further questoins regarding the data, please let us know by the following email address.
dbdc4-admin@googlegroups.com