Evaluating Dialogue Systems via an Opinion
type: Dataset
creator: Nedelchev, Rostislav
  type: Person
  given name: Rostislav
  family name: Nedelchev
date published: 2023-06-05
date modified: 2023-06-05
version: 1
description: Dialogue systems are a significant field of research and development in artificial intelligence. Until today, the evaluation of such algorithms happens in one fundamental way. They solve "hypothetical problems," i.e., dialog systems are tested by being asked to respond in specific scenarios and provide a "solution", i.e. a reply, to a "problem". The replies are either explicitly compared to reference responses using overlap-based metrics (e.g., BLEU) or are evaluated by human annotators, which can also be seen as compared to (implicit) references. Instead, we propose to ask dialogue systems to tell whether a sample "solution" to a sample "problem" is good or bad. In other words, we ask another dialog system whether a conversation is fluent and coherent or not, and to what degree. In our experiments, we show how to evaluate dialogue systems by "asking for an opinion" and that it indeed offers an additional perspective on assessing these methods.
keywords: Computer and Information Science
citation: urn:nbn:de:hbz:5-70983
license: http://creativecommons.org/publicdomain/zero/1.0
distribution: convai1_preprocess.py
distribution: convai2_preprocess.py
distribution: cornell_preprocess.py
distribution: env.yml
distribution: Readme.md
distribution: ubuntu_preprocess.py
distribution: convai1.json
distribution: conversation_length.pkl
distribution: sentences.pkl
distribution: sentence_length.pkl
distribution: convai2.json
distribution: chameleons.pdf
distribution: id2word.pkl
distribution: movie_characters_metadata.txt
distribution: movie_conversations.txt
distribution: movie_lines.txt
distribution: movie_titles_metadata.txt
distribution: raw_script_urls.txt
distribution: README.txt
distribution: word2id.pkl
distribution: configs.py
distribution: convai1_results.pickle.bz2
distribution: convai2_results.pickle.bz2
distribution: correlations.ipynb
distribution: data_loader.py
distribution: eval.py
distribution: eval_4models_convai.sh
distribution: eval_convai.py
distribution: eval_embed.py
distribution: gpt_convai_eval.py
distribution: solver.py
distribution: sqlitedict_compress.py
distribution: train.py
distribution: train_4models.sh
distribution: __init__.py
distribution: 30.pkl
distribution: config.txt
distribution: convai1.sqlite
distribution: convai2.sqlite
distribution: events.out.tfevents.1604760827.ff8fc1a53fe2
distribution: samples.txt
distribution: events.out.tfevents.1604756310.ff8fc1a53fe2
distribution: events.out.tfevents.1604773321.ff8fc1a53fe2
distribution: events.out.tfevents.1604766709.ff8fc1a53fe2

urn:nbn:de:hbz:5-70983
type: CreativeWork
name: Nedelchev, Rostislav: Automatic Evaluation of Dialogue-Systems Using Neural-Network Methods. - Bonn, 2023. - Dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn. Online-Ausgabe in bonndoc: https://nbn-resolving.org/urn:nbn:de:hbz:5-70983

convai1_preprocess.py
type: DataDownload
encoding format: text/x-python
content size: 6445

convai2_preprocess.py
type: DataDownload
encoding format: text/x-python
content size: 6460

cornell_preprocess.py
type: DataDownload
encoding format: text/x-python
content size: 8548

env.yml
type: DataDownload
encoding format: application/octet-stream
content size: 7745

Readme.md
type: DataDownload
encoding format: text/markdown
content size: 2000

ubuntu_preprocess.py
type: DataDownload
encoding format: text/x-python
content size: 8853

convai1.json
type: DataDownload
encoding format: application/json
content size: 5804611

conversation_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 4320

sentences.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 2542718

sentence_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 52383

convai2.json
type: DataDownload
encoding format: application/json
content size: 6636788

conversation_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 4954

sentences.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 2943221

sentence_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 60871

chameleons.pdf
type: DataDownload
encoding format: application/pdf
content size: 290691

id2word.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 299497

movie_characters_metadata.txt
type: DataDownload
encoding format: text/plain
content size: 705695

movie_conversations.txt
type: DataDownload
encoding format: text/plain
content size: 6760930

movie_lines.txt
type: DataDownload
encoding format: text/plain
content size: 34641919

movie_titles_metadata.txt
type: DataDownload
encoding format: text/plain
content size: 67289

raw_script_urls.txt
type: DataDownload
encoding format: text/plain
content size: 56177

README.txt
type: DataDownload
encoding format: text/plain
content size: 4182

word2id.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 299497

conversation_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 16644

sentences.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 5250859

sentence_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 123603

conversation_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 133094

sentences.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 42099789

sentence_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 997403

conversation_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 16644

sentences.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 5279906

sentence_length.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 124157

configs.py
type: DataDownload
encoding format: text/x-python
content size: 6870

convai1_results.pickle.bz2
type: DataDownload
encoding format: application/x-bzip2
content size: 13323619

convai2_results.pickle.bz2
type: DataDownload
encoding format: application/x-bzip2
content size: 21374451

correlations.ipynb
type: DataDownload
encoding format: application/x-ipynb+json
content size: 71458

data_loader.py
type: DataDownload
encoding format: text/x-python
content size: 2743

eval.py
type: DataDownload
encoding format: text/x-python
content size: 1168

eval_4models_convai.sh
type: DataDownload
encoding format: application/x-sh
content size: 1179

eval_convai.py
type: DataDownload
encoding format: text/x-python
content size: 1688

eval_embed.py
type: DataDownload
encoding format: text/x-python
content size: 1138

gpt_convai_eval.py
type: DataDownload
encoding format: text/x-python
content size: 7666

solver.py
type: DataDownload
encoding format: text/x-python
content size: 38317

sqlitedict_compress.py
type: DataDownload
encoding format: text/x-python
content size: 223

train.py
type: DataDownload
encoding format: text/x-python
content size: 1665

train_4models.sh
type: DataDownload
encoding format: application/x-sh
content size: 210

__init__.py
type: DataDownload
encoding format: text/x-python
content size: 0

30.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 186459282

config.txt
type: DataDownload
encoding format: text/plain
content size: 2077

convai1.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 868352

convai2.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 1146880

events.out.tfevents.1604760827.ff8fc1a53fe2
type: DataDownload
encoding format: application/octet-stream
content size: 3070

samples.txt
type: DataDownload
encoding format: text/plain
content size: 1017054

30.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 128405303

config.txt
type: DataDownload
encoding format: text/plain
content size: 2086

convai1.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 860160

convai2.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 1134592

events.out.tfevents.1604756310.ff8fc1a53fe2
type: DataDownload
encoding format: application/octet-stream
content size: 3070

samples.txt
type: DataDownload
encoding format: text/plain
content size: 986246

30.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 296152508

config.txt
type: DataDownload
encoding format: text/plain
content size: 2077

convai1.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 856064

convai2.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 1130496

events.out.tfevents.1604773321.ff8fc1a53fe2
type: DataDownload
encoding format: application/octet-stream
content size: 7540

samples.txt
type: DataDownload
encoding format: text/plain
content size: 1095517

30.pkl
type: DataDownload
encoding format: application/octet-stream
content size: 212482594

config.txt
type: DataDownload
encoding format: text/plain
content size: 2080

convai1.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 856064

convai2.sqlite
type: DataDownload
encoding format: application/octet-stream
content size: 1134592

events.out.tfevents.1604766709.ff8fc1a53fe2
type: DataDownload
encoding format: application/octet-stream
content size: 7540

samples.txt
type: DataDownload
encoding format: text/plain
content size: 1094130

ae_mapper.py
type: DataDownload
encoding format: text/x-python
content size: 6015

hred.py
type: DataDownload
encoding format: text/x-python
content size: 7938

seq2seq.py
type: DataDownload
encoding format: text/x-python
content size: 3668

vhcr.py
type: DataDownload
encoding format: text/x-python
content size: 19884

vhred.py
type: DataDownload
encoding format: text/x-python
content size: 12544

__init__.py
type: DataDownload
encoding format: text/x-python
content size: 259

beam_search.py
type: DataDownload
encoding format: text/x-python
content size: 6036

decoder.py
type: DataDownload
encoding format: text/x-python
content size: 12223

encoder.py
type: DataDownload
encoding format: text/x-python
content size: 8097

feedforward.py
type: DataDownload
encoding format: text/x-python
content size: 886

loss.py
type: DataDownload
encoding format: text/x-python
content size: 1680

rnncells.py
type: DataDownload
encoding format: text/x-python
content size: 2526

__init__.py
type: DataDownload
encoding format: text/x-python
content size: 147

bow.py
type: DataDownload
encoding format: text/x-python
content size: 1102

convert.py
type: DataDownload
encoding format: text/x-python
content size: 1922

embedding_metric.py
type: DataDownload
encoding format: text/x-python
content size: 1774

mask.py
type: DataDownload
encoding format: text/x-python
content size: 1207

pad.py
type: DataDownload
encoding format: text/x-python
content size: 953

probability.py
type: DataDownload
encoding format: text/x-python
content size: 860

tensorboard.py
type: DataDownload
encoding format: text/x-python
content size: 895

time_track.py
type: DataDownload
encoding format: text/x-python
content size: 1132

tokenizer.py
type: DataDownload
encoding format: text/x-python
content size: 2127

vocab.py
type: DataDownload
encoding format: text/x-python
content size: 4795

__init__.py
type: DataDownload
encoding format: text/x-python
content size: 273

Nedelchev, Rostislav
type: Person
given name: Rostislav
family name: Nedelchev

bonndata
type: DataCatalog

bonndata
type: Organization

bonndata
type: Organization