Evaluating Dialogue Systems via an Opinion (doi:10.60507/FK2/FX37GD)

View:

Part 1: Document Description
Part 2: Study Description
Part 5: Other Study-Related Materials
Entire Codebook

Evaluating Dialogue Systems via an Opinion

doi:10.60507/FK2/FX37GD

bonndata

2023-06-05

1

Nedelchev, Rostislav, 2023, "Evaluating Dialogue Systems via an Opinion", https://doi.org/10.60507/FK2/FX37GD, bonndata, V1

Evaluating Dialogue Systems via an Opinion

doi:10.60507/FK2/FX37GD

Nedelchev, Rostislav

Python

virtual environment like (Ana|Mini)conda

bonndata

Nedelchev, Rostislav

Nedelchev, Rostislav

2023-05-20

https://doi.org/10.60507/FK2/FX37GD

Computer and Information Science

Dialogue systems are a significant field of research and development in artificial intelligence. Until today, the evaluation of such algorithms happens in one fundamental way. They solve "hypothetical problems," i.e., dialog systems are tested by being asked to respond in specific scenarios and provide a "solution", i.e. a reply, to a "problem". The replies are either explicitly compared to reference responses using overlap-based metrics (e.g., BLEU) or are evaluated by human annotators, which can also be seen as compared to (implicit) references. Instead, we propose to ask dialogue systems to tell whether a sample "solution" to a sample "problem" is good or bad. In other words, we ask another dialog system whether a conversation is fluent and coherent or not, and to what degree. In our experiments, we show how to evaluate dialogue systems by "asking for an opinion" and that it indeed offers an additional perspective on assessing these methods.

Nedelchev, Rostislav: Automatic Evaluation of Dialogue-Systems Using Neural-Network Methods. - Bonn, 2023. - Dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn. Online-Ausgabe in bonndoc: https://nbn-resolving.org/urn:nbn:de:hbz:5-70983

urn:nbn:de:hbz:5-70983

Nedelchev, Rostislav: Automatic Evaluation of Dialogue-Systems Using Neural-Network Methods. - Bonn, 2023. - Dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn. Online-Ausgabe in bonndoc: https://nbn-resolving.org/urn:nbn:de:hbz:5-70983

Parts of the code are based on the following work: Yookoon Park and Jaemin Cho and Gunhee Kim "A Hierarchical Latent Structure for Variational Conversation Modeling", 2018, NAACL, DOI: http://dx.doi.org/10.18653/v1/N18-1162

Original code is located here: https://github.com/ctr4si/A-Hierarchical-Latent-Structure-for-Variational-Conversation-Modeling

Label:

convai1_preprocess.py

text/x-python

Label:

convai2_preprocess.py

text/x-python

Label:

cornell_preprocess.py

text/x-python

Label:

env.yml

application/octet-stream

Label:

Readme.md

text/markdown

Label:

ubuntu_preprocess.py

text/x-python

Label:

convai1.json

application/json

Label:

conversation_length.pkl

application/octet-stream

Label:

sentences.pkl

application/octet-stream

Label:

sentence_length.pkl

application/octet-stream

Label:

convai2.json

application/json

Label:

conversation_length.pkl

application/octet-stream

Label:

sentences.pkl

application/octet-stream

Label:

sentence_length.pkl

application/octet-stream

Label:

chameleons.pdf

application/pdf

Label:

id2word.pkl

application/octet-stream

Label:

movie_characters_metadata.txt

text/plain

Label:

movie_conversations.txt

text/plain

Label:

movie_lines.txt

text/plain

Label:

movie_titles_metadata.txt

text/plain

Label:

raw_script_urls.txt

text/plain

Label:

README.txt

text/plain

Label:

word2id.pkl

application/octet-stream

Label:

conversation_length.pkl

application/octet-stream

Label:

sentences.pkl

application/octet-stream

Label:

sentence_length.pkl

application/octet-stream

Label:

conversation_length.pkl

application/octet-stream

Label:

sentences.pkl

application/octet-stream

Label:

sentence_length.pkl

application/octet-stream

Label:

conversation_length.pkl

application/octet-stream

Label:

sentences.pkl

application/octet-stream

Label:

sentence_length.pkl

application/octet-stream

Label:

configs.py

text/x-python

Label:

convai1_results.pickle.bz2

application/x-bzip2

Label:

convai2_results.pickle.bz2

application/x-bzip2

Label:

correlations.ipynb

application/x-ipynb+json

Label:

data_loader.py

text/x-python

Label:

eval.py

text/x-python

Label:

eval_4models_convai.sh

application/x-sh

Label:

eval_convai.py

text/x-python

Label:

eval_embed.py

text/x-python

Label:

gpt_convai_eval.py

text/x-python

Label:

solver.py

text/x-python

Label:

sqlitedict_compress.py

text/x-python

Label:

train.py

text/x-python

Label:

train_4models.sh

application/x-sh

Label:

__init__.py

text/x-python

Label:

30.pkl

application/octet-stream

Label:

config.txt

text/plain

Label:

convai1.sqlite

application/octet-stream

Label:

convai2.sqlite

application/octet-stream

Label:

events.out.tfevents.1604760827.ff8fc1a53fe2

application/octet-stream

Label:

samples.txt

text/plain

Label:

30.pkl

application/octet-stream

Label:

config.txt

text/plain

Label:

convai1.sqlite

application/octet-stream

Label:

convai2.sqlite

application/octet-stream

Label:

events.out.tfevents.1604756310.ff8fc1a53fe2

application/octet-stream

Label:

samples.txt

text/plain

Label:

30.pkl

application/octet-stream

Label:

config.txt

text/plain

Label:

convai1.sqlite

application/octet-stream

Label:

convai2.sqlite

application/octet-stream

Label:

events.out.tfevents.1604773321.ff8fc1a53fe2

application/octet-stream

Label:

samples.txt

text/plain

Label:

30.pkl

application/octet-stream

Label:

config.txt

text/plain

Label:

convai1.sqlite

application/octet-stream

Label:

convai2.sqlite

application/octet-stream

Label:

events.out.tfevents.1604766709.ff8fc1a53fe2

application/octet-stream

Label:

samples.txt

text/plain

Label:

ae_mapper.py

text/x-python

Label:

hred.py

text/x-python

Label:

seq2seq.py

text/x-python

Label:

vhcr.py

text/x-python

Label:

vhred.py

text/x-python

Label:

__init__.py

text/x-python

Label:

beam_search.py

text/x-python

Label:

decoder.py

text/x-python

Label:

encoder.py

text/x-python

Label:

feedforward.py

text/x-python

Label:

loss.py

text/x-python

Label:

rnncells.py

text/x-python

Label:

__init__.py

text/x-python

Label:

bow.py

text/x-python

Label:

convert.py

text/x-python

Label:

embedding_metric.py

text/x-python

Label:

mask.py

text/x-python

Label:

pad.py

text/x-python

Label:

probability.py

text/x-python

Label:

tensorboard.py

text/x-python

Label:

time_track.py

text/x-python

Label:

tokenizer.py

text/x-python

Label:

vocab.py

text/x-python

Label:

__init__.py

text/x-python