Evaluating Dialogue Systems via an Opiniondoi:10.60507/FK2/FX37GDbonndata2023-06-051Nedelchev, Rostislav, 2023, "Evaluating Dialogue Systems via an Opinion", https://doi.org/10.60507/FK2/FX37GD, bonndata, V1Evaluating Dialogue Systems via an Opiniondoi:10.60507/FK2/FX37GDNedelchev, RostislavPythonvirtual environment like (Ana|Mini)condabonndataNedelchev, RostislavNedelchev, Rostislav2023-05-20Computer and Information ScienceDialogue systems are a significant field of research and development in artificial intelligence. Until today, the evaluation of such algorithms happens in one fundamental way. They solve "hypothetical problems," i.e., dialog systems are tested by being asked to respond in specific scenarios and provide a "solution", i.e. a reply, to a "problem". The replies are either explicitly compared to reference responses using overlap-based metrics (e.g., BLEU) or are evaluated by human annotators, which can also be seen as compared to (implicit) references. Instead, we propose to ask dialogue systems to tell whether a sample "solution" to a sample "problem" is good or bad. In other words, we ask another dialog system whether a conversation is fluent and coherent or not, and to what degree. In our experiments, we show how to evaluate dialogue systems by "asking for an opinion" and that it indeed offers an additional perspective on assessing these methods.<a href="http://creativecommons.org/publicdomain/zero/1.0">CC0 1.0</a>Nedelchev, Rostislav: Automatic Evaluation of Dialogue-Systems Using Neural-Network Methods. - Bonn, 2023. - Dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn. Online-Ausgabe in bonndoc: https://nbn-resolving.org/urn:nbn:de:hbz:5-70983urn:nbn:de:hbz:5-70983Nedelchev, Rostislav: Automatic Evaluation of Dialogue-Systems Using Neural-Network Methods. - Bonn, 2023. - Dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn. Online-Ausgabe in bonndoc: https://nbn-resolving.org/urn:nbn:de:hbz:5-70983Parts of the code are based on the following work: Yookoon Park and Jaemin Cho and Gunhee Kim "A Hierarchical Latent Structure for Variational Conversation Modeling", 2018, NAACL, DOI: http://dx.doi.org/10.18653/v1/N18-1162Original code is located here: https://github.com/ctr4si/A-Hierarchical-Latent-Structure-for-Variational-Conversation-Modelingconvai1_preprocess.pytext/x-pythonconvai2_preprocess.pytext/x-pythoncornell_preprocess.pytext/x-pythonenv.ymlapplication/octet-streamReadme.mdtext/markdownubuntu_preprocess.pytext/x-pythonconvai1.jsonapplication/jsonconversation_length.pklapplication/octet-streamsentences.pklapplication/octet-streamsentence_length.pklapplication/octet-streamconvai2.jsonapplication/jsonconversation_length.pklapplication/octet-streamsentences.pklapplication/octet-streamsentence_length.pklapplication/octet-streamchameleons.pdfapplication/pdfid2word.pklapplication/octet-streammovie_characters_metadata.txttext/plainmovie_conversations.txttext/plainmovie_lines.txttext/plainmovie_titles_metadata.txttext/plainraw_script_urls.txttext/plainREADME.txttext/plainword2id.pklapplication/octet-streamconversation_length.pklapplication/octet-streamsentences.pklapplication/octet-streamsentence_length.pklapplication/octet-streamconversation_length.pklapplication/octet-streamsentences.pklapplication/octet-streamsentence_length.pklapplication/octet-streamconversation_length.pklapplication/octet-streamsentences.pklapplication/octet-streamsentence_length.pklapplication/octet-streamconfigs.pytext/x-pythonconvai1_results.pickle.bz2application/x-bzip2convai2_results.pickle.bz2application/x-bzip2correlations.ipynbapplication/x-ipynb+jsondata_loader.pytext/x-pythoneval.pytext/x-pythoneval_4models_convai.shapplication/x-sheval_convai.pytext/x-pythoneval_embed.pytext/x-pythongpt_convai_eval.pytext/x-pythonsolver.pytext/x-pythonsqlitedict_compress.pytext/x-pythontrain.pytext/x-pythontrain_4models.shapplication/x-sh__init__.pytext/x-python30.pklapplication/octet-streamconfig.txttext/plainconvai1.sqliteapplication/octet-streamconvai2.sqliteapplication/octet-streamevents.out.tfevents.1604760827.ff8fc1a53fe2application/octet-streamsamples.txttext/plain30.pklapplication/octet-streamconfig.txttext/plainconvai1.sqliteapplication/octet-streamconvai2.sqliteapplication/octet-streamevents.out.tfevents.1604756310.ff8fc1a53fe2application/octet-streamsamples.txttext/plain30.pklapplication/octet-streamconfig.txttext/plainconvai1.sqliteapplication/octet-streamconvai2.sqliteapplication/octet-streamevents.out.tfevents.1604773321.ff8fc1a53fe2application/octet-streamsamples.txttext/plain30.pklapplication/octet-streamconfig.txttext/plainconvai1.sqliteapplication/octet-streamconvai2.sqliteapplication/octet-streamevents.out.tfevents.1604766709.ff8fc1a53fe2application/octet-streamsamples.txttext/plainae_mapper.pytext/x-pythonhred.pytext/x-pythonseq2seq.pytext/x-pythonvhcr.pytext/x-pythonvhred.pytext/x-python__init__.pytext/x-pythonbeam_search.pytext/x-pythondecoder.pytext/x-pythonencoder.pytext/x-pythonfeedforward.pytext/x-pythonloss.pytext/x-pythonrnncells.pytext/x-python__init__.pytext/x-pythonbow.pytext/x-pythonconvert.pytext/x-pythonembedding_metric.pytext/x-pythonmask.pytext/x-pythonpad.pytext/x-pythonprobability.pytext/x-pythontensorboard.pytext/x-pythontime_track.pytext/x-pythontokenizer.pytext/x-pythonvocab.pytext/x-python__init__.pytext/x-python