One-sentence Summary

Researchers at Heriot-Watt University and Microsoft Research present a hybrid generative-retrieval model built on $\text{GPT-2}$GPT-2, fine-tuned on MetaLWOz, that won the DSTC-8 fast domain adaptation task by using retrieval as a fallback for robust response generation, achieving state-of-the-art human-evaluation performance ($>4\%$>4% over second place on MetaLWOz) and competitive generalization to unseen MultiWOZ data.

Key Contributions

• Presents a hybrid generative-retrieval model built on GPT-2, fine-tuned on the multi-domain MetaLWOz dataset, which won the DSTC-8 Fast Domain Adaptation task.
• Combines robust and diverse generative response generation with a retrieval-based fallback for low-confidence cases, enabling effective transfer learning for few-shot goal-oriented dialogue adaptation.
• Achieves state-of-the-art performance on MetaLWOz in human evaluation, improving by over 4% over the second-place system, and shows competitive generalization to the unseen MultiWOZ dataset without prior exposure during main training.

Introduction

Goal-oriented dialogue systems face the challenge of adapting quickly to new domains with limited data, a critical requirement for industry deployment where labeled conversational data is scarce. Prior transfer learning approaches, while forming the basis of state-of-the-art domain adaptation methods, still fall short of performance levels that would justify direct adoption in production settings. The authors present a hybrid generative and retrieval approach that combines a generative model with retrieval logic as a fallback mechanism for low confidence cases. Their method achieves fast domain adaptation through transfer learning and won the DSTC-8 Fast Domain Adaptation task, attaining state-of-the-art performance in human evaluation. It also shows competitive generalization on the MultiWOZ dataset despite not being exposed to that data during main training. The authors acknowledge that data-efficient dialogue response generation remains an open problem and propose exploring meta-learning, or "learning to fine-tune," as a promising future direction to improve fine-tuning performance across multiple domains.

Experiment

In pairwise judging comparisons, the gold response ranked first with the highest win rate, while the submission followed in second place. The remaining systems trailed in descending order, with the baseline and another team performing worst. the submission placed second overall, narrowly behind the gold response. The gold response clearly outperformed all other systems, while the lowest-ranked team had a win rate more than 20 points below the leader.

GPT-2 based models outperform retrieval-only and HRED baselines across both pure and cross task settings. The GPT-2 hybrid consistently achieves the highest scores on the pure task, while GPT-2 with retrieval logic but without hybrid also performs well, particularly on cross task ROUGE-L. Retrieval-based methods lag behind generative models, especially on cross task. GPT-2 hybrid leads on pure task metrics, while retrieval-based methods are significantly lower. On cross task, GPT-2 with retrieval logic matches or exceeds hybrid on some metrics, but both surpass HRED and retrieval baselines. GPT-2 without support set performs worse than with support or hybrid, indicating the benefit of retrieval support.

On the MultiWOZ pure task benchmark, generative approaches generally outperform retrieval-based baselines on both intent and slot metrics. The GPT-2 supervised variant achieves the best combined intent and slot F1, while Team C records the highest standalone intent F1. HRED shows a notably lower intent score but a comparatively stronger combined score, suggesting a different error profile. GPT-2-sup leads in combined intent and slot F1, while Team C achieves the highest intent F1. Retrieval-based models (BERT and SP+FT) trail behind generative models on both metrics. HRED exhibits a large gap between intent and combined F1, indicating more slot-related errors.

The table shows example dialogue turns from a GPT-2 Hybrid system, contrasting expected gold responses with the model's predicted responses. In the first example, the model asks a factual historical question rather than following up on the user's stated interest in history. The second example illustrates a travel-related context where the wizard asks for departure details. In response to a user's stated interest in history, the model asks about Rome's founder instead of asking where the user would like to go. The model's predicted response shifts the conversation to a factual question, while the gold response aims to elicit more preferences from the user. The second context shows a train travel request, with the wizard prompting for departure location and time.

Across all MetaLWOz domains, the GPT-2-hybrid model generates responses more often than it retrieves them, with generated proportions consistently exceeding 57%. The preference for generation is most pronounced in the cross-task booking flight domain and least pronounced in tourism. Generated responses outnumber retrieved responses in every dataset and domain. The highest generation share occurs in cross-task booking flight at 68.2%, while tourism has the lowest at 57.4%. Pure task domains show generation rates between 57.4% and 64.1%, indicating a consistent but varying reliance on generation over retrieval.

The evaluation showed the proposed system ranking second overall, narrowly behind the gold response in pairwise comparisons, while the gold response clearly outperformed all other systems. Across benchmark tasks, GPT-2 based generative models consistently surpassed retrieval-only and HRED baselines in both pure and cross-task settings, with the hybrid variant leading on pure task metrics and retrieval support improving generation quality. On the MultiWOZ pure task, generative approaches beat retrieval baselines on intent and slot accuracy, with GPT-2 supervised achieving the best combined score, while HRED exhibited more slot-related errors than intent errors. Qualitative analysis revealed that the hybrid system sometimes shifted conversation toward factual questions instead of eliciting user preferences, and it favored generation over retrieval across all MetaLWOz domains, with generation rates ranging from 57% to 68%.