Publications
* Corresponding author
† Equal contribution
Preprint
!!!!!A Survey of Large Language Models
Wayne Xin Zhao,
Kun Zhou†,
Junyi Li†,
Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu,
Jian-Yun Nie,
Ji-Rong Wen
pdf / code
Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge
to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely
studied for language understanding and generation in the past two decades, evolving from statistical language models to neural
language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale
corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to
performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly,
when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance
improvement but also show some special abilities that are not present in small-scale language models. To discriminate the
difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant
size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the
launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an
important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this
survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular,
we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides,
we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.
Large reasoning models (LRMs) have demonstrated strong performance on complex reasoning tasks, but often suffer from overthinking, generating
redundant content regardless of task difficulty. Inspired by the dual process theory in cognitive science, we propose Adaptive Cognition
Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reasoning through adaptive cognitive
allocation and dynamic system switch. ACPO incorporates two key components: (1) introducing system-aware reasoning tokens to explicitly
represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and
token length budget to guide adaptive system switch and reasoning during reinforcement learning. To this end, we propose a two-stage training
strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit
thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning. Experimental
results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity,
achieving efficient hybrid reasoning.
Recent advances in web-augmented large language models (LLMs) have exhibited strong performance in complex reasoning tasks, yet these capabilities
are mostly locked in proprietary systems with opaque architectures. In this work, we propose ManuSearch, a transparent and modular multi-agent
framework designed to democratize deep search for LLMs. ManuSearch decomposes the search and reasoning process into three collaborative agents:
(1) a solution planning agent that iteratively formulates sub-queries, (2) an Internet search agent that retrieves relevant documents via real-time
web search, and (3) a structured webpage reading agent that extracts key evidence from raw web content. To rigorously evaluate deep reasoning abilities,
we introduce ORION, a challenging benchmark focused on open-web reasoning over long-tail entities, covering both English and Chinese. Experimental
results show that ManuSearch substantially outperforms prior open-source baselines and even surpasses leading closed-source systems. Our work paves
the way for reproducible, extensible research in open deep search systems.
Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive
capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning
significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient,
entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose
Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each
reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual
correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models
demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.
2025
Despite their outstanding capabilities, large language models (LLMs) are prone to hallucination and producing factually incorrect information.
This challenge has spurred efforts in attributed text generation, which prompts LLMs to generate content with supporting evidence.
In this paper, we propose a novel framework, called Think&Cite, and formulate attributed text generation as a multi-step reasoning problem
integrated with search. Specifically, we propose Self-Guided Monte Carlo Tree Search (SG-MCTS), which capitalizes on the self-reflection
capability of LLMs to reflect on the intermediate states of MCTS for guiding the tree expansion process. To provide reliable and comprehensive
feedback, we introduce Progress Reward Models to measure the progress of tree search from the root to the current state from two aspects,
i.e., generation and attribution progress. We conduct extensive experiments on three datasets and the results show that our approach significantly
outperforms baseline approaches.
Large language models (LLMs) have gained extended context windows through scaling positional encodings and lightweight continual pre-training.
However, this often leads to degraded performance on short-text tasks, while the reasons for this degradation remain insufficiently explored.
In this work, we identify two primary factors contributing to this issue: distribution drift in hidden states and attention scores, and
catastrophic forgetting during continual pre-training. To address these challenges, we propose Long Context Pre-training with Restoration
Distillation (LongReD), a novel approach designed to mitigate short-text performance degradation through minimizing the distribution discrepancy
between the extended and original models. Besides training on long texts, LongReD distills the hidden state of selected layers from the original
model on short texts. Additionally, LongReD also introduces a short-to-long distillation, aligning the output distribution on short texts with
that on long texts by leveraging skipped positional indices. Experiments on common text benchmarks demonstrate that LongReD effectively preserves
the model's short-text performance while maintaining comparable or even better capacity to handle long texts than baselines.
Large language models (LLMs) demonstrate exceptional capabilities, yet still face the hallucination issue. Typical text generation approaches adopt an
auto-regressive generation without deliberate reasoning, which often results in untrustworthy and factually inaccurate responses. In this paper,
we propose HaluSearch, a novel framework that incorporates tree search-based algorithms (e.g. MCTS) to enable an explicit slow thinking generation
process for mitigating hallucinations of LLMs during inference. Specifically, HaluSearch frames text generation as a step-by-step reasoning process,
using a self-evaluation reward model to score each generation step and guide the tree search towards the most reliable generation pathway for fully
exploiting the internal knowledge of LLMs. To balance efficiency and quality, we introduce a hierarchical thinking system switch mechanism inspired
by the dual process theory in cognitive science, which dynamically alternates between fast and slow thinking modes at both the instance and step levels,
adapting to the complexity of questions and reasoning states. We conduct extensive experiments on both English and Chinese datasets and the results
show that our approach significantly outperforms baseline approaches.
Large language models (LLMs) have demonstrated impressive translation capabilities even without being explicitly trained on parallel data. This
remarkable property has led some to believe that parallel data is no longer necessary for building multilingual language models. While some
attribute this to the emergent abilities of LLMs due to scale, recent work suggests that it is actually caused by incidental bilingual signals
present in the training data. Various methods have been proposed to maximize the utility of parallel data to enhance the multilingual capabilities
of multilingual encoder-based and encoder-decoder language models. However, some decoder-based LLMs opt to ignore parallel data instead. In this
work, we conduct a systematic study on the impact of adding parallel data on LLMs' multilingual capabilities, focusing specifically on translation
and multilingual common-sense reasoning. Through controlled experiments, we demonstrate that parallel data can significantly improve LLMs' multilingual
capabilities.
The rapidly changing nature of real-world information presents challenges for large language models (LLMs), which are typically trained on static
datasets. This limitation makes it difficult for LLMs to accurately perform tasks that require up-to-date knowledge, such as time-sensitive
question answering (QA). In this paper, we introduce DynaQuest, a Dynamic Question answering dataset reflecting knowledge updates in the real
world. DynaQuest is based on Wikipedia Infoboxes, which are frequently updated to reflect real-world changes. Our dataset is created by
automatically identifying and comparing changes between different versions of Wikipedia pages and generating question-answer pairs based on
these updates. To address the challenges posed by our dynamic dataset, we propose CARL, a Context-Aware Reinforcement Learning framework to
improve the performance of LLMs on time-sensitive question answering. We conduct experiments on our collected dataset across recent time periods
and demonstrate the effectiveness of our approach. Furthermore, we maintain a dynamic knowledge updating process, providing a periodically evolving
benchmark to continually evaluate LLMs' ability to answer time-sensitive questions.
Conversational recommendation systems (CRSs) use multi-turn interaction to capture user preferences and provide personalized recommendations.
A fundamental challenge in CRSs lies in effectively understanding user preferences from conversations. User preferences can be multifaceted
and complex, posing significant challenges for accurate recommendations even with access to abundant external knowledge. While interaction
with users can clarify their true preferences, frequent user involvement can lead to a degraded user experience. To address this problem,
we propose a generative reward model based simulated user, named GRSU, for automatic interaction with CRSs. The simulated user provides
feedback to the items recommended by CRSs, enabling them to better capture intricate user preferences through multi-turn interaction. Inspired
by generative reward models, we design two types of feedback actions for the simulated user: i.e., generative item scoring, which offers
coarse-grained feedback, and attribute-based item critique, which provides fine-grained feedback. To ensure seamless integration, these
feedback actions are unified into an instruction-based format, allowing the development of a unified simulated user via instruction tuning
on synthesized data. With this simulated user, automatic multi-turn interaction with CRSs can be effectively conducted. Furthermore, to
strike a balance between effectiveness and efficiency, we draw inspiration from the paradigm of reward-guided search in complex reasoning
tasks and employ beam search for the interaction process. On top of this, we propose an efficient candidate ranking method to improve the
recommendation results derived from interaction. Extensive experiments on public datasets demonstrate the effectiveness, efficiency, and
transferability of our approach.
In the era of vast digital information, the sheer volume and heterogeneity of available information present significant challenges for intricate information
seeking. Users frequently face multistep web search tasks that involve navigating vast and varied data sources. This complexity demands every step
remains comprehensive, accurate, and relevant. However, traditional search methods often struggle to balance the need for localized precision with
the broader context required for holistic understanding, leaving critical facets of intricate queries underexplored. In this paper, we introduce an
LLM-based search assistant that adopts a new information seeking paradigm with holistically guided Monte Carlo tree search (HG-MCTS). We reformulate
the task as a progressive information collection process with a knowledge memory and unite an adaptive checklist with multi-perspective reward modeling
in MCTS. The adaptive checklist provides explicit sub-goals to guide the MCTS process toward comprehensive coverage of complex user queries.
Simultaneously, our multi-perspective reward modeling offers both exploration and retrieval rewards, along with progress feedback that tracks completed
and remaining sub-goals, refining the checklist as the tree search progresses. By striking a balance between localized tree expansion and global guidance,
HG-MCTS reduces redundancy in search paths and ensures that all crucial aspects of an intricate query are properly addressed. Extensive experiments on
real-world intricate information seeking tasks demonstrate that HG-MCTS acquires thorough knowledge collections and delivers more accurate final responses
compared with existing baselines.
Adapting general large language models (LLMs) to specialized domains presents great challenges due to varied data distributions. This adaptation
typically requires continual pre-training on massive domain-specific corpora to facilitate knowledge memorization, followed by training to
apply this knowledge following human instructions and preferences. However, this method may result in inefficient knowledge memorization due
to a lack of awareness of knowledge utilization and imposes substantial demands on LLMs to simultaneously learn knowledge utilization and
format alignment with limited training samples. To facilitate the domain adaptation of LLM, we revise this process and propose a new domain
adaptation framework including domain knowledge learning and general format alignment, called Mix-CPT. Specifically, we first conduct a
knowledge mixture continual pre-training that concurrently focuses on knowledge memorization and utilization, allowing for mutual reinforcement.
To avoid catastrophic forgetting during the continual pre-training process, we further incorporate a logit swap self-distillation constraint.
Subsequently, leveraging the knowledge and capabilities acquired during continual pre-training, we efficiently perform instruction tuning and
alignment with a few general training samples to achieve format alignment. Extensive experiments demonstrate that our proposed Mix-CPT framework
can simultaneously improve the task-solving capabilities of LLMs on the target and general domains compared to the traditional adaptation methods.
The emergence of in-context learning (ICL) is potentially attributed to two major abilities: task recognition (TR) for recognizing the
task from demonstrations and utilizing pre-trained priors, and task learning (TL) for learning from demonstrations. However,
relationships between the two abilities and how such relationships affect the emergence of ICL is unclear. In this paper, we take
the first step by examining the pre-training dynamics of the emergence of ICL. With carefully designed metrics, we find that these
two abilities are, in fact, competitive during pre-training. Moreover, we observe a negative correlation between the competition and
the performance of ICL. Further analysis of common pre-training factors (i.e., model size, dataset size, and data curriculum) demonstrates
possible ways to regulate the competition. Based on these insights, we propose a simple yet effective method to better integrate these
two abilities for ICL at inference time. Through adaptive ensemble learning, the performance of ICL can be significantly boosted,
enabling two small models to outperform a larger one with more than twice the parameters.
Existing large language models (LLMs) show exceptional problem-solving capabilities but might struggle with complex reasoning tasks. Despite
the successes of chain-of-thought and tree-based search methods, they mainly depend on the internal knowledge of LLMs to search over
intermediate reasoning steps, limited to dealing with simple tasks involving fewer reasoning steps. In this paper, we propose RAG-Star,
a novel RAG approach that integrates the retrieved information to guide the tree-based deliberative reasoning process that relies on the
inherent knowledge of LLMs. By leveraging Monte Carlo Tree Search, RAG-Star iteratively plans intermediate sub-queries and answers for
reasoning based on the LLM itself. To consolidate internal and external knowledge, we propose an retrieval-augmented verification that
utilizes query- and answer-aware reward modeling to provide feedback for the inherent reasoning of LLMs. Our experiments involving
Llama-3.1-8B-Instruct and GPT-4o demonstrate that RAG-Star significantly outperforms previous RAG and reasoning methods.
2024
Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing
text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of
LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond
the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors
from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the
change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free
context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can
effectively extend the context window length.
Hallucination detection is a challenging task for large language models (LLMs), and existing studies heavily rely on powerful closed-source LLMs such as
GPT-4. In this paper, we propose an autonomous LLM-based agent framework, called HaluAgent, which enables relatively smaller LLMs (e.g. Baichuan2-Chat 7B)
to actively select suitable tools for detecting multiple hallucination types such as text, code, and mathematical expression. In HaluAgent, we integrate
the LLM, multi-functional toolbox, and design a fine-grained three-stage detection framework along with memory mechanism. To facilitate the effectiveness
of HaluAgent, we leverage existing Chinese and English datasets to synthesize detection trajectories for fine-tuning, which endows HaluAgent with the
capability for bilingual hallucination detection. Extensive experiments demonstrate that only using 2K samples for tuning LLMs, HaluAgent can perform
hallucination detection on various types of tasks and datasets, achieving performance comparable to or even higher than GPT-4 without tool enhancements
on both in-domain and out-of-domain datasets.
Considering the limited internal parametric knowledge, retrieval-augmented generation (RAG) has been widely used to extend the knowledge scope of large
language models (LLMs). Despite the extensive efforts on RAG research, in existing methods, LLMs cannot precisely assess the relevance of retrieved
documents, thus likely leading to misleading or even incorrect utilization of external knowledge (i.e., retrieved documents). To address this issue,
in this paper, we propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA). As the key motivation, we aim
to enhance the self-awareness of source relevance for LLMs, so as to adaptively utilize external knowledge in RAG systems. Specially, we develop a
new architecture for LLM based RAG system, by incorporating a specially designed rank head that precisely assesses the relevance of retrieved documents.
Furthermore, we propose an improved training method based on bi-granularity relevance fusion and noise-resistant training. By combining the improvements
in both architecture and training, our proposed REAR can better utilize external knowledge by effectively perceiving the relevance of retrieved documents.
Experiments on four open-domain QA tasks show that REAR significantly outperforms previous a number of competitive RAG approaches.
LLMBox: A Comprehensive Library for Large Language Models
Tianyi Tang, Yiwen Hu, Bingqian Li, Wenyang Luo, Zijing Qin, Haoxiang Sun, Jiapeng Wang, Shiyi Xu, Xiaoxue Cheng, Geyang Guo, Han Peng, Bowen Zheng,
Yiru Tang, Yingqian Min, Yushuo Chen, Jie Chen, Yuanqian Zhao, Luran Ding, Yuhao Wang, Zican Dong, Chunxuan Xia, Junyi Li, Kun Zhou, Wayne Xin Zhao*, Ji-Rong Wen
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL System Demonstrations), 2024
pdf / code
To facilitate the research on large language models (LLMs), this paper presents a comprehensive and unified library, LLMBox, to ease the development,
use, and evaluation of LLMs. This library is featured with three main merits: (1) a unified data interface that supports the flexible implementation
of various training strategies, (2) a comprehensive evaluation that covers extensive tasks, datasets, and models, and (3) more practical consideration,
especially on user-friendliness and efficiency. With our library, users can easily reproduce existing methods, train new models, and conduct
comprehensive performance comparisons. To rigorously test LLMBox, we conduct extensive experiments in a diverse coverage of evaluation settings,
and experimental results demonstrate the effectiveness and efficiency of our library in supporting various implementations related to LLMs.
In the era of large language models (LLMs), hallucination (i.e., the tendency to generate factually incorrect content) poses great challenge to
trustworthy and reliable deployment of LLMs in real-world applications. To tackle the LLM hallucination, three key questions should be well
studied: how to detect hallucinations (detection), why do LLMs hallucinate (source), and what can be done to mitigate them (mitigation). To
address these challenges, this work presents a systematic empirical study on LLM hallucination, focused on the the three aspects of hallucination
detection, source and mitigation. Specially, we construct a new hallucination benchmark HaluEval 2.0, and designs a simple yet effective detection
method for LLM hallucination. Furthermore, we zoom into the different training or utilization stages of LLMs and extensively analyze the potential
factors that lead to the LLM hallucination. Finally, we implement and examine a series of widely used techniques to mitigate the hallucinations in
LLMs. Our work has led to several important findings to understand the hallucination origin and mitigate the hallucinations in LLMs.
Chain-of-Thought (CoT) prompting can largely enhance the reasoning capabilities of large language models (LLMs), establishing itself as a primary approach to solve complex reasoning tasks. Existing CoT synthesis approaches usually focus on simpler reasoning tasks and result in low-quality and inconsistent CoT prompts.
In response to this challenge, we present an empirical investigation of CoT prompting and introduce CoTGenius, a novel framework designed for the automatic generation of superior CoT prompts. CoTGenius is developed based on three major evolution strategies, \ie complicate, diversify, and specify—alongside two filtering mechanisms: evolutionary success judgement and correctness verification. We further employ CoTGenius to create an extensive CoT dataset, and subsequently fine-tune the Llama 2-Chat 7B and 13B models on this dataset. We call the resulting model ChainLM.
To deal with the cumulative error issue in reasoning steps, we propose a step-level debating method, wherein multiple debaters discuss each reasoning step to arrive at the correct answer. Extensive experiments demonstrate that our ChainLM models exhibit enhanced proficiency in addressing a spectrum of complex reasoning problems compared to existing models.
In addition, we conduct an in-depth analysis of the impact of data categories within CoTGenius on the model performance.
Large language models (LLMs) have achieved dramatic proficiency over NLP tasks with normal length. Recently, multiple studies have committed to extending the context length and enhancing the long text modeling capabilities of LLMs.
To comprehensively evaluate the long context ability of LLMs, we propose BAMBOO, a multi-task long context benchmark. BAMBOO has been designed with four principles: comprehensive capacity evaluation, avoidance of data contamination, accurate automatic evaluation, and different length levels.
It consists of 10 datasets from 5 different long text understanding tasks, \ie question answering, hallucination detection, text sorting, language modeling, and code completion, to cover various domains and core capacities of LLMs.
We conduct experiments with five widely-used long-context models and further discuss five key questions for long text research.
In the end, we discuss problems of current long-context models and point out future directions for enhancing long text modeling capacities.
Text Generation aims to produce plausible and readable text in human language from input data. The resurgence of deep learning has greatly advanced this field,
in particular, with the help of neural generation models based on pre-trained language models (PLMs). Text generation based on PLMs is viewed as
a promising approach in both academia and industry. In this paper, we provide a survey on the utilization of PLMs in text generation. We begin
with introducing two key aspects of applying PLMs to text generation: 1) how to design an effective PLM to serve as the generation model; and
2) how to effectively optimize PLMs given the reference text and to ensure that the generated texts satisfy special text properties. Then, we
show the major challenges {that have arisen} in these aspects, as well as possible solutions for them. We also include a summary of various
useful resources and typical text generation applications based on PLMs. Finally, we highlight the future research directions which will further
improve these PLMs for text generation. This comprehensive survey is intended to help researchers interested in text generation problems to learn
the core concepts, the main techniques and the latest developments in this area based on PLMs.
2023
Large language models (LLMs), such as ChatGPT, are prone to generate hallucinations, i.e., content that conflicts with
the source or cannot be verified by the factual knowledge. To understand what types of content and to which extent
LLMs are apt to hallucinate, we introduce the Hallucination Evaluation for Large Language Models (HaluEval)
benchmark, a large collection of generated and human-annotated hallucinated samples for evaluating the performance
of LLMs in recognizing hallucination. To generate these samples, we propose a ChatGPT-based two-step framework,
i.e., sampling-then-filtering. Besides, we also hire some human labelers to annotate the hallucinations in
ChatGPT responses. The empirical results suggest that ChatGPT is likely to generate hallucinated content in
specific topics by fabricating unverifiable information (i.e., about 11.4% user queries). Moreover, existing
LLMs face great challenges in recognizing the hallucinations in texts. While, our experiments also prove that
the hallucination recognition can be improved by providing external knowledge or adding reasoning steps.
People often imagine relevant scenes to aid in the writing process. In this work, we aim to utilize visual
information for composition in the same manner as humans. We propose a method, LIVE, that makes pre-trained
language models (PLMs) Learn to Imagine for Visually-augmented natural language gEneration. First, we imagine
the scene based on the text: we use a diffusion model to synthesize high-quality images conditioned on the
input and output texts. Second, we use determine whether the text can evoke the imagination in posterior.
Finally, our imagination is dynamic, and we conduct scene imagination for each sentence, rather than imagining
a scene for a entire paragraph. Technically, we propose a novel plug-and-play fusion layer to obtain
visually-augmented textual representations for each text. Our vision-text fusion layer is compatible with
Transformer-based architecture. We have conducted extensive experiments on four generation tasks using
BART and T5, and the automatic results and human evaluation demonstrate the effectiveness of our proposed method.
Pre-trained language models (PLMs) have achieved notable success in natural language generation (NLG) tasks. Up to now, most of
the PLMs are pre-trained in an unsupervised manner using large-scale general corpus. In the meanwhile, an increasing number
of models pre-trained with less labeled data showcase superior performance compared to unsupervised models. Motivated by the
success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation.
For pre-training the text generation model MVP, we collect a labeled pre-training corpus from 45 datasets over seven generation
tasks. For each task, we further pre-train specific soft prompts to stimulate the model capacity in performing a specific task.
Extensive experiments have demonstrated the effectiveness of our supervised pre-training in a number of NLG tasks, and our
general methods achieve state-of-the-art performance on 12 of 17 datasets.
In this paper, we propose a novel language model guided captioning approach, Lamoc, for knowledge-based visual
question answering~(VQA). Our approach employs the generated captions by a captioning model as the context
of an answer prediction model, which is a Pre-Trained Language model~(PLM). As the major contribution, we
leverage the guidance and feedback of the prediction model to improve the capability of the captioning model.
In this way, the captioning model can become aware of the task goal and information need from the PLM.
To develop our approach, we design two specific training stages, where the first stage adapts the captioning
model to the prediction model (selecting more suitable caption propositions for training) and the second stage
tunes the captioning model according to the task goal (learning from feedback of the PLM). Extensive experiments
demonstrate the effectiveness of the proposed approach on the knowledge-based VQA task. Specifically,
on the challenging A-OKVQA dataset, Lamoc outperforms several competitive zero-shot methods and even achieves
comparable results to a fine-tuned VLP model.
Pretrained language models (PLMs) encode a large amount of world knowledge. However, as such knowledge is frozen at the
time of model training, the models become static and limited by the training data at that time.
In order to further improve the capacity of PLMs for knowledge-intensive tasks, we consider augmenting PLMs with
the large-scale web using search engine. Unlike previous augmentation sources (\eg Wikipedia data dump), the web
provides broader, more comprehensive and constantly updated information. In this paper, we present a web-augmented
PLM -- UniWeb, which is trained over 16 knowledge-intensive tasks in a unified text-to-text format. Instead of
simply using the retrieved contents from web, our approach has made two major improvements. Firstly, we propose
an adaptive search engine assisted learning method that can self-evaluate the confidence level of PLM’s predictions,
and adaptively determine when to refer to the web for more data, which can avoid useless or noisy augmentation
from web. Secondly, we design a pretraining task, \ie continual knowledge learning, based on salient spans
prediction, to reduce the discrepancy between the encoded and retrieved knowledge. Experiments on a wide
range of knowledge-intensive tasks show that our model significantly outperforms previous retrieval-augmented methods.
2022
We consider the text generation task under the approach of pre-trained language models (PLMs). Typically, an auto-regressive (AR)
paradigm is adopted for generating texts in a token-by-token manner. Despite many advantages of AR generation, it has been
widely blamed for its inefficient inference. Therefore, non-autoregressive (NAR) models are proposed to generate all target
tokens simultaneously. However, NAR models usually generate texts of lower quality due to the absence of token dependency
in the output text. In this paper, we propose ELMER: an efficient and effective PLM for NAR text generation to explicitly
model the token dependency during NAR generation. By leveraging the early exit technique, ELMER enables the token generations
at different layers, according to their prediction confidence (a more confident token will exit at a lower layer). Besides,
we propose a novel Layer Permutation Language Modeling to pre-train ELMER by permuting the exit layer for each
token in sequences. Experiments on three text generation tasks show that ELMER significantly outperforms NAR models and
further narrows the performance gap with AR PLMs (\eg ELMER (29.92) vs BART (30.61) ROUGE-L in XSUM) while achieving
over 10x inference speedups.
TextBox 2.0: A Text Generation Library with Pre-trained Language Models
Tianyi Tang†,
Junyi Li†,
Zhipeng Chen†,
Yiwen Hu,
Zhuohao Yu,
Wenxun Dai,
Wayne Xin Zhao*,
Jian-Yun Nie,
Ji-Rong Wen
The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, System Demonstration
pdf / code
To facilitate research on text generation, this paper presents a comprehensive, unified, and standardized library, TextBox 2.0,
focusing on the use of pre-trained language models (PLMs). To be comprehensive, our library considers 13 common text
generation tasks and their corresponding 83 datasets and incorporates 36 PLMs covering general, translation, dialogue,
controllable, distilled, Chinese, and lightweight PLMs. We also implement 4 efficient training strategies and provide 4
generation objectives for pre-training new PLMs from scratch. To be unified and standardized, we carefully design the
interfaces along the research pipeline (from data loading to training and evaluation), ensuring that each step can be
conducted in a unified, standard way. Though comprehensive and powerful, the use of our library is rather simple,
either by friendly Python API or command line. Besides, we perform extensive experiments to validate the effectiveness
of our library and provide useful methods to analyze the generated results.
Recently, pretrained language models (PLMs) have made exceptional success in language generation. To leverage the rich knowledge encoded
by PLMs, a simple yet powerful mechanism is to use prompts, in the form of either discrete tokens or continuous embeddings.
In existing studies, manual prompts are time-consuming and require domain expertise, while continuous prompts are typically
independent of the inputs. To address this issue, we propose a novel continuous prompting approach, called Context-Tuning,
to fine-tuning PLMs for natural language generation. Firstly, the prompts are derived based on the input text, so that they
can elicit useful knowledge from PLMs for generation. We refer to such prompts as contextualized prompts. Secondly, to further
enhance the relevance of the generated text to the inputs, we utilize continuous inverse prompting to refine the process of
natural language generation by modeling an inverse generation process from output to input. Moreover, we propose a lightweight
context-tuning, fine-tuning only 0.4% of parameters while retaining well performance.
As transformer evolves, pre-trained models have advanced at a breakneck pace in recent years. They have dominated the mainstream
techniques in natural language processing~(NLP) and computer vision~(CV). How to adapt pre-training to the field of
Vision-and-Language~(V-L) learning and improve downstream task performance becomes a focus of multimodal learning.
In this paper, we review the recent progress in Vision-Language Pre-Trained Models~(VL-PTMs). As the core content,
we first briefly introduce several ways to encode raw images and texts to single-modal embeddings before pre-training.
Then, we dive into the mainstream architectures of VL-PTMs in modeling the interaction between text and image
representations. We further present widely-used pre-training tasks, and then we introduce some common downstream tasks.
We finally conclude this paper and present some promising research directions. Our survey aims to provide researchers with
synthesis and pointer to related research.
Pretrained language models (PLMs) have dominated the majority of NLP tasks. While, little research has been conducted on
systematically evaluating the language abilities of PLMs. In this paper, we present a large-scale empirical study on
general language ability evaluation of PLMs (ElitePLM). In our study, we design four evaluation dimensions, i.e., memory,
comprehension, reasoning, and composition, to measure ten widely-used PLMs within five categories. Our empirical results
demonstrate that: (1) PLMs with varying training objectives and strategies are good at different ability tests; (2) fine-tuning
PLMs in downstream tasks is usually sensitive to the data size and distribution; (3) PLMs have excellent transferability between
similar tasks. Moreover, our final predicted results of PLMs can be reused as an open resource for more depth and granularity in
analyzing PLMs' language abilities. This paper can guide the future work to choose, apply, and design PLMs for specific tasks.
Pretrained language models (PLMs) have made remarkable progress in text generation tasks via fine-tuning. However, it is difficult
to fine-tune PLMs in a data-scarce situation. Therefore, it is non-trivial to develop a general and lightweight model that
can adapt to various text generation tasks based on PLMs. Prompt-based learning offers a potential solution. There are two
major challenges for applying prompt-based methods to data-scarce text generation tasks in a transferable setting. First,
it is difficult to effectively transfer prompts for new tasks. Second, it is important to design effective transferring
strategy considering both task- and instance-level information. To address these issues, we propose a novel prompt-based
transfer learning approach for text generation called PTG. PTG learns a set of source prompts for various source generation
tasks and then transfers these prompts to perform target generation tasks through an adaptive attention mechanism
considering both task- and instance-level information. In extensive experiments, PTG yields competitive or better
results than fine-tuning methods. We will release our source prompts as an open-source library, which can be added or reused
to improve new generation tasks for future researches.
2021
In this paper, we study the task of generating long and coherent text. In the literature, Generative Adversarial Nets (GAN) based methods
have been one of the mainstream approaches to generic text generation. We aim to improve two aspects of GAN-based methods in generic
text generation, namely long sequence optimization and semantic coherence enhancement. For this purpose, we propose a novel Multi-Level
Generative Adversarial Networks (MLGAN) for long and coherent text generation. Our approach explicitly models the text generation
process at three different levels, namely paragraph-, sentence- and word-level generation. At the top two levels, we generate
continuous paragraph vectors and sentence vectors as \emph{semantic sketches} to plan the entire content. While, at the bottom
level we generate discrete word tokens for realizing the sentences. Furthermore, we utilize a Conditional GAN architecture to
enhance the inter-sentence coherence by injecting paragraph vectors for sentence vector generation. Extensive experiments results
have demonstrated the effectiveness of the proposed model.
TextBox: A Unified, Modularized, and Extensible Framework for Text Generation
Junyi Li†,
Tianyi Tang†,
Gaole He,
Jinhao Jiang,
Xiaoxuan Hu,
Puzhao Xie,
Zhipeng Chen,
Zhuohao Yu,
Wayne Xin Zhao*,
Ji-Rong Wen
The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021, System Demonstration
pdf / code
We release an open library, called TextBox, which provides a unified, modularized, and extensible text generation framework. TextBox aims
to support a broad set of text generation tasks and models. In TextBox, we implements several text generation models on benchmark
datasets, covering the categories of VAE, GAN, pre-trained language models, etc. Meanwhile, our library maintains sufficient
modularity and extensibility by properly decomposing the model architecture, inference, learning process into highly reusable modules,
which allows easily incorporating new models into our framework. It is specially suitable for researchers and practitioners to
efficiently reproduce baseline models and develop new models. TextBox is implemented based on PyTorch, and released under
Apache License 2.0 at the link https://github.com/RUCAIBox/TextBox.
This paper studies how to automatically generate a natural language text that describes facts in knowledge graph (KG).
Considering a fewshot setting, we leverage the excellent capacities of pretrained language models (PLMs) in
language understanding and generation. We introduce three major technical contributions, namely representation
alignment for bridging the semantic gap between KG encodings and PLMs, relation-biased KG linearization for
deriving better input representations, and multitask learning for learning the correspondence between KG and text.
Extensive experiments on three benchmarks have demonstrated the effectiveness of our model on KG-to-text generation task.
In particular, our model outperforms existing systems on most few-shot settings.
Text generation has become one of the most important yet challenging tasks in natural language
processing (NLP). The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). In this paper,
we presents an overview of the major advances
achieved in the topic of PLM for text generation.
As the preliminaries, we present the general task
definition and briefly describe the mainstream architectures of PLMs. As the core content, we discuss how to adapt existing PLMs to model different
input data and satisfy the properties in the generated text. We further summarize several important
fine-tuning strategies for text generation. Finally,
we present several future directions and conclude
this paper. Our survey aims to provide text generation researchers a synthesis and pointer to relatedresearch.
As a natural language generation task, it is challenging to generate informative and coherent review text. In order to enhance
the informativeness of the generated text, existing solutions typically learn to copy entities or triples from knowledge graphs (KGs).
However, they lack overall consideration to select and arrange the
incorporated knowledge, which tends to cause text incoherence.
To address the above issue, we focus on improving entity-centric
coherence of the generated reviews by leveraging the semantic structure of KGs. In this paper, we propose a novel Coherence Enhanced
Text Planning model (CETP) based on knowledge graphs (KGs) to
improve both global and local coherence for review generation. The
proposed model learns a two-level text plan for generating a document: (1) the document plan is modeled as a sequence of sentence
plans in order, and (2) the sentence plan is modeled as an entitybased subgraph from KG. Local coherence can be naturally enforced
by KG subgraphs through intra-sentence correlations between entities. For global coherence, we design a hierarchical self-attentive
architecture with both subgraph- and node-level attention to enhance the correlations between subgraphs. To our knowledge, we
are the first to utilize a KG-based text planning model to enhance
text coherence for review generation. Extensive experiments on
three datasets confirm the effectiveness of our model on improving
the content coherence of generated texts.
2020
Personalized review generation (PRG) aims to automatically produce review text reflecting user preference, which is a challenging
natural language generation task. Most of previous studies do not
explicitly model factual description of products, tending to generate
uninformative content. Moreover, they mainly focus on word-level
generation, but cannot accurately reflect more abstractive user
preference in multiple aspects.
To address the above issues, we propose a novel knowledgeenhanced PRG model based on capsule graph neural network (CapsGNN). We first construct a heterogeneous knowledge graph (HKG)
for utilizing rich item attributes. We adopt Caps-GNN to learn
graph capsules for encoding underlying characteristics from the
HKG. Our generation process contains two major steps, namely
aspect sequence generation and sentence generation. First, based
on graph capsules, we adaptively learn aspect capsules for inferring the aspect sequence. Then, conditioned on the inferred aspect
label, we design a graph-based copy mechanism to generate sentences by incorporating related entities or words from HKG. To
our knowledge, we are the first to utilize knowledge graph for the
PRG task. The incorporated KG information is able to enhance user
preference at both aspect and word levels. Extensive experiments
on three real-world datasets have demonstrated the effectiveness
of our model on the PRG task.
The task of Knowledge Graph Completion (KGC) aims to automatically infer the missing fact information in Knowledge Graph (KG).In this paper, we take a new perspective that aims to leverage rich
user-item interaction data (user interaction data for short) for improving the KGC task. Our work is inspired by the observation that
many KG entities correspond to online items in application systems.
However, the two kinds of data sources have very different intrinsic
characteristics, and it is likely to hurt the original representation
performance using simple fusion strategy.
To address this challenge, we propose a novel adversarial learning approach for leveraging user interaction data for the KGC task.
Our generator is isolated from user interaction data, and improves
itself according to the feedback from the discriminator. The discriminator takes the learned useful information from user interaction
data as input, and gradually enhances the evaluation capacity in
order to identify the fake samples generated by the generator. To
discover implicit entity preference of users, we design an elaborate
collaborative learning algorithms based on graph neural networks,
which will be jointly optimized with the discriminator. Such an
approach is effective to alleviate the issues about data heterogeneity
and semantic complexity for the KGC task. Extensive experiments
on three real-world datasets have demonstrated the effectiveness
of our approach on the KGC task.
2019
Generating long and informative review text
is a challenging natural language generation
task. Previous work focuses on word-level
generation, neglecting the importance of topical and syntactic characteristics from natural
languages. In this paper, we propose a novel
review generation model by characterizing an
elaborately designed aspect-aware coarse-tofine generation process. First, we model the
aspect transitions to capture the overall content
flow. Then, to generate a sentence, an aspectaware sketch will be predicted using an aspectaware decoder. Finally, another decoder fills in
the semantic slots by generating corresponding words. Our approach is able to jointly
utilize aspect semantics, syntactic sketch, and
context information. Extensive experiments
results have demonstrated the effectiveness of
the proposed model.
|
