Paper list on reasoning in NLP. Here we mainly collect papers about datasets and methods using PLMs (under the progress).
See our survey on natural language reasoning:
Here are anohter two related surveys on LLM prompting:
- Reasoning with Language Model Prompting: A Survey | resources
- Towards Reasoning in Large Language Models: A Survey | resources
- Methodology
- Natural Language Reasoning
- Knowledge Graph Reasoning
- Mathematical Reasoning
-
Show Your Work: Scratchpads for Intermediate Computation with Language Models arXiv (2021)
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, Augustus Odena [pdf]
-
Chain of Thought Prompting Elicits Reasoning in Large Language Models arXiv (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou [pdf] [project]
-
Self-Consistency Improves Chain of Thought Reasoning in Language Models arXiv (2022)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou [pdf]
-
STaR: Bootstrapping Reasoning With Reasoning arXiv (2022)
Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman [pdf]
-
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning arXiv (2022)
Antonia Creswell, Murray Shanahan, Irina Higgins [pdf]
-
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models arXiv (2022)
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, Ed Chi [pdf]
-
Large Language Models are Zero-Shot Reasoners arXiv (2022)
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa [pdf] [project]
-
Language models show human-like content effects on reasoning arXiv (2022)
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill [pdf]
-
Language Model Cascades arXiv (2022)
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, Charles Sutton [pdf] [project]
-
Faithful Reasoning Using Large Language Models arXiv (2022)
Antonia Creswell, Murray Shanahan [pdf]
-
Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought arXiv (2022)
-
ThinkSum: Probabilistic reasoning over sets using large language models arXiv (2022)
Batu Ozturkler, Nikolay Malkin, Zhen Wang, Nebojsa Jojic [pdf]
-
Measuring and Narrowing the Compositionality Gap in Language Models arXiv (2022)
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis [pdf] [project]
Some datasets explicitly target at philosophical reasoning types, e.g. deduction, abduction and induction. Thus, we call them as ``classical logical reasoning tasks''. A key characteristic of this topic is that tasks are mostly artificial to study reasoning.
| Inference Type | Dataset | Size | Task | Link | Remark | |
|---|---|---|---|---|---|---|
| Deductive Reasoning | babI-15 | - | extraction | paper project |
synthetic | |
| RuleTaker | >500k | classification | paper project |
synthetic, the first large-scale benchmark. D*, Birds-Electricity, ParaRules | ||
| ProofWriter | >500k | classification | paper project |
improvement on RuleTaker, + open-world assumption | ||
| PARARULE Plus | 400k | classification | paper project |
improvement on ParaRules, addresses the depth imbalance issue | ||
| AAC | 710k | generation | paper project |
synthetic, syllogistic arguments | ||
| LogicInference | 200k | generation | paper project |
synthetic, more tasks | ||
| FOLIO | 1.4k | classification, generation | paper project |
expert-written, annotate with FOL | ||
| RobustLR | 120k | classification | paper project |
synthetic, robustness on logical semantics | ||
| Defeasible Reasoning | Abductive Reasoning | AbductionRules | - | generation | paper project |
variant of RuleTaker |
| ART (αNLI, αNLG) |
17.8k | classification, generation | paper project |
commonsense, based on ROCStories | ||
| Inductive Reasoning | bAbI-16 | - | extraction | paper project |
synthetic, induce-then-deduce | |
| CLUTRR | - | extraction | paper project |
synthetic, induce-then-deduce, kinship | ||
| DEER | 1.2k | generation | paper project |
induce explicit natural language rule (human-authored) from natural language facts (Webs) | ||
| Others | defeasibleNLI | 43.8k | classification, generation | paper project |
direction on evidence updation, based on the existing datasets | |
Papers on dataset artifacts:
-
On the Paradox of Learning to Reason from Data arXiv (2022)
Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, Guy Van den Broeck [pdf] [project]
-
Transformers as Soft Reasoners over Language IJCAI (2020)
Peter Clark, Oyvind Tafjord, Kyle Richardson [pdf] [project]
-
PRover: Proof Generation for Interpretable Reasoning over Rules EMNLP (2020)
Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, Mohit Bansal [pdf] [project]
-
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning NAACL (2021)
Swarnadeep Saha, Prateek Yadav, Mohit Bansal [pdf] [project]
-
Critical Thinking for Language Models IWCS (2021)
Gregor Betz, Christian Voigt, Kyle Richardson [pdf] [project]
-
Explainable Multi-hop Verbal Reasoning Through Internal Monologue NAACL (2021)
Zhengzhong Liang, Steven Bethard, Mihai Surdeanu [pdf] [project]
-
ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language ACL findings (2021)
-
Flexible Generation of Natural Language Deductions EMNLP (2021)
Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, Greg Durrett [pdf] [project]
-
FaiRR: Faithful and Robust Deductive Reasoning over Natural Language ACL (2022)
-
Interpretable Proof Generation via Iterative Backward Reasoning NAACL (2022)
Hanhao Qu, Yu Cao, Jun Gao, Liang Ding, Ruifeng Xu [pdf] [project]
-
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning arXiv (2022)
Antonia Creswell, Murray Shanahan, Irina Higgins [pdf]
-
Generating Natural Language Proofs with Verifier-Guided Search arXiv (2022)
-
ROBUSTLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners arXiv (2022)
-
Language models show human-like content effects on reasoning arXiv (2022)
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill [pdf]
-
Faithful Reasoning Using Large Language Models arXiv (2022)
Antonia Creswell, Murray Shanahan [pdf]
-
Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought arXiv (2022)
-
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language arXiv (2022)
Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, Deepak Ramachandran [pdf]
Defeasible reasoning:
-
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision AAAI (2020)
Faeze Brahman, Vered Shwartz, Rachel Rudinger, Yejin Choi [pdf] [project]
-
Could you give me a hint ? Generating inference graphs for defeasible reasoning ACL findings (2021)
Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yiming Yang, Eduard H. Hovy [pdf] [project]
-
Think about it! Improving defeasible reasoning by first modeling the question scenario EMNLP (2021)
Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, Eduard H. Hovy [pdf] [project]
-
Language Models as Inductive Reasoners arXiv (2022)
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, Furu Wei [pdf]
Task: given a premise-hypothesis pair, classify it into three classes: entailment, contradiction and neutral.
There are mainly three types of premise-hypothesis pairs in NLI task: paraphrase, compound semantics understanding (CSU), and reasoning. Here we just consider the last one.
| Premise | Hypothesis | |
|---|---|---|
| Paraphrase | Two doctors perform surgery on patient | Doctors are performing surgery |
| CSU | Two women are embracing while holding to go packages | Two women are holding packages (Two women are embracing) |
| Reasoning | A soccer game with multiple males playing (Soccer is a sport) |
Some men are playing a sport |
| Dataset | Size | Link | Remark |
|---|---|---|---|
| generic | |||
| SNLI | 570k | paper project |
the first large-scale NLI dataset one of the most typical |
| e-SNLI | - | paper project |
annotate natural language explanations for SNLI |
| MultiNLI | 433k | paper project |
cover more styles and topics than SNLI one of the most typical |
| DebiasedNLI | 7.5k | paper project |
debiased versions of SNLI & MultiNLI |
| XNLI | 7.5k | paper project |
cross-lingual, based on MultiNLI |
| MPE | 10k | paper project |
multiple premises |
| science | |||
| SciTail | 27k | paper project |
the first NLI dataset with entirely existing text |
| SciNLI | 107k | paper project |
data from scholarly papers |
Recently, some datasets are proposed to model different subjective opinions (on classifying into which class) of crowdworkers.
| Subjective Opinions | ||||
|---|---|---|---|---|
| Dataset | Domain | Size | Link | Remark |
| UNLI | generic | 61k | paper project |
subjective probability assessment (regression rather than binary), based on SNLI |
| ChaosNLI | generic | 464k | paper project |
human opinion distribution, based on SNLI, MultiNLI and αNLI |
Some datasets for other language:
| Other Languages | |||
|---|---|---|---|
| Dataset | Language | Link | Remark |
| NLI-TR | Turkish | paper project |
translate SNLI and MultiNLI |
| IndoNLI | Indonesian | paper project |
data collection protocol from MultiNLI |
Papers on dataset artifacts:
-
Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment LREC (2018)
Masatoshi Tsuchiya [pdf]
-
Annotation Artifacts in Natural Language Inference Data NAACL (2018)
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, Noah A. Smith [pdf]
-
Hypothesis Only Baselines in Natural Language Inference SEM (2019)
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, Benjamin Van Durme [pdf]
-
NILE : Natural Language Inference with Faithful Natural Language Explanations ACL (2020)
-
Identifying inherent disagreement in natural language inference NAACL (2021)
Xinliang Frederick Zhang, Marie-Catherine de Marneffe [pdf] [project]
-
KACE: Generating Knowledge Aware Contrastive Explanations for Natural Language Inference ACL (2021)
Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang, Haiqing Chen, Yin Zhang [pdf] [project]
-
Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference ACL findings (2021)
Hai Hu, He Zhou, Zuoyu Tian, Yiwen Zhang, Yina Patterson, Yanting Li, Yixin Nie, Kyle Richardson [pdf] [project]
-
Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates ACL (2022)
Kunxun Qi, Hai Wan, Jianfeng Du, Haolan Chen [pdf] [project]
-
Generating Intermediate Steps for NLI with Next-Step Supervision arXiv (2022)
Deepanway Ghosal, Somak Aditya, Monojit Choudhury [pdf]
This topic studies answering the complex questions that require to reason over evidences scattered in different contexts. The term ``hop'' here indicates the number of contexts required to reason. There are two settings on the required contexts: (1) all provided along with some distractors (i.e. distractor), (2) need to be retrieved (i.e. retrieval).
Some datasets annotate the ground supporting evidences (paragraph-level, sentence-level, or triple-level), decomposed sub-questions (and the corresponding evidences), or reasoning paths.
| Dataset | Size | Knowledge Source | Setting | Answer Type | Evidence | Link | Remark |
|---|---|---|---|---|---|---|---|
| generic | |||||||
| WikiHop | 51k | Wikipedia | distractor | choice | - | paper project |
one of the most typical |
| HotpotQA | 112k | Wikipedia | distractor, retrieval | span, yes/no | sentence | paper project |
the most popular one |
| R4C | 4.6k | - | - | - | triple | paper project |
annotate atomic facts for HotpotQA |
| BeerQA | 530 | Wikipedia | retrieval | span, yes/no | - | paper project |
more hops |
| 2WikiMultiHopQA | 192k | Wikipedia | distractor | span | sentence triple |
paper project |
similar to WikiHop |
| MuSiQue | 25k | Wikipedia | distractor | span | paragraph sub-questions |
paper project |
more hops |
| StrategyQA | 2.7k | Wikipedia | retrieval | yes/no | paragraph sub-questions |
paper project |
implicit mult-hop questions |
| specific domain | |||||||
| MedHop | 2.5k | Medline | distractor | choice | - | paper project |
medicine. similar to WikiHop |
| QASC | 9.9k | WorldTree | retrieval | choice | sentence | paper project |
science |
| eQASC | - | - | - | - | reasoning path | paper project |
annotate reasoning paths for QASC |
Papers on dataset artifacts:
-
Understanding Dataset Design Choices for Multi-hop Reasoning NAACL (2019)
Jifan Chen, Greg Durrett [pdf]
-
Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA ACL (2019)
-
Compositional Questions Do Not Necessitate Multi-hop Reasoning ACL (2019)
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke Zettlemoyer [pdf] [project]
-
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning EMNLP (2020)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal [pdf] [project]
-
Dynamically Fused Graph Network for Multi-hop Reasoning ACL (2019)
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, Yong Yu [pdf] [project]
-
Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs ACL (2019)
Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, Bowen Zhou [pdf]
-
Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction ACL (2019)
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, Junji Tomita [pdf]
-
Multi-hop Reading Comprehension through Question Decomposition and Rescoring ACL (2019)
Sewon Min, Victor Zhong, Luke Zettlemoyer, Hannaneh Hajishirzi [pdf] [project]
-
Differentiable Reasoning over a Virtual Knowledge Base ICLR (2020)
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, William W. Cohen [pdf] [project]
-
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention ICLR Poster (2020)
Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul N. Bennett, Saurabh Tiwary [pdf] [project]
-
Low-Resource Generation of Multi-hop Reasoning Questions ACL (2020)
Jianxing Yu, Wei Liu, Shuang Qiu, Qinliang Su, Kai Wang, Xiaojun Quan, Jian Yin [pdf]
-
SRLGRN: Semantic Role Labeling Graph Reasoning Network EMNLP (2020)
Chen Zheng, Parisa Kordjamshidi [pdf]
-
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering ICLR (2020)
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, Caiming Xiong [pdf] [project]
-
Robustifying Multi-hop QA through Pseudo-Evidentiality Training ACL (2021)
Kyungjae Lee, Seung-won Hwang, Sang-eun Han, Dohyeon Lee [pdf]
-
Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension EMNLP (2021)
Naoya Inoue, Harsh Trivedi, Steven Sinha, Niranjan Balasubramanian, Kentaro Inui [pdf] [project]
-
Generative Context Pair Selection for Multi-hop Question Answering EMNLP (2021)
Dheeru Dua, Cícero Nogueira dos Santos, Patrick Ng, Ben Athiwaratkun, Bing Xiang, Matt Gardner, Sameer Singh [pdf] [project]
-
Breadth First Reasoning Graph for Multi-hop Question Answering NAACL (2021)
Yongjie Huang, Meng Yang [pdf]
-
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval ICLR Poster (2021)
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick S. H. Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, Barlas Oguz [pdf] [project]
-
Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval NAACL (2021)
Chen Zhao, Chenyan Xiong, Jordan L. Boyd-Graber, Hal Daumé III [pdf] [project]
-
Unsupervised Multi-hop Question Answering by Question Generation NAACL (2021)
Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, William Yang Wang [pdf] [project]
-
If You Want to Go Far Go Together: Unsupervised Joint Candidate Evidence Retrieval for Multi-hop Question Answering NAACL (2021)
-
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval NIPS (2021)
Omar Khattab, Christopher Potts, Matei A. Zaharia [pdf] [project]
-
Modeling Multi-hop Question Answering as Single Sequence Prediction ACL (2022)
Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Nitish Shirish Keskar, Caiming Xiong [pdf]
-
CQG: A Simple and Effective Controlled Generation Framework for Multi-hop Question Generation ACL (2022)
Zichu Fei, Qi Zhang, Tao Gui, Di Liang, Sirui Wang, Wei Wu, Xuanjing Huang [pdf] [project]
Commonsense reasoning deals with implicit commonsense knowledge, which may be non-trivial to machines since they are difficult to retrieve from the web due to reporting bias. While it is named with "reasoning", the common theme of this topic is commonsense knowledge rather than reasoning. Here, we only list reasoning datasets.
There are mainly three types of reasoning datasets towards "what" (i.e. assertions or events), "what if / why" (e.g. causal and temporal relations between events), and "how" (i.e. actions) respectively.
"What" commonsense reasoning require combining multiple pieces of knowledge that some are from external knowledge sources while others are commonsense knowledge.
| "What" Commonsense Reasoning | |||||
|---|---|---|---|---|---|
| Dataset | Size | Other Knowledge Type / Source | Task | Link | Rationale |
| OpenBookQA | 6k | science / WorldTree |
multi-choice QA | paper project |
ground science facts |
| OpenCSR | 20k | science / WorldTree, ARC corpus |
free-form QA | paper project |
- |
| CREAK | 13k | entity / Wikipedia |
claim verification | paper project |
explanation |
"What if / Why" commonsense reasoning often reasons for causal and temporal relations between events. There are two causal relations: causes and effects, which can be seen as backward causal reasoning and forward causal reasoning respectively.
| "What if / Why" Commonsense Reasoning | |||||
|---|---|---|---|---|---|
| Dataset | Size | Direction | Task | Link | Remark |
| ROCStories | 50k | temporal | 2-choice QA | paper project |
- |
| SWAG | 113k | temporal | multi-choice QA | paper project |
- |
| HellaSwag | 20k | temporal | multi-choice QA | paper project |
an upgraded SWAG |
| COPA | 1k | both | 2-choice QA | paper project |
- |
| Social-IQA | 38k | both | multi-choice QA | paper project |
social situations |
| e-CARE | 21k | both | 2-choice QA | paper project |
with ground supporting facts |
| WIQA | 40k | forward | multi-choice QA | paper project |
about nature processes |
| TIMETRAVEL | 29k | forward | generation | paper project |
counterfactual reasoning |
| ART | 20k | backward | 2-choice/generation | paper project |
abductive commonsense reasoning |
| TellMeWhy | 30k | backward | free-form QA | paper project |
each annotated 3 possible answers |
| WikiWhy | 9k | backward | free-form QA | paper project |
about Wikipedia entities / events |
"How" commonsense reasoning is mainly about "how to do it". It is often related to problem-solving or decision-making.
| "How" Commonsense Reasoning | |||||
|---|---|---|---|---|---|
| Dataset | Size | Source | Task | Link | Remark |
| WikiHow Goal-Step | 1489k | WikiHow, generated | multi-choice | paper project |
goals, steps, and temporal ordering |
| PIQA | 21k | human-authored | 2-choice | paper project |
physical |
Some datasets involve multiple types of reasoning.
| Hybrid Commonsense Reasoning | ||||
|---|---|---|---|---|
| Dataset | Size | Task | Link | Remark |
| CSQA | 12k | multi-choice QA | paper project |
ConceptNet concepts |
| CoS-E | - | - | paper project |
annotate explanations for CSQA |
| ECQA | - | - | paper project |
annotate commonsense facts for CSQA |
| CSQA2 | 14k | boolen QA | paper project |
data construction via gamification |
| CosmosQA | 35k | multi-choice QA | paper project |
reading comprehension on blogs |
| Moral Stories | 12k | multi-choice QA | paper project |
situated reasoning with social norms |
-
Attention Is (not) All You Need for Commonsense Reasoning ACL (2019)
Tassilo Klein, Moin Nabi [pdf]
-
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction ACL (2019)
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, Yejin Choi [pdf] [project]
-
Explain Yourself! Leveraging Language Models for Commonsense Reasoning ACL (2019)
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, Richard Socher [pdf] [project]
-
Commonsense Knowledge Mining from Pretrained Models EMNLP (2019)
Joe Davison, Joshua Feldman, Alexander M. Rush [pdf]
-
How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG EMNLP (2019)
Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung [pdf] [project]
-
Guided Generation of Cause and Effect IJCAI (2020)
Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, Benjamin Van Durme [pdf] [project]
-
Contrastive Self-Supervised Learning for Commonsense Reasoning ACL (2020)
-
Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning ACL (2020)
Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot, Louise Naudin [pdf]
-
Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder ACL (2020)
Daya Guo, Duyu Tang, Nan Duan, Jian Yin, Daxin Jiang, Ming Zhou [pdf] [project]
-
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering EMNLP (2020)
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, Xiang Ren [pdf] [project]
-
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning EMNLP (2020)
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, Yejin Choi [pdf] [project]
-
Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering EMNLP (2020)
Pratyay Banerjee, Chitta Baral [pdf]
-
Unsupervised Commonsense Question Answering with Self-Talk EMNLP (2020)
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi [pdf]
-
Paragraph-level Commonsense Transformers with Recurrent Memory AAAI (2021)
Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, Yejin Choi [pdf] [project]
-
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering AAAI (2021)
Kaixin Ma, Filip Ilievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, Alessandro Oltramari [pdf] [project]
-
Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models ACL (2021)
Peter West, Ximing Lu, Ari Holtzman, Chandra Bhagavatula, Jena D. Hwang, Yejin Choi [pdf] [project]
-
Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models ACL (2021)
Mingyue Han, Yinglin Wang [pdf]
-
Learning Event Graph Knowledge for Abductive Reasoning ACL (2021)
-
ExCAR: Event Graph Knowledge Enhanced Explainable Causal Reasoning ACL (2021)
Li Du, Xiao Ding, Kai Xiong, Ting Liu, Bing Qin [pdf] [project]
-
Differentiable Open-Ended Commonsense Reasoning NAACL (2021)
Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William W. Cohen [pdf] [project]
-
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering NAACL (2021)
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec [pdf] [project]
-
Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models EMNLP (2021)
-
Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models EMNLP (2021)
Kaixin Ma, Filip Ilievski, Jonathan Francis, Satoru Ozaki, Eric Nyberg, Alessandro Oltramari [pdf] [project]
-
Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning EMNLP (2021)
Ruben Branco, António Branco, João António Rodrigues, João Ricardo Silva [pdf] [project]
-
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference EMNLP findings (2021)
-
SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning NIPS (2021)
Aaron Chan, Jiashu Xu, Boyuan Long, Soumya Sanyal, Tanishq Gupta, Xiang Ren [pdf] [project]
-
GreaseLM: Graph REASoning Enhanced Language Models ICLR Spotlight (2022)
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, Jure Leskovec [pdf] [project]
-
Generated Knowledge Prompting for Commonsense Reasoning ACL (2022)
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi [pdf] [project]
-
JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering NAACL (2022)
-
Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoning NAACL (2022)
Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-won Hwang, Jinyoung Yeo [pdf]
-
On Curriculum Learning for Commonsense Reasoning NAACL (2022)
Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-won Hwang, Jinyoung Yeo [pdf] [project]
-
Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference NAACL (2022)
Emils Kadikis, Vaibhav Srivastav, Roman Klinger [pdf] [project]
-
Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge NAACL (2022)
Ian Porada, Alessandro Sordoni, Jackie Chi Kit Cheung [pdf]
-
ROCK: Causal Inference Principles for Reasoning about Commonsense Causality ICML (2022)
Jiayao Zhang, Hongming Zhang, Weijie J. Su, Dan Roth [pdf] [project]
-
ALERT: Adapting Language Models to Reasoning Tasks arXiv (2022)
Ping Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissy, Gargi Ghosh, Mona T. Diab, Asli Celikyilmaz [pdf]
-
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations EMNLP (2022)
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi [pdf] [project]
-
Using Commonsense Knowledge to Answer Why-Questions EMNLP (2022)
Yash Kumar Lal, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond J. Mooney, Niranjan Balasubramanian [pdf] [project]
| KB | Type of Knowledge | Format of Knowledge | Link |
|---|---|---|---|
| CYC | generic | LISP-style logic | paper project |
| ConceptNet | linguistics | triple | paper project |
| ConceptNet 5.5 | linguistics | triple | paper project |
| GenericsKB | generic | statement | paper project |
| Event2Mind | mental state | statement | paper project |
| ATOMIC | social causality | statement | paper project |
| ATOMIC 2020 | +physical and eventive causality | statement | paper project |
| Social-Chem-101 | rules-of-thumb | statement | paper project |
There are some datasets collected from realistic examinations or tests or explicitly designed to challenge LLMs, which may require domain-specific knowledge and multiple types of reasoning skills.
| Realistic Examinations | ||||||
|---|---|---|---|---|---|---|
| Dataset | Size | Domain | Source | Task | Link | Remark |
| AR-LSAT | 2k | law | law school admission test | multi-choice QA | paper project |
- |
| HEAD-QA | 6.7k | healthcare | specialized healthcare examination | multi-choice QA | paper project |
- |
| AI2-ARC | 7.7k | science | grade-school standardized test | multi-choice QA | paper project |
- |
| EntailmentBank | 2k | - | - | entailment tree generation | paper project |
reasoning paths to hypotheses from AI2-ARC |
| ReClor | 6k | generic | standardized graduate admission examination | RC + multi-choice QA | paper project |
- |
| MetaLogic | 1k | - | - | logic metagraph generation | paper project |
reasoning graphs for passages in ReClor |
| LogiQA | 8k | generic | national civil servants examination of China | RC + multi-choice QA | paper project |
- |
| ConTRoL | 8k | generic | competitive selection and recruitment test | NLI | paper project |
passage-level |
| Diagnostic Benchmarks for LLMs | |||
|---|---|---|---|
| Benchmark | Tasks | Link | Remark |
| BIG-Bench | 204 | paper project |
believed to be beyond the capabilities of current PLMs |
| BBH | 23 | paper project |
challenging BIG-Bench tasks |
| MMLU | 57 | paper project |
across a diverse set of subjects that humans learn |
Knowledge graph completion task aims to complete the graph, while multi-hop reasoning over KG is the task querying in incomplete graphs, both of which require reasoning over knowledge graphs. Temporal knowledge graph reasoning aims to predict links in future with the past quadruples.
-
Collaborative Policy Learning for Open Knowledge Graph Reasoning EMNLP (2019)
Cong Fu, Tong Chen, Meng Qu, Woojeong Jin, Xiang Ren [pdf] [project]
-
DIVINE: A Generative Adversarial Imitation Learning Framework for Knowledge Graph Reasoning EMNLP (2019)
-
Learning Collaborative Agents with Rule Guidance for Knowledge Graph Reasoning EMNLP (2020)
Deren Lei, Gangrong Jiang, Xiaotao Gu, Kexuan Sun, Yuning Mao, Xiang Ren [pdf] [project]
-
Incorporating Graph Attention Mechanism into Knowledge Graph Reasoning Based on Deep Reinforcement Learning EMNLP (2019)
Heng Wang, Shuangyin Li, Rong Pan, Mingzhi Mao [pdf] [project]
-
Dynamically Pruned Message Passing Networks for Large-scale Knowledge Graph Reasoning ICLR Poster (2020)
Xiaoran Xu, Wei Feng, Yunsheng Jiang, Xiaohui Xie, Zhiqing Sun, Zhi-Hong Deng [pdf] [project]
-
Inductive Relation Prediction by Subgraph Reasoning ICML (2020)
Komal K. Teru, Etienne G. Denis, William L. Hamilton [pdf] [project]
-
Adapting Meta Knowledge Graph Information for Multi-Hop Reasoning over Few-Shot Relations EMNLP (2019)
Xin Lv, Yuxian Gu, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu [pdf] [project]
-
Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph EMNLP (2020)
Xin Lv, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Wei Zhang, Yichi Zhang, Hao Kong, Suhui Wu [pdf] [project]
-
UniKER: A Unified Framework for Combining Embedding and Definite Horn Rule Reasoning for Knowledge Graph Inference EMNLP (2021)
Kewei Cheng, Ziqing Yang, Ming Zhang, Yizhou Sun [pdf]
-
Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability EMNLP (2021)
Xin Lv, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Yichi Zhang, Zelin Dai [pdf] [project]
-
GMH: A General Multi-hop Reasoning Model for KG Completion EMNLP (2021)
Yao Zhang, Hongru Liang, Adam Jatowt, Wenqiang Lei, Xin Wei, Ning Jiang, Zhenglu Yang [pdf]
-
Neural-Symbolic Commonsense Reasoner with Relation Predictors ACL (2021)
Farhad Moghimifar, Lizhen Qu, Yue Zhuo, Gholamreza Haffari, Mahsa Baktashmotlagh [pdf] [project]
-
Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings ICLR Poster (2020)
-
Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs NIPS (2020)
-
Probabilistic Entity Representation Model for Reasoning over Knowledge Graphs NIPS (2021)
Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, Chandan K. Reddy [pdf] [project]
-
ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs NIPS (2021)
Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, Feng Wu [pdf] [project]
-
Complex Query Answering with Neural Link Predictors ICLR Oral (2021)
Erik Arakelyan, Daniel Daza, Pasquale Minervini, Michael Cochez [pdf] [project]
-
Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs ICLR Poster (2021)
-
Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs ACL (2021)
Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang, Xueqi Cheng [pdf]
-
Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning ACL (2022)
Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, Xueqi Cheng [pdf] [project]
-
Quantum Embedding of Knowledge for Reasoning NIPS (2019)
Dinesh Garg, Shajith Ikbal, Santosh K. Srivastava, Harit Vishwakarma, Hima P. Karanam, L. Venkata Subramaniam [pdf] [project]
-
Scalable Neural Methods for Reasoning With a Symbolic Knowledge Base ICLR Poster (2020)
William W. Cohen, Haitian Sun, R. Alex Hofer, Matthew Siegler [pdf]
-
Probabilistic Logic Neural Networks for Reasoning NIPS (2019)
Meng Qu, Jian Tang [pdf]
-
RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs ICLR Poster (2021)
Meng Qu, Junkun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio, Jian Tang [pdf] [project]
-
Efficient Probabilistic Logic Reasoning with Graph Neural Networks ICLR Poster (2020)
Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, Le Song [pdf]
-
Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning NAACL (2021)
Xuelu Chen, Michael Boratko, Muhao Chen, Shib Sankar Dasgupta, Xiang Lorraine Li, Andrew McCallum [pdf] [project]
-
Multimodal Analogical Reasoning over Knowledge Graphs ICLR (2023)
Ningyu Zhang, Lei Li, Xiang Chen, Xiaozhuan Liang, Shumin Deng, Huajun Chen [pdf] [project]
-
Analysing Mathematical Reasoning Abilities of Neural Models ICLR Poster (2019)
David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli [pdf] [project]
-
HOList: An Environment for Machine Learning of Higher-Order Theorem Proving ICML (2019)
Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, Stewart Wilcox [pdf] [project]
-
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs EMNLP (2019)
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner [pdf] [project]
-
IsarStep: a Benchmark for High-level Mathematical Reasoning ICLR Poster (2021)
Wenda Li, Lei Yu, Yuhuai Wu, Lawrence C. Paulson [pdf] [project]
-
Towards Table-to-Text Generation with Numerical Reasoning ACL (2021)
Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, Hiroya Takamura [pdf] [project]
-
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning ACL (2021)
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, Song-Chun Zhu [pdf] [project]
-
FINQA: A Dataset of Numerical Reasoning over Financial Data EMNLP (2021)
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R. Routledge, William Yang Wang [pdf] [project]
-
SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables NIPS (2021)
Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, Iryna Gurevych [pdf] [project]
-
MULTIHIERTT: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data ACL (2022)
Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang [pdf] [project]
-
NUMGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks ACL (2022)
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Singh Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan [pdf] [project]
-
Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems NAACL (2019)
-
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning EMNLP (2019)
Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li [pdf] [project]
-
NumNet: Machine Reading Comprehension with Numerical Reasoning EMNLP (2019)
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, Zhiyuan Liu [pdf] [project]
-
Mathematical Reasoning in Latent Space ICLR Oral (2020)
Dennis Lee, Christian Szegedy, Markus N. Rabe, Sarah M. Loos, Kshitij Bansal [pdf]
-
Neural Module Networks for Reasoning over Text ICLR Poster (2020)
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, Matt Gardner [pdf] [project]
-
Injecting Numerical Reasoning Skills into Language Models ACL (2020)
-
Question Directed Graph Attention Network for Numerical Reasoning over Text EMNLP (2020)
Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, Wei Chu [pdf]
-
Mathematical Reasoning via Self-supervised Skip-tree Training ICLR Spotlight (2021)
Markus Norman Rabe, Dennis Lee, Kshitij Bansal, Christian Szegedy [pdf]
-
Incorporating External Knowledge to Enhance Tabular Reasoning NAACL (2021)
-
Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning ACL (2021)
Piotr Piekos, Mateusz Malinowski, Henryk Michalewski [pdf]
-
GraphMR: Graph Neural Network for Mathematical Reasoning ACL (2021)
Weijie Feng, Binbin Liu, Dongpeng Xu, Qilong Zheng, Yun Xu [pdf] [project]
-
LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning ICML (2021)
Yuhuai Wu, Markus N. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, Christian Szegedy [pdf] [project]
-
Numerical reasoning in machine reading comprehension tasks: are we there yet? EMNLP (2021)
Hadeel Al-Negheimish, Pranava Madhyastha, Alessandra Russo [pdf]
-
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction ACL (2022)
-
FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining ACL (2022)
Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng, Dongmei Zhang [pdf] [project]
-
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning ACL (2022)
Vivek Gupta, Shuo Zhang, Alakananda Vempala, Yujie He, Temma Choji, Vivek Srikumar [pdf] [project]
-
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills ACL (2022)
-
OPERA: Operation-Pivoted Discrete Reasoning over Text NAACL (2022)
Yongwei Zhou, Junwei Bao, Chaoqun Duan, Haipeng Sun, Jiahui Liang, Yifan Wang, Jing Zhao, Youzheng Wu, Xiaodong He, Tiejun Zhao [pdf] [project]
Fei YU
@article{yu2023natural,
title={Natural Language Reasoning, A Survey},
author={Yu, Fei and Zhang, Hongbo and Wang, Benyou},
journal={arXiv preprint arXiv:2303.14725},
year={2023}
}