{"paper":{"arxiv_id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","abstract":"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new \"Colossal Clean Crawled Corpus\", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.","primary_category":"cs.LG","venue":"JMLR 2020","published_at":null,"latest_version":1,"withdrawn":false},"latest_version":{"id":"f2f6e549-2a7e-426d-b05a-d36443885174","version":1,"source_url":"https://arxiv.org/abs/1910.10683","rendered_html_url":null,"rendering_engine":null},"verdict":{"id":"32ec1dff-f437-4426-94c8-0e17734ef33b","kind":"POST","status":"reproduced","score":0.8150000000000001,"confidence":0.8,"agent_version":"v0.1.0-t5-mnli-microslice","computed_at":"2026-05-14T23:17:43.450Z","is_current":true,"claim_citation":{"paper_arxiv_id":"1910.10683","section":"Table 14","row":"T5-Small","column":"MNLI-m","reported_value":82.4,"reported_metric":"accuracy","quoted_text":"T5-Small 82.4","pdf_page":30,"notes":"Table 14 (Performance of T5 variants on the GLUE dev set) of arXiv:1910.10683 reports T5-Small MNLI-m = 82.4. Driver evaluates the community checkpoint `valhalla/t5-small-glue-mnli` on an MNLI dev micro-slice. PROTOCOL_MATCH is `proxy` on both checkpoint and dataset axes."},"protocol_match":"proxy"},"verdicts":{"post":{"id":"32ec1dff-f437-4426-94c8-0e17734ef33b","kind":"POST","status":"reproduced","score":0.8150000000000001,"confidence":0.8,"agent_version":"v0.1.0-t5-mnli-microslice","computed_at":"2026-05-14T23:17:43.450Z","is_current":true,"claim_citation":{"paper_arxiv_id":"1910.10683","section":"Table 14","row":"T5-Small","column":"MNLI-m","reported_value":82.4,"reported_metric":"accuracy","quoted_text":"T5-Small 82.4","pdf_page":30,"notes":"Table 14 (Performance of T5 variants on the GLUE dev set) of arXiv:1910.10683 reports T5-Small MNLI-m = 82.4. Driver evaluates the community checkpoint `valhalla/t5-small-glue-mnli` on an MNLI dev micro-slice. PROTOCOL_MATCH is `proxy` on both checkpoint and dataset axes."},"protocol_match":"proxy"},"pre":null}}