{"paper":{"arxiv_id":"2003.10555","title":"ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators","abstract":"Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark.","primary_category":"cs.CL","venue":"ICLR 2020","published_at":null,"latest_version":1,"withdrawn":false},"latest_version":{"id":"f4240064-0764-4a8d-944a-3b5652cb1497","version":1,"source_url":"https://arxiv.org/abs/2003.10555","rendered_html_url":null,"rendering_engine":null},"verdict":{"id":"80802f3f-82f9-4551-9174-68038c26f3f5","kind":"POST","status":"reproduced","score":0.8766666666666666,"confidence":0.8,"agent_version":"v0.1.0-electra-mnli-microslice","computed_at":"2026-05-14T23:17:46.124Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2003.10555","section":"Table 8","row":"ELECTRA-Base","column":"MNLI","reported_value":88.5,"reported_metric":"accuracy","quoted_text":"ELECTRA-Base 88.5","pdf_page":8,"notes":"Table 8 (Dev set GLUE results) of arXiv:2003.10555 reports ELECTRA-Base MNLI-m = 88.5. The matching HuggingFace checkpoint is `howey/electra-base-mnli`. Driver evaluates an MNLI dev micro-slice, so PROTOCOL_MATCH is `proxy`."},"protocol_match":"proxy"},"verdicts":{"post":{"id":"80802f3f-82f9-4551-9174-68038c26f3f5","kind":"POST","status":"reproduced","score":0.8766666666666666,"confidence":0.8,"agent_version":"v0.1.0-electra-mnli-microslice","computed_at":"2026-05-14T23:17:46.124Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2003.10555","section":"Table 8","row":"ELECTRA-Base","column":"MNLI","reported_value":88.5,"reported_metric":"accuracy","quoted_text":"ELECTRA-Base 88.5","pdf_page":8,"notes":"Table 8 (Dev set GLUE results) of arXiv:2003.10555 reports ELECTRA-Base MNLI-m = 88.5. The matching HuggingFace checkpoint is `howey/electra-base-mnli`. Driver evaluates an MNLI dev micro-slice, so PROTOCOL_MATCH is `proxy`."},"protocol_match":"proxy"},"pre":null}}