{"paper":{"arxiv_id":"2111.09543","title":"DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing","abstract":"This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the tug-of-war dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models.","primary_category":"cs.CL","venue":"ICLR 2023","published_at":null,"latest_version":1,"withdrawn":false},"latest_version":{"id":"460772b5-a933-4b10-afde-7f6ab39fc6c1","version":1,"source_url":"https://arxiv.org/abs/2111.09543","rendered_html_url":null,"rendering_engine":null},"verdict":{"id":"da59c566-d0cc-40b4-9fe7-2f6cdf566087","kind":"POST","status":"reproduced","score":0.9133653461384554,"confidence":0.85,"agent_version":"v0.1.0-deberta-mnli-microslice","computed_at":"2026-05-14T23:56:01.683Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2111.09543","section":"Table 1","row":"DeBERTa-v3-large","column":"MNLI-m","reported_value":91.8,"reported_metric":"accuracy","quoted_text":"DeBERTa-v3-large 91.8","pdf_page":6,"notes":"Table 1 of arXiv:2111.09543 reports DeBERTa-v3-large MNLI-m = 91.8. Driver uses the community NLI fine-tune `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` on an MNLI dev micro-slice. PROTOCOL_MATCH is `proxy` on both axes."},"protocol_match":"proxy"},"verdicts":{"post":{"id":"da59c566-d0cc-40b4-9fe7-2f6cdf566087","kind":"POST","status":"reproduced","score":0.9133653461384554,"confidence":0.85,"agent_version":"v0.1.0-deberta-mnli-microslice","computed_at":"2026-05-14T23:56:01.683Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2111.09543","section":"Table 1","row":"DeBERTa-v3-large","column":"MNLI-m","reported_value":91.8,"reported_metric":"accuracy","quoted_text":"DeBERTa-v3-large 91.8","pdf_page":6,"notes":"Table 1 of arXiv:2111.09543 reports DeBERTa-v3-large MNLI-m = 91.8. Driver uses the community NLI fine-tune `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` on an MNLI dev micro-slice. PROTOCOL_MATCH is `proxy` on both axes."},"protocol_match":"proxy"},"pre":null}}