{"paper":{"arxiv_id":"2308.12950","title":"Code Llama: Open Foundation Models for Code","abstract":"We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively.","primary_category":"cs.CL","venue":"arXiv 2023","published_at":null,"latest_version":1,"withdrawn":false},"latest_version":{"id":"6a6b32a4-c22d-4442-8620-0ef7042e8fa1","version":1,"source_url":"https://arxiv.org/abs/2308.12950","rendered_html_url":null,"rendering_engine":null},"verdict":{"id":"f156bd8c-aca4-4a2d-9c66-bf50bf28965e","kind":"POST","status":"partial","score":1.709291715799857,"confidence":0.6,"agent_version":"v0.1.0-codellama-pythonppl-microslice","computed_at":"2026-05-14T23:53:26.429Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2308.12950","section":"Table 2","row":"Code Llama - Python 7B","column":"HumanEval pass@1","reported_value":38.4,"reported_metric":"pass@1","quoted_text":"38.4","pdf_page":7,"notes":"Table 2 of arXiv:2308.12950 reports Code Llama - Python 7B HumanEval pass@1 = 38.4. Driver measures Python perplexity (sanity probe), not HumanEval. PROTOCOL_MATCH = `unknown` because the metric measured differs from the metric the paper reports. Validator C1 gate prevents publication of WRONG."},"protocol_match":"unknown"},"verdicts":{"post":{"id":"f156bd8c-aca4-4a2d-9c66-bf50bf28965e","kind":"POST","status":"partial","score":1.709291715799857,"confidence":0.6,"agent_version":"v0.1.0-codellama-pythonppl-microslice","computed_at":"2026-05-14T23:53:26.429Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2308.12950","section":"Table 2","row":"Code Llama - Python 7B","column":"HumanEval pass@1","reported_value":38.4,"reported_metric":"pass@1","quoted_text":"38.4","pdf_page":7,"notes":"Table 2 of arXiv:2308.12950 reports Code Llama - Python 7B HumanEval pass@1 = 38.4. Driver measures Python perplexity (sanity probe), not HumanEval. PROTOCOL_MATCH = `unknown` because the metric measured differs from the metric the paper reports. Validator C1 gate prevents publication of WRONG."},"protocol_match":"unknown"},"pre":null}}