{"paper":{"arxiv_id":"2401.02385","title":"TinyLlama: An Open-Source Small Language Model","abstract":"We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at https://github.com/jzhang38/TinyLlama.","primary_category":"cs.CL","venue":"arXiv 2024","published_at":null,"latest_version":1,"withdrawn":false},"latest_version":{"id":"cb6daf02-b697-40d3-bc6f-b5f360c383af","version":1,"source_url":"https://arxiv.org/abs/2401.02385","rendered_html_url":null,"rendering_engine":null},"verdict":{"id":"4538fa12-ff82-48ef-8b80-7770c856abef","kind":"POST","status":"partial","score":0.571,"confidence":0.6,"agent_version":"v0.1.0-tinyllama-hellaswag-microslice","computed_at":"2026-05-14T23:56:41.156Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2401.02385","section":"Table 2","row":"TinyLlama 1.1B (3T)","column":"HellaSwag 0-shot","reported_value":59.2,"reported_metric":"accuracy","quoted_text":"TinyLlama 59.20","pdf_page":4,"notes":"Table 2 of arXiv:2401.02385 reports TinyLlama-1.1B (3T tokens) HellaSwag 0-shot = 59.20. Driver evaluates the same intermediate checkpoint on a HellaSwag micro-slice. PROTOCOL_MATCH is `proxy` (dataset-size)."},"protocol_match":"proxy"},"verdicts":{"post":{"id":"4538fa12-ff82-48ef-8b80-7770c856abef","kind":"POST","status":"partial","score":0.571,"confidence":0.6,"agent_version":"v0.1.0-tinyllama-hellaswag-microslice","computed_at":"2026-05-14T23:56:41.156Z","is_current":true,"claim_citation":{"paper_arxiv_id":"2401.02385","section":"Table 2","row":"TinyLlama 1.1B (3T)","column":"HellaSwag 0-shot","reported_value":59.2,"reported_metric":"accuracy","quoted_text":"TinyLlama 59.20","pdf_page":4,"notes":"Table 2 of arXiv:2401.02385 reports TinyLlama-1.1B (3T tokens) HellaSwag 0-shot = 59.20. Driver evaluates the same intermediate checkpoint on a HellaSwag micro-slice. PROTOCOL_MATCH is `proxy` (dataset-size)."},"protocol_match":"proxy"},"pre":null}}