Live counts
- Skipped (total)3
- Not attempted3
- Out of budget0
Skipped papers
- 2026-05-14 23:24ZNOT ATTEMPTEDCVPR 2022cs.CV
gated_dataset_no_access
v0.1.0-convnext-imagenet-microslice - 2026-05-14 22:40ZNOT ATTEMPTEDarXiv preprintcs.CL
gated_model_no_access
v0.1.0-llama2-hellaswag-microslice - 2026-05-06 17:17ZNOT ATTEMPTEDarXiv preprintcs.CL
closed_weights_not_attempted
v0.1.0-palm2-not-attempted
Why these were skipped
- Gated weights. Models behind a HuggingFace license gate (Llama 2, Gemma, etc.) cannot be loaded inside the hermetic Modal sandbox because the platform contract forbids platform-wide HF_TOKEN secrets (PRD §18.X.1).
- Methodological mismatch. Papers where the reported headline depends on a measurement protocol the v0.1 platform cannot reproduce honestly (custom fine-tuned checkpoint not released, evaluation on internal benchmarks, etc.) get a
not_attemptedstub rather than a misleading PARTIAL. - Out of budget. The v0.1 automated reproduction budget caps Modal walltime and GPU spend per paper. Models that exceed that cap are queued out and the row carries an
out_of_budgetstatus. A principal can manually un-block by re-running with an elevated budget; that future re-run, when it lands, will replace the row.