Methodology

• Hardware: Test hardware details will be published with the first results. (exact GPUs, drivers, RAM published per run).
• Runtimes: {{RUNTIMES e.g. llama.cpp build, vLLM, Ollama version}}.
• Tasks: fixed, versioned task suites: structured extraction, coding fixes with unit tests, long-document summarization, multi-step tool calling. Prompts published.
• Scoring: automatic where possible (tests pass, JSON field accuracy); rubric-based otherwise with published rubric.
• Speed: prompt processing and generation tokens/sec, single stream, stated context length.
• Reproducibility: configs and raw outputs in https://localevals.com.
• Independence: sponsors never influence scores; sponsored test requests are labeled and results are published as measured.