GitHub shares insights on LLM evaluation for production workflows

GitHub's evaluation of language models (LLMs) for secret scanning highlights the challenges of transitioning from benchmark performance to real-world application. Secret scanning identifies potentially leaked credentials like tokens and keys in repositories, but false positives can waste developer time. GitHub's LLM-based system aimed to reduce false positives while maintaining sufficient recall to ensure security workflows remain safe. The company emphasized starting with product decisions rather than technical adjustments, prioritizing precision to minimize false positives while keeping recall within a predefined safety threshold. Operational guardrails like latency, cost, and reliability were also critical to ensure practical deployment. This approach provides a framework for evaluating LLMs in code analysis, security, and other production workflows.