Hamel Husain
ML engineering & tooling
ML engineer, formerly at GitHub (Copilot) and Airbnb. Writes on practical ML engineering.
Recent activity
-
Do Automated Evals Work?
We compared 100 human annotated traces against automated eval systems. Here's what we found.
Read more → -
“It’s Hard to Eval” Is a Product Smell
.da-fig { max-width: 600px; margin: 1.6rem auto; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; /* proof-surface tint: neutrals nudged toward one hue at low chroma. Change --proof-hue (and --proof-chroma) to retint the whole panel. Hue 75 = warm stone. */ --proof-hue: 75; --proof-chroma: 1; --proof-surface: oklch(0.985 calc(0.006 * var(--proof-chroma)) var(--proof-hue)); /* the disclosure header is a control: step it darker + a touch warmer than
Read more → -
The Revenge of the Data Scientist
img.img-fluid { border: 1px solid rgba(255, 255, 255, 0.25); border-radius: 4px; } Is the heyday of the data scientist over? The Harvard Business Review once called it “The Sexiest Job of the 21st Century.” 1 In tech, data scientist roles were often among the best paid. 2 The job also demanded an unusual mix of skills: Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician. — JosH100 ( @josh_wills ) May 3, 2012
Read more → -
Evals Skills for Coding Agents
Today, I’m publishing evals-skills , a set of skills for AI product evals 1 . They guard against common mistakes I’ve seen helping 50+ companies and teaching 4,000+ students in our course . Why Skills for Evals Coding agents now instrument applications, run experiments, analyze data, and build interfaces. I’ve been pointing them at evals. OpenAI’s Harness Engineering article makes the case well. They built a product entirely with Codex agents — three engineers, five months, ~1 million lines of c
Read more → -
Why I Stopped Using nbdev
Programmers love to proclaim they’ve found the best tool. Paul Graham called Lisp his “ secret weapon .” DHH described Ruby as “ a magical glove that just fit my brain perfectly .” Pieter Levels ships million-dollar products with vanilla PHP and jQuery . These declarations aren’t about the languages themselves. They’re about developers finding tools that fit how they think. When the environment clicks, you move fast. I had that experience with nbdev , a development environment for literate progr
Read more → -
Selecting The Right AI Evals Tool
Over the past year, I’ve focused heavily on AI Evals , both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that comparisons become outdated immediately. Having used many of the popular eval tools, I can genuinely say t
Read more → -
LLM Evals: Everything You Need to Know
This document curates the most common questions Shreya and I received while teaching 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. For a guided path through the rest of our evals work, use the AI evals topic hub . 👉 Want to learn more about AI Evals? Check out our AI Evals course . It’s a live cohort with hands on exercises and office hours. Here is a 25% discount code for readers. 👈 Listen to
Read more → -
A Field Guide to Rapidly Improving AI Products
Most AI teams focus on the wrong things. Here’s a common scene from my consulting work: AI TEAM Here’s our agent architecture – we’ve got RAG here, a router there, and we’re using this new framework for… ME [Holding up my hand to pause the enthusiastic tech lead.] “Can you show me how you’re measuring if any of this actually works?” … Room goes quiet This scene has played out dozens of times over the last two years. Teams invest weeks building complex AI systems, but can’t tell me if their chang
Read more →