Intro

I recently had the pleasure of interviewing Scott Cunningham.

I'm sure many of you know him already, but in case you've had your head in the sand, Scott is a professor of economics at Baylor University, the author of Causal Inference: The Mixtape, and the author of a terrific Substack where, for the last six months, he has been writing prolifically on the use of Claude Code for economics research.

You can watch the full interview on YouTube, read Scott's Substack, and I've also made the full transcript available for newsletter subscribers on my site.

In this article I just want to give you my own reflections on what I found particularly interesting, enlightening, and useful from the discussion.

Double-checking Claude Code's output

We began by discussing a collaboration Scott is doing publicly with Caitlin Myers of Middlebury on a new paper.

The paper is being ideated, conceptualized, and written with Scott and Caitlin working together in a series of YouTube videos using Claude Code.

One of Scott's observations from the collaboration is that he had more of a willingness to trust Claude Code's output than Caitlin did. She was much more meticulous, in Scott's view, than he was.

One thing they did was get a former student of Caitlin's, Hannah Sayer, to act as an RA and try to verify independently everything Claude was doing in collecting data, merging it, and so on.

What Hannah found was not, as I understood it, that Claude had completely failed or that the project had to be thrown out. She found some issues in the raw data itself: things like county names or words in the Excel files that were wrong. They then went back over what Claude had done and, at least in Scott's telling, the issues did not seem to change the substantive results very much.

This strikes at the hard question that I think everyone is trying to think through. What is the appropriate amount of validation or verification to do of work produced by agentic coding tools? Where is it merited? Where is it not merited? How exactly should it be done?

In this particular case it seems like Hannah found some things that Claude missed but, since they didn't change the result substantively, this is a step that could have been skipped. But there may have been other papers or other contexts in which the manual checks Hannah did would have found important misses by Claude from the raw data.

What exactly then is the optimal policy for the applied researcher?

Turning repeated checks into skills

The idea I proposed to Scott, and which may be a useful direction for you all to think about, is that if you find yourself regularly doing some kind of manual verification, something I'd recommend is creating some combination of skills and sub-agents to automate these types of verification checks that Claude Code, on a first pass, seems not to get.

Now that might seem counterintuitive because Claude Code made the error, so how can Claude Code solve the error?

The way you use the same underlying model (i.e. the harness) makes a big difference in what results you get.

The benefit of sub-agents is that you can focus the attention of your AI tool on a specific use case and codify some sort of checklist that makes problematic behavior much less likely.

Now if you wanted to completely test some sort of automated verification approach, then what you would also have to do is create what are called evals. Essentially, these are tests for expected behavior so you can evaluate the quality of your combination of skills and sub-agents.

In my opinion this is the direction that I expect applied microeconomics research to go over time. More and more of what applied microeconomists do with their brains will be delegated out to well-constructed agentic systems.

Why expertise still matters

One astute point that Scott made later in the discussion is that he thinks the returns to knowing econometrics well increase rather than decrease due to AI. The reason he thinks this is that you can only work with AI to the extent of your level of understanding. If you have a poor understanding of econometrics, then AI can produce output that superficially looks right but in fact has some substantive error or some taste-based error.

As another example of this, recently Anthropic released their Fable model. This is a roughly equivalent model to Mythos, which Anthropic announced about a month ago and was released only to select firms using it for security hardening purposes.

Tyler Cowen proposed a test of the Fable model: have it ask for and produce an answer to a graduate-level microeconomic theory problem. Ben Golub of Northwestern University and Refine implemented this test, but he found all sorts of problems, both substantive and taste-based/contextual, with the problem and solution that Fable gave him. (Ben Golub's post)

In my own domains of expertise I also see the limits to blindly trusting AI on your hardest problems.

Agentic causal inference systems

However, what I think is possible is the creation of agentic systems that take best practices in causal inference and map them out in a decision tree-like structure. This would allow you to evaluate, in a certain situation, exactly which causal approach and which standard error computation approach you ought to take.

Now there may be taste-based decisions also to make here. Different researchers, for example, have different opinions on the credibility or usefulness of synthetic control methods. And so you'd have to take a stance.

But I do believe that more automation of applied microeconomics research is possible in a way that is not just creating AI slop and is generally useful.

The Fable model, which Anthropic recently released, is only generally available until June 23. And so what I'm going to be doing, in part, is playing around with the creation of agentic causal systems. If you have a Claude Code subscription, I also recommend that you use Fable on your hardest problems while it's available.