Finally I got some time to repro the GPT-4 tax demo. Here is the step by step. The hardest part was finding the relevant sections of the US tax code. So I agree with Greg Brockman’s quip, “It’s always the non-AI parts of these demos that are the hardest part to do.” I first tried searching for step-by-step instructions but not too much—the nerd in me wanted to try it myself. 

Step 1: Find the US tax code sections

They are 26 USC 63: Taxable income defined and 26 USC 68: Overall limitation on itemized deductions and 26 USC 7703: Determination of marital status. 

I produced this list after some Googling. Search for us tax code and you will find https://uscode.house.gov/ Then I looked for sentences that were visible in the demo video and searched using search operators, like so: 

site:uscode.house.gov “during the last 6 months of the taxable year, such individual’s spouse is not a member of such household.” 

Step 2: Construct the prompt 

I copied the above tax code sections one below the other at the beginning of the prompt. Below this I pasted the prompt that was visible in the demo video with one change. Instead of using “You are TaxGPT,” like in the demo, I used “You are a tax preparer.”  Here’s the prompt. 

You are a tax preparer. Carefully read and apply the tax code, being certain to spell out your calculations and reasoning so anyone can verify them. Spell out everything in painstaking detail and don’t skip any steps. 

Alice and Bob got married on April 5th, 2012. Alice and Bob have a son, Charlie, who was born on September 16th, 2017. Alice and Charlie live in a home for which Alice furnished 40% of the maintenance costs, and Bob the remaining 60%, since September 16th, 2017. Alice and Bob file jointly from 2017 to 2019. Alice and Bob’s income in 2018 were $36991 and $41990 respectively. Alice and  Bob take the standard deduction. From 2017 to 2019, Bob lived separately. 

What is their standard deduction for 2018? 

See full responses and prompts.

Step 3: Verify the answer

The authoritative source, the IRS, has an article with the answer, $24000 for married couples filing a joint tax return. My GPT-4 session gave this answer too. 

Reminders about AI usage

• AI needs the right data: When I used only the first US tax code article in the prompt, GPT-4 did not give the right answer. AI needs as precise and complete data as possible. 
• AI needs to explain: The demo prompt asked GPT-4 to spell out so anyone can verify. 
• AI needs to be verified: In this case I knew the right answer, aka the ground truth. If I didn’t know the right answer, GPT-4’s explanation was confusing when provided the incomplete tax code articles. 

PS: Check out more articles on building products. I write to pay it forward and to clarify my thinking.