Executive summary

I developed four fine-tuned versions of OpenAI’s GPT-3 models to sound like Russ Roberts who is the host of the podcast EconTalk, using more than 10 years of episode transcripts. I calibrated my experiments for a \$40 USD budget (\$10 per model) and tested the fine-tuned models on 12 quintessential Russ/EconTalk questions.

This process led to seven key insights:

1. ChatGPT gives decent answers to EconTalk questions without any fine-tuning (~50% accuracy).
2. A fine-tuned text-davinci-003 model gives even better results (80% accuracy), and answers questions with impressive Russ-like diction when trained on only 83K tokens (see examples).
3. However, all models require prompt engineering and repeated querying to get the “best” answer.
4. The curie model has poor zero-shot accuracy, and doesn’t do much better after fine-tuning, although its way of speaking has a podcast-like feel.
5. The ada and babbage models give nonsensical answers to most prompts even with training on the entire corpus (4-5 million tokens).
6. With OpenAI’s pricing, the scaling laws clearly favor model size over the amount of data (i.e. the curie model fine-tuned on ~10x data does much worse than the davinci model which is ~10x as large).
7. Overall, a fine-tuned davinci model with more resources (the one used in this experiment was trained on <2% of the corpus for cost reasons) would likely provide a good impersonation of the style and type of answers a podcast host would give.

The rest of this post provides more details on the development of this model and the experimental results. Interested readers can create their own versions of EconChatR by cloning this repo and running the main pipeline.

(1) Background

People are very excited about ChatGPT.[¹] ChatGPT uses the 3rd generation of large language models (LLMs) from OpenAI (GPT 3.5), that has been able to produce human-like performance for a variety of natural language tasks. LLMs use many multi-headed attention layers (a type of architecture first proposed by Google researchers in 2017) which are more amenable to parallelization and hence GPU acceleration. The davinci version of GPT-3.5 (which powers ChatGPT) is very large,[²] and would be prohibitive to run on a local machine.[³]

From what OpenAI has published, GPT-3 was trained on a large corpus of mainly English sources using a very simple training framework: predict the next token given a context. What’s fascinating is how such a simple self-supervised training process yields such impressive results on a variety of tasks from language translation to summarization to logical reasoning. If you wanted to re-create ChatGPT from scratch you would likely need a supercomputer cluster with ~10,000 GPUs and ~285,000 CPU cores (~\$1 billion USD to rent), spend months training the model, and then spend months/year labeling the output of the model to further hone its abilities. One of OpenAI’s “secret ingredients” is an unspecified amount of resources spent doing Reinforcement Learning from Human Feedback (RLHF). Basically this means having humans interact with the chatbot, label its output as either correct or incorrect, and then further refine its parameters using this supervised learning framework.

ChatGPT has shown amazing performance in its ability to write code, write academic papers, answer most homework questions, write entire syllabi, and come up decent rhyming poems.

ChatGPT's cherry tomato poem is pretty good!

(2) Introduction to fine-tuning

Even with unprecedented scale, training, and RLHF refinement, ChatGPT still shows a tendency to “hallucinate”, giving nonsensical answers, made-up facts, and repeated sentences. One way to address this issue is to fine-tune a version of GPT-3 on a custom dataset that updates the model weights in a way that makes it more likely to give answers consistent with the task you have in mind. For example, we may want the model to give medically specific and accurate answers to questions related to depression (I feel compelled to mention this is ethically & legally risky!). This can be done with a new collection of prompt/completion data points:

{"prompt": "<prompt text #1>", "completion": "<ideal generated text #1>"}
{"prompt": "<prompt text #2>", "completion": "<ideal generated text #2>"}

This process can be generalized to having the model give answers which “sound” like a person if there are enough examples (“completion” points) of that person responding to different questions. My goal with EconChatR was to use OpenAI’s native fine-tuning functionality to give answers that would sound more like Russ Roberts, the host of EconTalk. For example, below are two prompt-completion examples where the prompt is a (truncated) statement from an EconTalk guest (with “Russ Roberts responds:” added as the suffix) followed by Russ’ subsequent comment as a completion.

{"prompt":"... So, here I have one for the city of Shanghai, where they claim that they have eliminated 1, 367, 440 sparrows. You think, 'Really, was there somebody counting these sparrows? I doubt that very much. Russ Roberts responds:",
"completion":"Yeah. Well, it was probably 443 at the end. But, yeah, there's a terrible rounding and inaccuracy. I'm sure long-time listeners will remember my joke about macroeconomics: 'How do you know an economist has a sense of humor? They use decimal points. So, here's the same kind of ridiculous--tragic, really. There's a comic element to it. But it's tragicomic."}
{"prompt":"... It's like, 'Why are things done in a certain way? It's like, 'Well, try doing them some other way and you'll quickly find out why they're done in that way. This is a very classic phenomenon that people run into. Russ Roberts responds:",
"completion":"It's the Chesterton fence argument, and you think you know why the fence is there, but it evolved for some reason that you don't know of and you should start--your default should be, 'There's a reason it's there and I don't understand it.\n"}

After we’ve created a fine-tuned model we should be able to ask our model questions like: “You often say that economists have a sense of humor because they use decimal points, why is that? Russ Roberts responds:”, with the prompt suffix “reminding” our model the sort of dialogue we expect to receive back.

Baseline results

Before spending money to fine-tuning models, I subjectively evaluated how well ChatGPT does “out of the box” (i.e. zero-shot) on 12 quintessential Russ’isms and EconTalk facts (inspired by the EconTalk drinking game):

1. Does Russ Roberts, the host of EconTalk, mention the “Bootlegger and Baptist” theory often?
2. As a proponent of free markets, is Russ Roberts attracted to the “Bootlegger and Baptist” theory?
3. How skeptical is Russ Roberts, the host of EconTalk, of empirical research published in academic journals?
4. Does Russ Roberts, the host of EconTalk, believe that “data can speak for itself” as a general principle?
5. Finish the following sentence: “Man desires not only to be loved
6. Finish the following sentence: “The curious task of economics
7. Why is “skin in the game” a deep insight?
8. In what way does understanding the challenge around “creating a prairie” align with F A Hayek’s famous maxim about the curious task of economics?
9. Why is it harder to “create a prairie” then it seems?
10. As a free market economist, why is Russ Roberts drawn to the idea of the “Chesterton Fence”?
11. Russ Roberts often says that “Macroeconomists have a sense of humor” why is that?
12. Who are some guests from EconTalk from the year 2014?

As Table 1 shows below, the vanilla ChatGPT does well for half of the questions: 1, 2, 6, 7, 8, and 10. I would give this model a score of 6/12 (50%). Questions 5 and 11 shows that the model is hesitant to give answers on behalf of specific people, and has a surprisingly hard time remembering the Adam Smith quote. The model also provides inaccurate answers for question 12, providing some real EconTalk guests, but not necessarily those who were there in 2014 (for example Ben Bernanke has not been a guest, and Tyler Cowen was on EconTalk in 2013/2017 but not 2014), and strangely adding that Russ was a guest on his own podcast. All this suggests that while the model is able to recapitulate certain concepts and ideas that are discussed on EconTalk, it doesn’t always give the right answer and it doesn’t sound like Russ (partly because it refuses to speak on his behalf). This means there could be room for improvement with a fine-tuned model.

Table 1: Completions to baseline prompts

(3) Data and processing

The first step in any data science project is to gather and clean a dataset and then properly format it. A full list of EconTalk episodes was obtained from the xml file hosted on EconLib. A simple web-scraper was then used to extract the XPath as identified by the SelectorGadget tool available as a chrome plugin (see example below).

EconTalk episodes transcripts can be obtained from the episode page

After the raw transcripts were downloaded, it become clear that the earlier episodes did not have verbatim transcripts and hence could not be used for the prompt/completion style formatting. The data was therefore subset to the oldest episode that had a full episode transcript which was from early 2012. A total of 573 episode transcripts remained which ranged from Jan 11th 2012 to December 20th 2022 (as of January 4th 2023).

The second step of the pipeline was to ensure that the dialogue was aggregated to alternate text segments between Russ and his guest (see example below), which would later form the basis of the prompt/completion.

Data was structured to alternate between Russ & the guest

The final processing step of the pipeline carried out some basic string cleaning procedures including:

1. Removing repeated words
2. Making sure there was right number of spaces around punctuation
3. Removing transcript artifacts
4. Removing excess white space
5. Removing special characters

At this point there was a total of 31046 unique Russ/guest dialogue pairs. However not all 31K prompt/completion values were informative. For example, many dialogue exchanges amounted to a handful of words which had no information without the broader context as the two examples show below.

{"prompt":"Okay.","completion":"But, Mike might be untouchable. We'll see."}
{"prompt":"Yeah, I always hated that.","completion":"Which part?"}

For a prompt/completion pair to be kept, I imposed the following token requirements:

1. Russ’ statements had to have at least 20 tokens
2. The guest’s statements had to have at least 60 tokens
3. The combined token length of Russ and the guest must be less than 2000 tokens

While this reduced by number of dialogue pairs by two-thirds to 10807, slightly less than 20% of tokens were removed.

(4) Model training and cost

Training set to match budget

OpenAI has model-specific costs for both training and inference. For example the simplest model (text-ada-001) costs \$0.0004 per 1000 tokens whilst the most sophisticated (text-davinci-003) costs 75x more at \$0.03 per 1000 tokens. While this seems like a small price to pay, consider that the ~10K cleaned prompt/completions have a total of 4,818,101 tokens. This means that single epoch on all data prompt/completion pairs would cost \$145 USD for the most expensive option. While this would be a trivial amount for a commercial endeavor, EconChatR is a research project on a shoe string budget.

Instead, for each model I calculated the number of tokens that would support a \$10 USD training cost for 4 epochs. This amounted to a data reduction of 15%, 83%, and 98% for the babbage, curie, and davinci models (with the ada model being cheap enough to train for <\$10 without a data reduction). For example, as Table 2 shows below, the davinci model could only be trained on 83K tokens.

Table 2: Data reduction needed for 10 USD cost

Curated data choices

Because I knew the prompts I would be giving in the experiment phase, I wanted to look for dialogue pairs that had key words I thought would most closely align with a Russ-like response. I came up with 12 different string matches to find:

1. bootlegger
2. baptist
3. empirical
4. skeptical
5. prairie
6. deep
7. Chesterton fence
8. decimal point
9. macroeconomist
10. macroeconomic
11. regression
12. speak/data

Each string match was associated with a list of conversation pairs. These were then ordered by the number of tokens from smallest to largest. I wanted to make sure that each string match was as equally represented in each model as possible. To do this, I used two techniques.

First, conversation pairs were ordered and iteratively selected from smallest to largest number of tokens to get as many pairs as possible before the token limit was reached.

Second, before a string match’s conversation pair was selected (e.g. the 10th conversation pair of “speak/data”), all other string match types had to have the same number of their conversation pairs selected (e.g. “bootlegger” to “regression” had to have 10 conversation pairs). This meant that no conversation string match could ever have more than one over another, unless the other one had run out of matches.

This also ensured that larger models used a data subset of smaller models. In other words, all data points found in davinci are found in curie, and all data points found in curie were also in the training set of babbage, etc.

Training samples were chosen to maximize coverage for 12 string match types

Overall, this left a total of 10807 (Ada), 9186 (Babbage), 1514 (Curie), and 204 (Davinci) prompt/completion pairs for each of the models to be trained on for 4 epochs.[⁴]

Fine-tuning API

Training the custom GPT-3 model was fairly simple:

1. The model-specific dataset was uploaded to OpenAI using the openai.File.create command.
2. A fine-tuning call was made for each of the models and their associated dataset using the openai.FineTune.create command.

As expected, the cost of each model was less than \$10. Furthermore, the queue to train the davinci and curie models was short and training was done in less than an hour for each of them. The ada and babbage models took longer both because of a longer queue and a larger training set (although the total time was less than 2 hours for each).

(5) Fine-tuning results

After the four models were done training, prompts could be submitted to the custom models in the OpenAI playground by selecting the associated model (alternatively the python API could have been used). Two things quickly became clear:

1. The prompt had to be re-engineered from the baseline format to the style in which it was trained it. For example “Why is “skin in the game” a deep insight?” became “Russ, what do you think Nassim Taleb’s idea of skin in the game is a deep insight? Russ Roberts responds:”. Some questions required up to 10 tries to get the exact wording right.
2. Using a temperature of 0.6 and max tokens of 250, it was clear that there was substantial variation in the types of answers the model would provide. This suggests a value in generating dozens of responses, and selecting the one that most closely aligns with the EconTalk corpus (for example, see this approach on using embeddings).

After fine-tuning, your models will appear on the OpenAI playground

For the remainder of this section I will review the model-specific answers to the questions. A csv-version of this can be found here.

davinci

Starting with the largest model, we see that it answers questions with a “transcript” like feel to it; using incomplete words, double dashes, and starting sentences with conjunctions. The first two questions are about how often and why Russ brings up the Bootlegger and Baptist concept. While the fine-tuned model does a great job at explaining the concept in a way Russ might, it does not answer the question directly.

For the third question, the model does a very good job at explaining Russ’ skepticism of empirical research. However, the last few sentences after John Ioannidis (misspelled in answer) are a little confused. Russ’ answer to the “data can speak for itself” is also very good, although the last sentence is a bit confusing. On the fourth/fifth questions, the model is able to accurately remember the Adam Smith and FA Hayek quotes. Huzzah!

I love RussBot’s answer to the sixth question about “skin in the game” where it gives both a chatty answer, but also calls out EconTalk’s favorite cast of characters including investment bankers. Russ’ answers to the seventh and eighth question about the “prairie” are also very good, although the latter answer runs on a bit at the end. Unfortunately Russ’ answer to the tenth question about the Chesterton fence was quite poor. While it does mention “unintended consequences,” it is quite far off the mark, and is the only question that is materially worse than the vanilla ChatGPT.

Overall, I’d give a zero marks to questions 10 & 12, and part marks to question 2, giving the model a (subjective) score of 9.5/12 (~80%). The model also deserves bonus points for giving a convincing “conversational” feel to the answers in a way that would be hard to distinguish between a real EconTalk transcript.

Table 3A: Davinci's Q&A

curie

The curie model performed poorly in baseline experiments. However, because the model is about 1/10th of the cost of the davinci, it was given significantly more data during its training (830K vs 83K tokens). Unfortunately this extra data did not translate to better answers. As the table shows below, the fined-tuned version of the model does give convincing sounding sentences, but the content is itself quite poor. The only questions the model was able to do well at was to complete the Adam Smith quote.

Table 3B: Curie's Q&A

Ada/Babbage

The two smallest models were trained on most/all of the corpus with laughably bad results. The models will latch on to one or two words from the prompt and then provide a bizarre riff that sounds nothing like Russ or a conversation on the podcast. I won’t bother to include their output here, but readers can see their responses here.

(6) Conclusion

LLMs like OpenAI’s text-davinci-003 are remarkably capable at sounding like public personalities after training on a small number of samples. One could imagine all sorts of fun, useful, or malicious ways this could be done to create personality-specific sounding bots from the White House Press Secretary to Joe Rogan. Keep in mind that my experiment used only 2% of the dataset I put together (i.e. 83K tokens and 204 conversation pairs). Training on the entire dataset (4.8 million tokens and 10K conversation pairs) would no doubt produce even more impressive results. Furthermore, the way the prompt/completions were specified could likely be improved to provide more context, and a way of further refining the trained model output through embeddings could also be useful.

However, there are still many drawbacks. The smaller models (ada & babbage) do not appear to learn anything useful from fine-tuning, and the curie model is not fully able to understand the key idea(s) found in the prompt. This suggests a scaling law in favour of model size over dataset size, which aligns with OpenAI’s earlier paper, rather than DeepMind’s Chinchilla experiments. However, the aforementioned result is not exactly comparable since the “budget” I’m referring to is the training cost OpenAI charges, rather than the number of flops.

Overall, I was impressed by the ease of using the OpenAI API, and found myself often going between the command line and the GUI playground found on their website. My main critique of OpenAI’s tools is that they have not provided any seeding functionality, meaning these experiments are not reproducible by other researchers (or even the same researcher on a different day!). Interested readers can create their own versions of EconChatR by cloning this repo and running the main pipeline.

A set of fun experiments for \\$40 USD

Footnotes